search

genomehubs / demo

Example configurations to set up custom Ensembl sites using GenomeHubs
genomehubs.org
MIT License
2 stars 2 forks source link

import.sh won't run #1

Closed guyleonard closed 7 years ago

guyleonard commented 7 years ago

Distributor ID: Ubuntu Description: Ubuntu 14.10 Release: 14.10 Codename: utopic

Followed instructions here - http://genomehubs.org/documentation/installing-docker/ and here https://easy-import.readme.io/docs/quick-start-guide with same results as below...?

Any thoughts? Seems to fail at step 3 on permissions!?

bash demo/import.sh Step 1. Set up mySQL container b730d173423ec43c53b16c1edb03c680cad41365162e783df612681c775c29a2 Step 2. Set up template database using EasyMirror Working on ftp://ftp.ensemblgenomes.org/pub/release-32/pan_ensembl/mysql//ncbi_taxonomy as ncbi_taxonomy ncbi_taxonomy.ncbi_taxa_name: Records: 2230273 Deleted: 0 Skipped: 0 Warnings: 0 ncbi_taxonomy.ncbi_taxa_node: Records: 1473354 Deleted: 0 Skipped: 0 Warnings: 0 Working on ftp://ftp.ensemblgenomes.org/pub/release-32/metazoa/mysql//melitaea_cinxia_core_32_85_1 as melitaea_cinxia_core_32_85_1 melitaea_cinxia_core_32_85_1.alt_allele: Records: 0 Deleted: 0 Skipped: 0 Warnings: 0 CLIPPED Step 3. Import sequences, prepare gff and import gene models mkdir: cannot create directory 'operophtera_brumata_v1_core_32_85_1': Permission denied /import/startup.sh: line 51: cd: operophtera_brumata_v1_core_32_85_1: No such file or directory mkdir: cannot create directory 'log': Permission denied importing sequences tee: log/import_sequences.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. Obru_genes.gff.gz: Permission denied ERROR: could not wget http://download.lepbase.org/v4/provider/Obru_genes.gff.gz to Obru_genes.gff.gz preparing gff tee: log/prepare_gff.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. Obru_genes.gff.gz: Permission denied ERROR: could not wget http://download.lepbase.org/v4/provider/Obru_genes.gff.gz to Obru_genes.gff.gz importing gene models tee: log/import_gene_models.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. Obru_genes.gff.gz: Permission denied ERROR: could not wget http://download.lepbase.org/v4/provider/Obru_genes.gff.gz to Obru_genes.gff.gz Step 4. Export sequences, export json and index database for imported Operophtera brumata mkdir: cannot create directory 'operophtera_brumata_v1_core_32_85_1': Permission denied /import/startup.sh: line 51: cd: operophtera_brumata_v1_core_32_85_1: No such file or directory mkdir: cannot create directory 'log': Permission denied exporting sequences mkdir: cannot create directory '/import/download/sequence': Permission denied tee: log/export_sequences.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.meta' doesn't exist at /ensembl/ensembl/modules/Bio/EnsEMBL/DBSQL/BaseMetaContainer.pm line 140, line 94. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.meta' doesn't exist at /ensembl/ensembl/modules/Bio/EnsEMBL/DBSQL/BaseMetaContainer.pm line 140, line 94. /import/startup.sh: line 143: cd: exported: No such file or directory ls: cannot access Operophtera_brumata_v1.scaffolds.fa: No such file or directory ls: cannot access Operophtera_brumata_v1.cds.fa: No such file or directory ls: cannot access Operophtera_brumata_v1.proteins.fa: No such file or directory

cp: cannot stat 'exported/Operophtera_brumata_v1.scaffolds.fa': No such file or directory parallel: Warning: $SHELL not set. Using /bin/sh. Can't do inplace edit: /import/blast/ is not a regular file. Can't rename /import/blast/.scaffolds.fa /import/blast/_scaffolds.fa: No such file or directory Can't rename /import/blast/.cds.fa /import/blast/_cds.fa: No such file or directory Can't rename /import/blast/.proteins.fa /import/blast/_proteins.fa: No such file or directory gzip: exported/.fa: No such file or directory mv: cannot stat 'exported/.gz': No such file or directory exporting json mkdir: cannot create directory '/import/download/json': Permission denied mkdir: cannot create directory '/import/download/json': Permission denied mkdir: cannot create directory '/import/download/json': Permission denied mkdir: cannot create directory '/import/download/json': Permission denied tee: log/export_json.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.meta' doesn't exist at /ensembl/ensembl/modules/Bio/EnsEMBL/DBSQL/BaseMetaContainer.pm line 140, line 94. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.meta' doesn't exist at /ensembl/ensembl/modules/Bio/EnsEMBL/DBSQL/BaseMetaContainer.pm line 140, line 94. done mv: cannot stat 'web/.codon-usage.json': No such file or directory mv: cannot stat 'web/.assembly-stats.json': No such file or directory mv: cannot stat 'web/*.meta.json': No such file or directory indexing database tee: log/index_database.err: No such file or directory keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. keys on reference is experimental at /ensembl/easy-import/modules/EasyImport/Core.pm line 738. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.meta' doesn't exist at /ensembl/easy-import/modules/EasyImport/Search.pm line 26, line 94. DBD::mysql::st fetchrow_array failed: fetch() without execute() at /ensembl/easy-import/modules/EasyImport/Search.pm line 27, line 94. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.seq_region' doesn't exist at /ensembl/easy-import/modules/EasyImport/Search.pm line 11, line 94. DBD::mysql::st fetchrow_array failed: fetch() without execute() at /ensembl/easy-import/modules/EasyImport/Search.pm line 12, line 94. DBD::mysql::st execute failed: Table 'operophtera_brumata_v1_core_32_85_1.gene' doesn't exist at /ensembl/easy-import/modules/EasyImport/Search.pm line 39, line 94. DBD::mysql::st fetchrow_array failed: fetch() without execute() at /ensembl/easy-import/modules/EasyImport/Search.pm line 40, line 94. Unable to set up GenomeHubs site, removing containers genomehubs-mysql genomehubs-mysql Error response from daemon: no such id: genomehubs-ensembl Error: failed to stop containers: [genomehubs-ensembl] Error response from daemon: no such id: genomehubs-h5ai Error: failed to stop containers: [genomehubs-h5ai] Error response from daemon: no such id: genomehubs-sequenceserver Error: failed to stop containers: [genomehubs-sequenceserver]

rjchallis commented 7 years ago

Sorry it didn't work out of the box for you. My best guess here is that you are running the script as a user with a UID other than 1000, which is difficult to support in a generic script like this.

All the GenomeHubs containers use UID 1000 so the default user on a new Ubuntu install can run them - workarounds to get the script to work with a different UID (e.g. setting a+w permissions on the mounted directories) could lead to the containers creating files that can't be removed by the user that created them so I opted for assuming that it would be run by the default user.

Currently this is mentioned as a note in the comments at the top of the script but I should probably add a UID test before the rest of the script runs so it could give a more informative error message and not attempt a setup that won't work.

guyleonard commented 7 years ago

Aha, indeed I see the note now. Does that make it essentially 'unrunnable' on a shared system then? I could change my user to UID1000 on my own machines but not on machines I don't admin...?

rjchallis commented 7 years ago

That's probably a fair assessment - there are too many nuances with Docker permissions to keep the demo script simple without assuming access to UID1000 and as we've spun up new vms to test new iterations of the script, we've not paid so much attention to it.

I think there should be a solution if I stop mounting some of the directories that are just mounted for convenience (e.g. /ensembl/logs) and edit the demo script to run the containers that need to write to mounted directories (EasyImport and SequenceServer) with -u $USER:$GROUPS. This will require a little modification to the containers to support it but I think it should resolve your issue.

guyleonard commented 7 years ago

Thanks! Although your reply maybe suggests that rolling my own from EasyImport wouldn't necessarily need to use UID 1000? WIll keep trying :)

rjchallis commented 7 years ago

Just updated containers and committed a fix for this so it should now run with any UID - hopefully this will now run properly for you.

guyleonard commented 7 years ago

Thank you so much! :) It now works!

rjchallis commented 7 years ago

Glad to hear it's working for you now.