only1chunts
commented
10 months ago So I have rerun the script again and it is half working... The MD5sum values are indeed added to the database, BUT not being displayed in the mockup page (probably the same issue as described in other tickers #1528 ) the on-screen output is copied below, but here is my summary:
- The MD5sum values are added to the database (but not displaying)
- The File sizes are NOT being added to the database
- There are no files listed in the "updating-file-size-102462.txt" file
- There are no files listed in the "invalid-urls-102462.txt" log file
- It has created the readme-102462.txt file in the beta server ready to be copied over.
[chrish@ip-10-99-0-237 ~]$ sudo /home/centos/postUpload.sh 102462
* About to update files' size for 102462
Exception 'yii\db\Exception' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p:648
Error Info:
Array
(
[0] => 08006
[1] => 7
[2] => timeout expired
)
Caused by: Exception 'PDOException' with message 'SQLSTATE[08006] [7] timeout ex pired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p:722
Stack trace:
#0 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(722): PDO->__construct('pgsql:host=rds-...', 'gigadb', Object(SensitiveParamet erValue), NULL)
#1 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(637): yii\db\Connection->createPdoInstance()
#2 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(1067): yii\db\Connection->open()
#3 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(1054): yii\db\Connection->getMasterPdo()
#4 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Schema.php(46 1): yii\db\Connection->getSlavePdo()
#5 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(946): yii\db\Schema->quoteValue('102462')
#6 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(2 11): yii\db\Connection->quoteValue('102462')
#7 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1 126): yii\db\Command->getRawSql()
#8 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1 147): yii\db\Command->logQuery('yii\\db\\Command:...')
#9 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(4 24): yii\db\Command->queryInternal('fetch', NULL)
#10 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Query.php(28 7): yii\db\Command->queryOne()
#11 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/ActiveQuery. php(304): yii\db\Query->one(NULL)
#12 /gigadb/app/tools/files-metadata-console/components/DatasetFilesUpdater.php( 50): yii\db\ActiveQuery->one()
#13 /gigadb/app/tools/files-metadata-console/controllers/UpdateController.php(35 ): app\components\DatasetFilesUpdater->updateFileSize()
#14 [internal function]: app\controllers\UpdateController->actionFileSize()
#15 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/InlineActi on.php(57): call_user_func_array(Array, Array)
#16 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Controller .php(178): yii\base\InlineAction->runWithParams(Array)
#17 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Control ler.php(180): yii\base\Controller->runAction('file-size', Array)
#18 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Module.php (552): yii\console\Controller->runAction('file-size', Array)
#19 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Applica tion.php(180): yii\base\Module->runAction('update/file-siz...', Array)
#20 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Applica tion.php(147): yii\console\Application->runAction('update/file-siz...', Array)
#21 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Applicatio n.php(384): yii\console\Application->handleRequest(Object(yii\console\Request))
#22 /gigadb/app/tools/files-metadata-console/yii(20): yii\base\Application->run( )
#23 {main}
Done with updating files' size for 102462. Nb of successful changes saved in fil e: /home/centos/uploadLogs/updating-file-size-102462.txt
* About to check that file urls are valid for 102462
Exception 'yii\db\Exception' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p:648
Error Info:
Array
(
[0] => 08006
[1] => 7
[2] => timeout expired
)
Caused by: Exception 'PDOException' with message 'SQLSTATE[08006] [7] timeout ex pired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p:722
Stack trace:
#0 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(722): PDO->__construct('pgsql:host=rds-...', 'gigadb', Object(SensitiveParamet erValue), NULL)
#1 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(637): yii\db\Connection->createPdoInstance()
#2 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(1067): yii\db\Connection->open()
#3 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(1054): yii\db\Connection->getMasterPdo()
#4 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Schema.php(46 1): yii\db\Connection->getSlavePdo()
#5 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.ph p(946): yii\db\Schema->quoteValue('102462')
#6 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(2 11): yii\db\Connection->quoteValue('102462')
#7 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1 126): yii\db\Command->getRawSql()
#8 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1 147): yii\db\Command->logQuery('yii\\db\\Command:...')
#9 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(4 24): yii\db\Command->queryInternal('fetch', NULL)
#10 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Query.php(28 7): yii\db\Command->queryOne()
#11 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/ActiveQuery. php(304): yii\db\Query->one(NULL)
#12 /gigadb/app/tools/files-metadata-console/components/FilesURLsFetcher.php(35) : yii\db\ActiveQuery->one()
#13 /gigadb/app/tools/files-metadata-console/controllers/CheckController.php(37) : app\components\FilesURLsFetcher->verifyURLs()
#14 [internal function]: app\controllers\CheckController->actionValidUrls()
#15 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/InlineActi on.php(57): call_user_func_array(Array, Array)
#16 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Controller .php(178): yii\base\InlineAction->runWithParams(Array)
#17 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Control ler.php(180): yii\base\Controller->runAction('valid-urls', Array)
#18 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Module.php (552): yii\console\Controller->runAction('valid-urls', Array)
#19 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Applica tion.php(180): yii\base\Module->runAction('check/valid-url...', Array)
#20 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Applica tion.php(147): yii\console\Application->runAction('check/valid-url...', Array)
#21 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Applicatio n.php(384): yii\console\Application->handleRequest(Object(yii\console\Request))
#22 /gigadb/app/tools/files-metadata-console/yii(20): yii\base\Application->run( )
#23 {main}
Done with checking that file urls are valid for 102462. Invalid Urls (if any) ar e save in file: /home/centos/uploadLogs/invalid-urls-102462.txt
* About to update files' MD5 Checksum as file attribute for 102462
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/10246 2/102462.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/10246 2/102462.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/102462/1024 62.md5
Saved md5 file attribute with id: 673934
Saved md5 file attribute with id: 673935
Saved md5 file attribute with id: 673936
Saved md5 file attribute with id: 673937
Saved md5 file attribute with id: 673938
Saved md5 file attribute with id: 673939
Saved md5 file attribute with id: 673940
Saved md5 file attribute with id: 673941
Saved md5 file attribute with id: 673942
Saved md5 file attribute with id: 673943
Saved md5 file attribute with id: 673944
Saved md5 file attribute with id: 673945
Saved md5 file attribute with id: 673946
Saved md5 file attribute with id: 673947
Saved md5 file attribute with id: 673948
Saved md5 file attribute with id: 673949
Saved md5 file attribute with id: 673950
Saved md5 file attribute with id: 673951
Saved md5 file attribute with id: 673952
Saved md5 file attribute with id: 673953
Saved md5 file attribute with id: 673954
Saved md5 file attribute with id: 673955
Saved md5 file attribute with id: 673956
Saved md5 file attribute with id: 673957
Saved md5 file attribute with id: 673958
Saved md5 file attribute with id: 673959
Done with updating files' MD5 Checksum as file attribute for 102462. Process sta tus is saved in file: /home/centos/uploadLogs/updating-md5checksum-102462.txt
* About to create the README file for 102462
[DOI] 10.5524/102462
[Title] Supporting data for "Evolutionary genomics of three agricultural pest
moths reveals rapid evolution of host adaptation and immune-related genes"
[Release Date] 2023-11-03
[Citation] Kawahara, AY; Plotkin, D; Weng, Y; Pathour, SR; Godfrey, KR; Parker,
BM; Wist, T (2023): Supporting data for "Evolutionary genomics of three
agricultural pest moths reveals rapid evolution of host adaptation and
immune-related genes" GigaScience Database. https://dx.doi.org/10.5524/102462
[Data Type] Software,Genomic
[Data Summary] Understanding the genotype of pest species provides an important
baseline for designing integrated pest management (IPM) strategies. Recently
developed long-read sequence technologies make it possible to compare genomic
features of non-model pest species to disclose the evolutionary path underlying
the pest species profiles. While hundreds of published genomes of lepidopteran
species (moths and butterflies) are now available, only a few genome assemblies
are publicly available from this diverse and notorious agricultural pest family,
Gelechiidae. We sequenced and assembled genomes for three gelechiids:
<i>Phthorimaea absoluta</i> (tomato leafminer), <i>Keiferia lycopersicella</i>
(tomato pinworm), and <i>Scrobipalpa atriplicella</i> (goosefoot groundling
moth). We compared these genomes with published genomes of <i>Phthorimaea
operculella</i> and Pectinophora gossypiella, and found that the three
solanaceous feeding species, <i>Ph. absoluta, K. lycopersicella</i>, and <i>Ph.
operculella</i> are clustered together. Gene family evolution analyses with the
five species show clear gene family expansions on hostplant associated genes for
the three solanaceous feeding species. These genes are involved in host
compounding sensing (e.g., gustatory receptors), detoxification (e.g.,
Cytochrome P450, Glucose-methanol-choline oxidoreductase, Glutathione
S-transferase, Insect cuticle proteins, and UDP-glucuronosyl), and digestion
(e.g., serine proteases and Aminopeptidase N-type). A gene ontology enrichment
analysis of rapid evolving genes also suggests enriched functions in host
sensing and immunity. Our results indicate that host plant adaptation and
pathogen defense could be important drivers in species diversification among
gelechiid moths.
[File Location] https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/102462 /
[File name] - [File Description]
readme_102462.txt -
astral_species_tree.tre - The species tree drawn with ASTRAL3 based on the 4,8 76 single copy gene trees
0_pope_result_uniq - The result of gene family expansion/contraction analysis on *Phthorimaea operculella* using CAFE v5.0.0 with significant cutoff at p-valu e=0.01
1_kely_result_uniq - The result of gene family expansion/contraction analysis on *Keiferia lycopersicella* using CAFE v5.0.0 with significant cutoff at p-valu e=0.01
2_tuta_result_uniq - The result of gene family expansion/contraction analysis on *Phthorimaea absoluta* using CAFE v5.0.0 with significant cutoff at p-value=0 .01
3_0_result_uniq - The result of gene family expansion/contraction analysis on internal node #3 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Phthorimae a operculella*
3_1_result_uniq - The result of gene family expansion/contraction analysis on internal node #3 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Keiferia l ycopersicella*
4_scat_result_uniq - The result of gene family expansion/contraction analysis on *Scrobipalpa atriplicella* using CAFE v5.0.0 with significant cutoff at p-val ue=0.01
5_0_result_uniq - The result of gene family expansion/contraction analysis on internal node #5 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Phthorimae a operculella*
5_1_result_uniq - The result of gene family expansion/contraction analysis on internal node #5 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Keiferia l ycopersicella*
5_2_result_uniq - The result of gene family expansion/contraction analysis on internal node #5 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Phthorimae a absoluta*
6_pego_result_uniq - The result of gene family expansion/contraction analysis on *Pectinophora gossypiella* using CAFE v5.0.0 with significant cutoff at p-val ue=0.01
7_0_result_uniq - The result of gene family expansion/contraction analysis on internal node #7 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Phthorimae a operculella*
7_1_result_uniq - The result of gene family expansion/contraction analysis on internal node #7 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Keiferia l ycopersicella*
7_2_result_uniq - The result of gene family expansion/contraction analysis on internal node #7 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Phthorimae a absoluta*
7_5_result_uniq - The result of gene family expansion/contraction analysis on internal node #7 (see header of HOGs_gamma_change_matrix.tsv) using CAFE v5.0.0 with significant cutoff at p-value=0.01 and functional annotation of *Pectinopho ra gossypiella*
HOGs_gamma_change_matrix.tsv - The raw family size change output of CAFE v5.0. 0
HOG_for_cafe.tsv - The Hierarchical Orthologous Groups (HOGs) data matrix gene rated from OrthoFinder for CAFE v5.0.0 analysis
significant_change_HOGs.tsv - All HOGs passed through the significant cutoff a t p-vlaue=0.01
supermatrix_tree.tre - The molecular phylogeny drawn with the concatinated sup ermatrix from the 4,876 single copy gene sequences using IQ-TREE v2.2.2
super_matrix_alignment.fasta - The concatinated supermatrix from the 4,876 ali gned single copy gene sequences
Table_S1_gFACs_results.csv - Supplemental table S1, the gene model summaries u sing gFACs
Table_S2_list_of_rapidly_evolve_HOGs.csv - Supplemental table S2, the list of significanly expanding/contracting HOGs
Table_S3_immunity_HOGs.csv - Supplemental table S3, the significanly expanding /contracting HOGs that are associated with immunity
Table_S4_host_adaptation_HOGs.csv - Supplemental table S4, the significanly ex panding/contracting HOGs that are associated with host adaptation
Table_S5_enriched_GO_summary.csv - Supplemental table S5, the enriched gene on tology (GO) terms based on the significanly expanding/contracting HOGs
genome_assembling_of_gelechiid_moths-main.zip - Archival copy of the GitHub re pository https://github.com/yimingweng/genome_assembling_of_gelechiid_moths, dow nloaded 5-Oct-2023. The genome assembly project for the Tomato Pinworm, Keiferia lycopersicella (Walsingham), Phthorimaea absoluta (Tuta absoluta, tomato leafmi ner), and Scrobipalpa atriplicella (goosefoot groundling moth). Released under a MIT license. Please see the current GitHub repository for the most recent updat es.
[License]
All files and data are distributed under the CC0 1.0 Universal (CC0 1.0) Public
Domain Dedication (https://creativecommons.org/publicdomain/zero/1.0/), unless
specifically stated otherwise, see http://gigadb.org/site/term for more details.
[Comments]
[End]
Done with creating the README file for 102462. The README file is saved in file: /home/centos/uploadLogs/readme-102462.txt
All postUpload logs have been moved to: /home/chrish/uploadDir
PostUpload jobs done!
kencho51
rija
User story
Acceptance criteria
Additional Info
Note- the onscreen output of the postUpload script says that it has completed, but when you look in the database on beta none of the sizes or md5sums have been added.
PR #1248 says that its added the file sizes to the postUpload script, but its mot working on beta so maybe there is just some issue with getting that deployed on beta?
Product Backlog Item Ready Checklist
Product Backlog Item Done Checklist