Open only1chunts opened 7 months ago
Tested on production staging:
% ssh -i output/privkeys-18.166.89.134/ken ken@18.166.89.134
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Mon Feb 5 03:10:38 2024 from 223.197.187.121
[ken@ip-10-99-0-147 ~]$ sudo /home/centos/postUpload.sh 100006
* About to update files' size for 100006
nb. changes: 7
Done with updating files' size for 100006. Nb of successful changes saved in file: /home/centos/uploadLogs/updating-file-size-100006.txt
* About to check that file urls are valid for 100006
| URL | Issue |
Done with checking that file urls are valid for 100006. Invalid Urls (if any) are save in file: /home/centos/uploadLogs/invalid-urls-100006.txt
* About to update files' MD5 Checksum as file attribute for 100006
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/100006/100006.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/100006/100006.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/100006/100006.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/100006/100006.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/100006/100006.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/100006/100006.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/101001_102000/100006/100006.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/101001_102000/100006/100006.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/100001_101000/100006/100006.md5
Saved md5 file attribute with id: 10674
Saved md5 file attribute with id: 10672
Saved md5 file attribute with id: 10673
Saved md5 file attribute with id: 10671
Saved md5 file attribute with id: 10670
Saved md5 file attribute with id: 10669
Saved md5 file attribute with id: 10675
Done with updating files' MD5 Checksum as file attribute for 100006. Process status is saved in file: /home/centos/uploadLogs/updating-md5checksum-100006.txt
* About to create the README file for 100006
[DOI] 10.5524/100006
[Title] Genomic data from Adelie penguin (<em>Pygoscelis adeliae</em>).
[Release Date] 2011-07-06
[Citation] Wang, J; Zhang, G; Lambert, DM (2011): Genomic data from Adelie
penguin (<em>Pygoscelis adeliae</em>). GigaScience Database.
https://dx.doi.org/10.5524/100006
[Data Type] Genomic
[Data Summary] The Adelie penguin (<em>Pygoscelis adeliae</em>) is an iconic
penguin of moderate stature and a tuxedo of black and white feathers. The
penguins are only found in the Antarctic region and surrounding islands. Being
very sensitive to climate change, and due to changes in their behavior based on
minor shifts in climate, they are often used as a barometer of the Antarctic.
<br>
With its status as one of the adorable and cuddly flightless birds of
Antarctica, they serve as an example for conservation, and as a result they are
now categorised at low risk for endangerment. The sequence of the penguin can
be of use in understanding the genetic underpinnings of its evolutionary traits
and adaptation to its extreme environment; its unique system of feathers; its
prowess as a diver; and its sensitivity to climate change. We hope that this
genome data will further our understanding of one of the most remarkable
creatures to waddle the planet Earth.<br> We sequenced the genome of an adult
male from Inexpressible Island, Ross Sea, Antartica (provided by David Lambert)
to a depth of approximately 60X with short reads from a series of libraries with
various insert sizes (200bp- 20kb).
The assembled scaffolds of high quality sequences total 1.23 Gb, with the contig
and scaffold N50 values of 19 kb and 5 mb respectively. We identified 15,270
protein-coding genes with a mean length of 21.3 kb.
[File Location] https://ftp.cngb.org/pub/gigadb/pub/10.5524/100001_101000/100006/
[File name] - [File Description]
readme.txt -
Pygoscelis_adeliae.fa.gz - assembled scaffolds from sequence data
Pygoscelis_adeliae.gff.gz - coding sequence annotation of assembly
Pygoscelis_adeliae.RepeatMasker.out.gz - repeat masker results
Pygoscelis_adeliae.cds.gz - coding sequence predictions on genome assembly
Pygoscelis_adeliae.scaf.fa.gz -
Pygoscelis_adeliae.pep.gz - peptide translations of CDS predictions
[License]
All files and data are distributed under the CC0 1.0 Universal (CC0 1.0) Public
Domain Dedication (https://creativecommons.org/publicdomain/zero/1.0/), unless
specifically stated otherwise, see http://gigadb.org/site/term for more details.
[Comments]
[End]
Done with creating the README file for 100006. The README file is saved in file: /home/centos/uploadLogs/readme-100006.txt
All postUpload logs have been moved to: /home/ken/uploadDir
PostUpload jobs done!
[ken@ip-10-99-0-147 ~]$
TODO: 1) Test on bastion using a more recent dataset spreadsheet 2) whatever the outcome, going forward, the MD5 Checksum file will be uploaded to Wasabi (manually for now) to avoid issuer CNGB FTP server, which means the postUpload script will need a change to look for the MD5 files on Wasabi
- Test on bastion using a more recent dataset spreadsheet
% ssh -i output/privkeys-18.166.89.134/ken ken@18.166.89.134
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Mon Feb 5 04:03:43 2024 from 223.197.187.121
[ken@ip-10-99-0-147 ~]$ ls
uploadDir
[ken@ip-10-99-0-147 ~]$ sudo /home/centos/postUpload.sh 102487
* About to update files' size for 102487
nb. changes: 6
Done with updating files' size for 102487. Nb of successful changes saved in file: /home/centos/uploadLogs/updating-file-size-102487.txt
* About to check that file urls are valid for 102487
| URL | Issue |
Done with checking that file urls are valid for 102487. Invalid Urls (if any) are save in file: /home/centos/uploadLogs/invalid-urls-102487.txt
* About to update files' MD5 Checksum as file attribute for 102487
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/102487/102487.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/102487/102487.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/102487/102487.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/102487/102487.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/102487/102487.md5
Saved md5 file attribute with id: 674793
Saved md5 file attribute with id: 674794
Saved md5 file attribute with id: 674795
Saved md5 file attribute with id: 674796
Saved md5 file attribute with id: 674797
Saved md5 file attribute with id: 674836
Done with updating files' MD5 Checksum as file attribute for 102487. Process status is saved in file: /home/centos/uploadLogs/updating-md5checksum-102487.txt
* About to create the README file for 102487
[DOI] 10.5524/102487
[Title] Supporting data for "ToxCodAn-Genome: an automated pipeline for
toxin-gene annotation in genome assembly of venomous lineages"
[Release Date] 2023-12-14
[Citation] Durham, AM; Nachtigall, PG; Rokyta, DR; Junqueira-de-Azevedo, IL
(2023): Supporting data for "ToxCodAn-Genome: an automated pipeline for
toxin-gene annotation in genome assembly of venomous lineages" GigaScience
Database. https://dx.doi.org/10.5524/102487
[Data Type] Software,Genomic
[Data Summary] The rapid development of sequencing technologies resulted in a
wide expansion of genomics studies using venomous lineages. This facilitated
research focusing on understanding the evolution of adaptive traits and the
search for novel compounds that can be applied in agriculture and medicine.
However, the toxin annotation of genomes is a laborious and time-consuming task,
and no consensus pipeline is currently available. No computational tool
currently exists to address the challenges specific to toxin annotation and to
ensure the reproducibility of the process. Here, we present ToxCodAn-Genome, the
first software designed to perform automated toxin annotation in genomes of
venomous lineages. This pipeline was designed to retrieve the full-length coding
sequences of toxins and to allow the detection of novel truncated paralogs and
pseudogenes. We tested ToxCodAn-Genome using 12 genomes of venomous lineages and
achieved high performance on recovering their current toxin annotations. This
tool can be easily customized to allow improvements in the final toxin
annotation set and can be expanded to virtually any venomous lineage.
ToxCodAn-Genome is fast, allowing it to run on any personal computer, but it can
also be executed in multi-core mode, taking advantage of large high-performance
servers. In addition, we provide a guide to direct future research in the
venomics field to ensure a confident toxin annotation in the genome being
studied. As a case study, we sequenced and annotated the toxin repertoire of
<i>Bothrops alternatus</i>, which may facilitate future evolutionary and
biomedical studies using vipers as models.ToxCodAn-Genome and the guide are
freely available at <a href="https://github.com/pedronachtigall/ToxCodAn-Genome"
target="_blank">https://github.com/pedronachtigall/ToxCodAn-Genome</a>.
[File Location] https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/102487/
[File name] - [File Description]
readme_102487.txt -
ToxCodAn-master.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxCodAn downloaded 01-Dec-2023. ToxCodAn - a computational tool designed to detect and annotate toxin genes in transcriptome assembly. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
ToxCodAn-Genome-main.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxCodAn-Genome downloaded 01-Dec-2023. ToxCodAn-Genome - a computational tool designed to annotate toxin genes in genomes of venomous lineages. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
CTL_alphabeta_ALIGNED.fasta.contree - Phylogenetic tree file used for phylogenetic analysis.
CTL_alphabeta_ALIGNED.fasta - Multiple-sequence alignment file used for phylogenetic analysis.
ToxcodanGenomeFigures-main.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxcodanGenomeFigures downloaded 01-Dec-2023. ToxcodanGenomeFigures - repository containing the input files and code to generate the figures in ToxCodAn-Genome manuscript. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
[License]
All files and data are distributed under the CC0 1.0 Universal (CC0 1.0) Public
Domain Dedication (https://creativecommons.org/publicdomain/zero/1.0/), unless
specifically stated otherwise, see http://gigadb.org/site/term for more details.
[Comments]
[End]
Done with creating the README file for 102487. The README file is saved in file: /home/centos/uploadLogs/readme-102487.txt
All postUpload logs have been moved to: /home/ken/uploadDir
PostUpload jobs done!
[ken@ip-10-99-0-147 ~]$
% ssh -i output/privkeys-18.167.104.80/ken ken@18.167.104.80
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Tue Feb 6 02:02:24 2024 from 223.197.187.121
[ken@ip-10-99-0-90 ~]$ ls
uploadDir
[ken@ip-10-99-0-90 ~]$ sudo /home/centos/postUpload.sh 102487
* About to update files' size for 102487
Exception 'yii\db\Exception' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php:648
Error Info:
Array
(
[0] => 08006
[1] => 7
[2] => timeout expired
)
Caused by: Exception 'PDOException' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php:722
Stack trace:
#0 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(722): PDO->__construct('pgsql:host=rds-...', 'gigadb', Object(SensitiveParameterValue), NULL)
#1 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(637): yii\db\Connection->createPdoInstance()
#2 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(1067): yii\db\Connection->open()
#3 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(1054): yii\db\Connection->getMasterPdo()
#4 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Schema.php(461): yii\db\Connection->getSlavePdo()
#5 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(946): yii\db\Schema->quoteValue('102487')
#6 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(211): yii\db\Connection->quoteValue('102487')
#7 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1126): yii\db\Command->getRawSql()
#8 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1147): yii\db\Command->logQuery('yii\\db\\Command:...')
#9 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(424): yii\db\Command->queryInternal('fetch', NULL)
#10 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Query.php(287): yii\db\Command->queryOne()
#11 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/ActiveQuery.php(304): yii\db\Query->one(NULL)
#12 /gigadb/app/tools/files-metadata-console/components/DatasetFilesUpdater.php(50): yii\db\ActiveQuery->one()
#13 /gigadb/app/tools/files-metadata-console/controllers/UpdateController.php(35): app\components\DatasetFilesUpdater->updateFileSize()
#14 [internal function]: app\controllers\UpdateController->actionFileSize()
#15 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#16 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#17 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Controller.php(180): yii\base\Controller->runAction('file-size', Array)
#18 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Module.php(552): yii\console\Controller->runAction('file-size', Array)
#19 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('update/file-siz...', Array)
#20 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('update/file-siz...', Array)
#21 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application->handleRequest(Object(yii\console\Request))
#22 /gigadb/app/tools/files-metadata-console/yii(20): yii\base\Application->run()
#23 {main}
Done with updating files' size for 102487. Nb of successful changes saved in file: /home/centos/uploadLogs/updating-file-size-102487.txt
* About to check that file urls are valid for 102487
Exception 'yii\db\Exception' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php:648
Error Info:
Array
(
[0] => 08006
[1] => 7
[2] => timeout expired
)
Caused by: Exception 'PDOException' with message 'SQLSTATE[08006] [7] timeout expired'
in /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php:722
Stack trace:
#0 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(722): PDO->__construct('pgsql:host=rds-...', 'gigadb', Object(SensitiveParameterValue), NULL)
#1 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(637): yii\db\Connection->createPdoInstance()
#2 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(1067): yii\db\Connection->open()
#3 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(1054): yii\db\Connection->getMasterPdo()
#4 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Schema.php(461): yii\db\Connection->getSlavePdo()
#5 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Connection.php(946): yii\db\Schema->quoteValue('102487')
#6 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(211): yii\db\Connection->quoteValue('102487')
#7 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1126): yii\db\Command->getRawSql()
#8 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(1147): yii\db\Command->logQuery('yii\\db\\Command:...')
#9 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Command.php(424): yii\db\Command->queryInternal('fetch', NULL)
#10 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/Query.php(287): yii\db\Command->queryOne()
#11 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/db/ActiveQuery.php(304): yii\db\Query->one(NULL)
#12 /gigadb/app/tools/files-metadata-console/components/FilesURLsFetcher.php(35): yii\db\ActiveQuery->one()
#13 /gigadb/app/tools/files-metadata-console/controllers/CheckController.php(37): app\components\FilesURLsFetcher->verifyURLs()
#14 [internal function]: app\controllers\CheckController->actionValidUrls()
#15 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#16 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#17 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Controller.php(180): yii\base\Controller->runAction('valid-urls', Array)
#18 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Module.php(552): yii\console\Controller->runAction('valid-urls', Array)
#19 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('check/valid-url...', Array)
#20 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/console/Application.php(147): yii\console\Application->runAction('check/valid-url...', Array)
#21 /gigadb/app/tools/files-metadata-console/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application->handleRequest(Object(yii\console\Request))
#22 /gigadb/app/tools/files-metadata-console/yii(20): yii\base\Application->run()
#23 {main}
Done with checking that file urls are valid for 102487. Invalid Urls (if any) are save in file: /home/centos/uploadLogs/invalid-urls-102487.txt
* About to update files' MD5 Checksum as file attribute for 102487
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/102487/102487.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/104001_105000/102487/102487.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/102487/102487.md5
No file found at https://ftp.cngb.org/pub/gigadb/pub/10.5524/103001_104000/102487/102487.md5
Processing https://ftp.cngb.org/pub/gigadb/pub/10.5524/102001_103000/102487/102487.md5
Saved md5 file attribute with id: 674793
Saved md5 file attribute with id: 674794
Saved md5 file attribute with id: 674795
Saved md5 file attribute with id: 674796
Saved md5 file attribute with id: 674797
Saved md5 file attribute with id: 674836
Done with updating files' MD5 Checksum as file attribute for 102487. Process status is saved in file: /home/centos/uploadLogs/updating-md5checksum-102487.txt
* About to create the README file for 102487
[DOI] 10.5524/102487
[Title] Supporting data for "ToxCodAn-Genome: an automated pipeline for
toxin-gene annotation in genome assembly of venomous lineages"
[Release Date] 2023-12-14
[Citation] Durham, AM; Nachtigall, PG; Rokyta, DR; Junqueira-de-Azevedo, IL
(2023): Supporting data for "ToxCodAn-Genome: an automated pipeline for
toxin-gene annotation in genome assembly of venomous lineages" GigaScience
Database. https://dx.doi.org/10.5524/102487
[Data Type] Software,Genomic
[Data Summary] The rapid development of sequencing technologies resulted in a
wide expansion of genomics studies using venomous lineages. This facilitated
research focusing on understanding the evolution of adaptive traits and the
search for novel compounds that can be applied in agriculture and medicine.
However, the toxin annotation of genomes is a laborious and time-consuming task,
and no consensus pipeline is currently available. No computational tool
currently exists to address the challenges specific to toxin annotation and to
ensure the reproducibility of the process. Here, we present ToxCodAn-Genome, the
first software designed to perform automated toxin annotation in genomes of
venomous lineages. This pipeline was designed to retrieve the full-length coding
sequences of toxins and to allow the detection of novel truncated paralogs and
pseudogenes. We tested ToxCodAn-Genome using 12 genomes of venomous lineages and
achieved high performance on recovering their current toxin annotations. This
tool can be easily customized to allow improvements in the final toxin
annotation set and can be expanded to virtually any venomous lineage.
ToxCodAn-Genome is fast, allowing it to run on any personal computer, but it can
also be executed in multi-core mode, taking advantage of large high-performance
servers. In addition, we provide a guide to direct future research in the
venomics field to ensure a confident toxin annotation in the genome being
studied. As a case study, we sequenced and annotated the toxin repertoire of
<i>Bothrops alternatus</i>, which may facilitate future evolutionary and
biomedical studies using vipers as models.ToxCodAn-Genome and the guide are
freely available at <a href="https://github.com/pedronachtigall/ToxCodAn-Genome"
target="_blank">https://github.com/pedronachtigall/ToxCodAn-Genome</a>.
[File Location] https://s3.ap-northeast-1.wasabisys.com/gigadb-datasets/live/pub/10.5524/102001_103000/102487/
[File name] - [File Description]
ToxcodanGenomeFigures-main.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxcodanGenomeFigures downloaded 01-Dec-2023. ToxcodanGenomeFigures - repository containing the input files and code to generate the figures in ToxCodAn-Genome manuscript. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
CTL_alphabeta_ALIGNED.fasta - Multiple-sequence alignment file used for phylogenetic analysis.
ToxCodAn-Genome-main.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxCodAn-Genome downloaded 01-Dec-2023. ToxCodAn-Genome - a computational tool designed to annotate toxin genes in genomes of venomous lineages. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
readme_102487.txt -
CTL_alphabeta_ALIGNED.fasta.contree - Phylogenetic tree file used for phylogenetic analysis.
ToxCodAn-master.zip - Archival copy of the GitHub repository https://github.com/pedronachtigall/ToxCodAn downloaded 01-Dec-2023. ToxCodAn - a computational tool designed to detect and annotate toxin genes in transcriptome assembly. This project is licensed under the GPL3.0 license. Please refer to the GitHub repository for most recent updates.
[License]
All files and data are distributed under the CC0 1.0 Universal (CC0 1.0) Public
Domain Dedication (https://creativecommons.org/publicdomain/zero/1.0/), unless
specifically stated otherwise, see http://gigadb.org/site/term for more details.
[Comments]
[End]
Done with creating the README file for 102487. The README file is saved in file: /home/centos/uploadLogs/readme-102487.txt
All postUpload logs have been moved to: /home/ken/uploadDir
PostUpload jobs done!
[ken@ip-10-99-0-90 ~]$
This postUpload.sh
script uses docker-compose run --rm tool /app/yii readme/create --doi "$DOI" | tee "$outputDir/readme-$DOI.txt"
to create readme files but does not upload them to Wasabi.
To create readme files and upload them to Wasabi, we need to use:
$ ./createReadme.sh --doi 100142 --outdir /app/readmeFiles --wasabi --use-live-data --apply
@rija @only1chunts As well as creating readme files, other functionalities that this postUpload.sh
script performs are:
All three functionalities currently rely on dataset files and the doi.md5 file being located in GigaDB's public dataset directories. Therefore, this postUpload.sh
script only works when a dataset is published and has been made public.
We can update md5 file attributes before dataset publication if I have access to the doi.md5 file. The other two functionalities require dataset files to be made public.
The difficulties that have been previously observed when checking for broken dataset file URLs and updating file sizes using postUpload.sh may be related to the present file locations. Once files are moved to Wasabi then less likely these problems will occur.
User story
Acceptance criteria
Additional Info
When I ran the post upload on dataset 102495 today (1Feb2024) it updated some of the md5sum values but not all, and the error log is unclear to me why, and it didn't give an obvious error message, with the last line of the ouput just saying "PostUpload jobs done!" see below.
None of the file sizes were added to the database.
Command line output of postUpload script
Product Backlog Item Ready Checklist
Product Backlog Item Done Checklist