Automate upload to GISAID, Github, Google Drive, SD County

[kramesh1]

Need to automate the upload of files to these different sources

GISAID

• Need to fix header errors Github Google Cloud
• No need to wait for analysis to be complete to start this. In a fully automated pipeline, Server can start uploading these files right away NCBI
• To to solve backlog of uploads and batch our current set SD County
• Need to autogenerate log file to send to SD county so that they can get sequence data to the state. Should be straightforward from log that is generated.

Will review each of these portions with Mark and Karthik G. before taking out branch against them.

[kramesh1]

Edit: Separating out the upload pipe for NCBI into its own issue since that infrastructure has not been built at all vs. the locations listed here all have existing pipelines that could just use increased streamlining

[kramesh1]

Specific todo list here of upload features

• [ ] Automate upload to GISAID via their API
• [ ] Automate upload to google drive using gsutils
• [ ] Automate upload to github via manual upload, regeneration of readme, and push of that repo

[kramesh1]

Laying out the structure here to be built in a branch:

From any location, a user initiates a bjorn_utils command to "upload". This upload command does 3 things:

1. Copies the sequences, bam files, and metadata to /alfheim/hcov-19-genomics/
2. Uploads the data to gisaid from alfheim using the following located in /code, with the save out locations being in hcov-19-genomics as well

./gisaid_uploader CoV upload --fasta /asgard/2021-10-25_release/msa/consensus_sequences/2021-10-25_release_unaligned_combined.fa --csv /asgard/2021-10-25_release/gisaid_metadata.csv --failedout /asgard/2021-10-25_release/gisaid_failed_metadata.csv

3. Uploads the bam files and sequences to google drive via gsutils
4. Transfers the sequences and relevant metadata to the HCoV-19-Genomics repo, updates the readme, and uploads them