mpoelchau commented 3 years ago

Manual annotations

[x] Manual annotation QC
[x] OGS generation

OGS remapping

[x] request whole-genome alignment
[x] run remap-gff3 on OGS

Data management

[x] submit OGS to NCBI
[x] submit OGS to ADC
[x] submit manual annotations on old assembly version to ADC

Content deletion and addition steps

[x] Run final-workflow.cwl on docker container - use the localdata branch (no RefSeq assembly available).
- Data locations: see below
- Docker needed 14Gb allocation for bigwig step
[x] Apollo2-stage: delete existing Apollo2 Ladona fulva organism (using UI, or web services, or apollo python library). Apollo2 stage URL: https://apollo2-stage-node1.nal.usda.gov/apollo/jbrowse
[x] Pull processed data to Apollo2-stage
[x] run movedata.cwl workflow as self user
[x] Apollo2-stage: Add new Apollo2 data using createOrganism.py or createOrganism.cwl (parses YAML file)
[x] Genomics-workspace-stage: use CLI to add new data. Change jbrowse settings to new assembly. Keep old blast files for now.
[x] Test, if all works push to production
[x] Tripal: add new genome assembly and annotations analysis pages, link to Organism page. Don't delete old assembly and annotation pages
[x] deprecate old Apollo1 war and change link in 'available genome browsers' page (@mpoelchau)
[x] data downloads setup (use create-symlink.cwl)
[x] Final Test List
[x] Advertising

Data location:

genome assembly: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/376/725/GCA_000376725.2_Lful_2.0/GCA_000376725.2_Lful_2.0_genomic.fna.gz
genome annotations: https://data.nal.usda.gov/dataset/ladona-fulva-official-gene-set-ladfulogsv10

mpoelchau commented 3 years ago

@suryasaha the localdata branch of the Organism_onboarding repo now works for files that aren't retrieved from a URL. I need to add documentation, but you can use the file final_workflow_short.yml as a template. It adds the full path for the genome, gff, and other fasta files.

mpoelchau commented 3 years ago

@suryasaha can you remind me where this is at?

suryasaha commented 2 years ago

Add fasta

(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addblast Ladona fulva -t nucleotide Genome Assembly -f /usr/local/i5k/media/blast/db/GCA_000376725.2_Lful_2.0_genomic.fna -d 'Ladona fulva genome assembly, Lful_2.0'
you can move to makeblastdb and populate sequence step
(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addblast Ladona fulva -t nucleotide Transcript -f /usr/local/i5k/media/blast/db/ladful_OGSv1.0_trans.fa -d 'Ladona fulva Official Gene Set ladful OGSv1.0, transcripts'
you can move to makeblastdb and populate sequence step
(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addblast Ladona fulva -t peptide Protein -f /usr/local/i5k/media/blast/db/ladful_OGSv1.0_pep.fa -d 'Ladona fulva Official Gene Set ladful OGSv1.0, translated CDS'
you can move to makeblastdb and populate sequence step
(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addblast Ladona fulva -t peptide Protein -f /usr/local/i5k/media/blast/db/ladful_OGSv1.0_cds.fa -d 'Ladona fulva Official Gene Set ladful OGSv1.0, CDS'
you can move to makeblastdb and populate sequence step

Make blast DBs and populate

python manage.py blast_utility /usr/local/i5k/media/blast/db/GCA_000376725.2_Lful_2.0_genomic.fna -m
python manage.py blast_utility /usr/local/i5k/media/blast/db/GCA_000376725.2_Lful_2.0_genomic.fna -p

python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_trans.fa -m
python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_trans.fa -p

python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_pep.fa -m
python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_pep.fa -p

python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_cds.fa -m
python manage.py blast_utility /usr/local/i5k/media/blast/db/ladful_OGSv1.0_cds.fa -p

Make visible

python manage.py blast_shown /usr/local/i5k/media/blast/db/GCA_000376725.2_Lful_2.0_genomic.fna --shown true

python manage.py blast_shown /usr/local/i5k/media/blast/db/ladful_OGSv1.0_trans.fa --shown true

python manage.py blast_shown /usr/local/i5k/media/blast/db/ladful_OGSv1.0_pep.fa --shown true

python manage.py blast_shown /usr/local/i5k/media/blast/db/ladful_OGSv1.0_cds.fa --shown true

Add genome assembly fasta file to jbrowse

(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addjbrowse /usr/local/i5k/media/blast/db/GCA_000376725.2_Lful_2.0_genomic.fna https://apollo.nal.usda.gov/apollo/Ladona%20fulva/jbrowse/
JBrowseSetting Successfully Created.

Set up hmmer for peptide fasta fila:

(i5k) [i5k@i5k-stage-node1 ~]$ python manage.py addhmmer Ladona fulva -f /usr/local/i5k/media/blast/db/ladful_OGSv1.0_pep.fa -d "Ladona fulva Official Gene Set ladful OGSv1.0, translated CDS"
Success

suryasaha commented 2 years ago

BLAST Protein - Ladona fulva Official Gene Set ladful OGSv1.0, CDS shoud be Nucleotide - Ladona fulva Official Gene Set ladful OGSv1.0, CDS

This needs a fix @mpoelchau Sorry for the error!

suryasaha commented 2 years ago

https://apollo2-stage-node1.nal.usda.gov/apollo/4517586/jbrowse/index.html

Congratulations, JBrowse is on the web!
However, JBrowse could not start, either because it has not yet been configured      and loaded with data, or because of an error.
Error message(s):
Failed to load resource: ColorByType/View/Track/ColorByTypeDraggable.

suryasaha commented 2 years ago

https://apollo.nal.usda.gov/apollo/jbrowse still has old genome

suryasaha commented 2 years ago

Does this need a new annotation page? Already have https://data.nal.usda.gov/dataset/ladona-fulva-official-gene-set-ladfulogsv10

Assembly analysis page https://i5k.nal.usda.gov/bio_data/954997

mpoelchau commented 2 years ago

Yes, it needs a new annotation page. I'll set it up.

mpoelchau commented 2 years ago

Annotation page done: https://i5k.nal.usda.gov/bio_data/954998

mpoelchau commented 2 years ago

@suryasaha I just removed Ladona from prod apollo - the new one is good to load.

suryasaha commented 2 years ago