Genome Annotation with Apollo

nathandunn commented 4 years ago

[x] merge Apollo tutorial into Refining Annotations with Apollo: http://localhost:4000/training-material/topics/genome-annotation/tutorials/apollo/tutorial.html to -> http://localhost:4000/training-material/topics/genome-annotation/tutorials/annotation-with-apollo/tutorial.html
[x] configure slides in main menu with intro slides! or just copy to slides.html locally
[x] finish create refinements, etc.
[ ] add lots of screenshots

abretaud commented 4 years ago

I've (finally) collected data for eukaryote annotation, it's there: https://usegalaxy.eu/u/abretaud/h/apollo-eukaryote-data For some reason, the jbrowse dataset is broken, do you have an idea why @hexylena?

It's based on data from https://training.galaxyproject.org/training-material/topics/genome-annotation/tutorials/annotation-with-maker/tutorial.html : it's the same genome and it uses the result of this tutorial (maker output)

Using SRA data, there's also some rnaseq (only kept region NC_003421.2:400000-600000 to have a small file, don't remember why I chose this one), a few variants (called with freebayes), and some NR proteins and transcripts alignments (aligned with exonerate).

hexylena commented 4 years ago

It's missing this file, not sure why.

https://usegalaxy.eu/datasets/11ac94870d0bb33a6cd00f017f1a801f/display/dist/main.bundle.js

hexylena commented 4 years ago

Oh, the html files directory is missing for that dataset. How odd!

hexylena commented 4 years ago

@bgruening @gmauro could you move jbrowse back from a pulsar machine? It's not compatible with pulsar as reported by @Slugger70 a while ago.

abretaud commented 4 years ago

I just realised I uploaded a wrong gff file, I just fixed it in the history

nathandunn commented 4 years ago

Excited to see all this progress. Let me know if there is anything I can do.

abretaud commented 2 years ago

Some news here, finally!

I have rebased everything (kept a copy of @nathandunn html slides locally, but removed them from the pr as it introduced too much noise in the diff, I'll look into them later I guess)
reduced a bit the data (removed unmapped reads from bams, reduced the number of blast hits)
pushed it to zenodo

I will continue working on it the coming days.

For some reasons I can't make the rule based uploader work by following the upload step, maybe there's something wrong in the file list

hexylena commented 2 years ago

awesome @abretaud!

abretaud commented 2 years ago

Interesting, uploading the bam files from the classic upload box works, but uploading them with rule-based uploaded fails. I see something like this in the logs on our server:

galaxy.metadata DEBUG 2021-05-27 10:00:17,028 [pN:main.job-handlers.2,p:11737,w:0,m:2,tN:SlurmRunner.work_thread-1] setting metadata externally failed for HistoryDatasetAssociation 2833260: Traceback (most recent call last):
  File "/blabla/lib/galaxy/metadata/set_metadata.py", line 238, in set_metadata_portable
    set_meta(dataset, file_dict)
  File "/blabla/lib/galaxy/metadata/set_metadata.py", line 132, in set_meta
    set_meta_with_tool_provided(new_dataset_instance, file_dict, set_meta_kwds, datatypes_registry, max_metadata_value_size)
  File "/blabla/lib/galaxy/metadata/set_metadata.py", line 96, in set_meta_with_tool_provided
    dataset_instance.datatype.set_meta(dataset_instance, **set_meta_kwds)
  File "/blabla/lib/galaxy/datatypes/binary.py", line 521, in set_meta
    pysam.index(dataset.file_name, index_file.file_name)
  File "/blabla/.venv/lib/python3.6/site-packages/pysam/utils.py", line 75, in __call__
    stderr))
pysam.utils.SamtoolsError: 'samtools returned with error 1: stdout=, stderr=samtools index: "/blabla/database/jobs_directory/000/225/225810/outputs/galaxy_dataset_ce766fc4-0a3c-4650-b223-6072b9ea3545.dat" is in a format that cannot be usefully indexed\n'

Looks like the bam file is empty/not where it should be!? Strange... I'll fill a galaxy issue later I guess

abretaud commented 2 years ago

Oops, looks like https://github.com/TAMU-CPT/blastview/ is broken with latest Apollo, there's no menu to create user-created annotation from features... I guess adding the track type to the list of types to transform could fix it... I'll test that I suppose

abretaud commented 2 years ago

@hexylena would it be hard to push this patch on usegalaxy.eu/apollo ? https://github.com/GMOD/Apollo/pull/2621 I tested on apollo.genouest.org, it works fine

abretaud commented 2 years ago

Most of it should be ready for a review now, would you have time @hexylena? There's still the "collaborating" part -> which way do we want to advertise on usegalaxy.eu? Using the "Register Account with Apollo" tool and sending credentials to collaborators?

hexylena commented 2 years ago

Will look at, this looks amazing.

I'm not sure what to do with collab, credential sharing is not optimal. What was the progress on the self-permission manager thing y'all wrote? Is that something I should deploy that will provide that?

abretaud commented 2 years ago

Hum, https://github.com/genouest/permapol/ is working IIRC... But I wanted to change a few things to make it more usable. I'll have a look this week and see if it can be ready for usegalaxy.eu. Would it be ok to deploy it with the docker container?

hexylena commented 2 years ago

yep, absolutely

hexylena commented 2 years ago

And for training, I guess we could put them all on one organism, not sure how to accomplish that though, that everyone is added to one org whenever they join

hexylena commented 2 years ago

Oh, I guess by using permapol to have them join a pre-shared org.

abretaud commented 2 years ago

Ok, I think the tutorial is complete now, made a few permapol screenshots from my local setup. Any review welcome of course!

Also added @mboudet as contributor as he made the greatest part of permapol

Before merging, I'd like to wait (tomorrow?) for the new version of permapol to be built/deployed, maybe in the meantime I'll have time to make a few slides

Almost done, it feels good :)

hexylena commented 2 years ago

Sounds good, I'm happy with the new portions, let me know when I should update permapol.

abretaud commented 2 years ago

Ah yes, @mboudet added himself to CONTRIBUTORS.yaml in https://github.com/galaxyproject/training-material/pull/2553, I'll resolve the conflict

permapol is rebuilt now, you can update it

abretaud commented 2 years ago

ah, you're faster than me :)

hexylena commented 2 years ago

https://usegalaxy.eu/apollo-permapol/ redeployed, does that look right?

abretaud commented 2 years ago

Looks exactly as intended thanks :)

nathandunn commented 2 years ago

Awesome!

galaxyproject / training-material

Genome Annotation with Apollo #1484