Proteogenomics workflow

[chambm]

Can someone (@PratikDJagtap) point me to the Galaxy-P proteogenomic workflow into which I should integrate my Omicron tools, e.g. CustomProDB and PSM2SAM? I checked the "Published workflows" section of the public Galaxy-P site and it's not there. We can discuss any design considerations for the fused workflow here.

I see there's a "Tool needed" label; it begs the question, why is there no "Workflow needed" label? Pinging @bgruening because I don't know who better to ask. ;)

[bgruening]

For me a workflow is just a higher level abstraction, based on tools. I added this label, but it would be nice to explicitly state which tools are needed :)

Thanks @chambm!!!

[jj-umn]

@chambm I generally build a workflow and test it in galaxy. All the tools in the workflow should be retrievable from the same toolshed, preferably https://toolshed.g2.bx.psu.edu To publish the workflow in the toolshed, it should have a repository_dependencies.xml that contains all the tool dependencies. @PratikDJagtap Perhaps you and Getiria can work on a demo workflow.

[chambm]

Silly me, I didn't even search the toolshed for workflows. Actually I've never installed a workflow from the toolshed before. :open_mouth: This looks like a good candidate? https://toolshed.g2.bx.psu.edu/view/galaxyp/proteomics_rnaseq_sap_db_workflow/3a11830963e3

[bgruening]

Or this one? https://github.com/bgruening/galaxytools/tree/master/workflows/glimmer3 We accepting PR here :)

[chambm]

This work is aimed at human (possibly mouse) data, so we don't need to infer annotations. That's a whole other bag of genes!

[PratikDJagtap]

Hello Matt and Bjoern,

I will get back to you with answers by evening today.

Regards, Pratik

[jj-umn]

Here's the workflows we used at GCC2016 tutorial: https://github.com/galaxyproteomics/tools-galaxyp/tree/master/workflows/gcc2016_tutorial

[bgruening]

@chambm this was just an example of workflows in git and github with example data and so on :) It's all on the TS as well.

[PratikDJagtap]

Hello Matt,

I will work on workflow components from the workflow that we generated for GCC2016 (mentioned by JJ below). JJ, we can also look at workflows from ABRF2016.

https://github.com/galaxyproteomics/tools-galaxyp/tree/master/workflows/ gcc2016_tutorial

We will need a few hours to come up with a hybrid workflow after discussion with the Galaxy-P team. If required, it will also be a good idea to have a telephone / google hangouts session.

Regards, Pratik

[chambm]

This is a well-written tutorial! https://netfiles.umn.edu/users/pjagtap/ABRF%202016/ABRF_2016_SW4_Galaxy_for_Multi-Omics.pdf

I look forward to hearing your ideas for a hybrid. I worry a bit when I look at the GCC workflow's complexity. Galaxy's job failure handling just isn't robust enough (yet) to handle failures in such a complex workflow. It was a pain to rerun subsets of the Omicron workflow when one step failed halfway through the workflow, and that workflow is about a tenth of the size.

[PratikDJagtap]

Hello Matt,

Yes - we will need to run these as sub-workflows which we did for the workshop and also suggest users to use when running it for their projects.

As you might be aware, Galaxy also offers ability to rerun the subsequent steps in case of a workflow failure (and once the failed job issue is taken care of) so that user need not start from the beginning.

Looking forward to a hybrid OMicron-GalaxyP workflow!

Regards,

Pratik

On Thu, Nov 3, 2016 at 11:18 AM, Matt Chambers notifications@github.com wrote:

This is a well-written tutorial! https://netfiles.umn.edu/ users/pjagtap/ABRF%202016/ABRF_2016_SW4_Galaxy_for_Multi-Omics.pdf

I look forward to hearing your ideas for a hybrid. I worry a bit when I look at the GCC workflow's complexity. Galaxy's job failure handling just isn't robust enough (yet) to handle failures in such a complex workflow. It was a pain to rerun subsets of the Omicron workflow when one step failed halfway through the workflow, and that workflow is about a tenth of the size.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70#issuecomment-258192722, or mute the thread https://github.com/notifications/unsubscribe-auth/AKL0OjkwPqFNOAaRbFmPvMpKW2vRgF7Nks5q6glmgaJpZM4KngwS .

[chambm]

I should clarify that it was failures in a dataset collection that caused the problem. Jobs on single input files could be easily rerun. But jobs where only a few files from a collection failed could not be rerun. Yet using collections reduces the history size by a factor of , e.g. 10-25 fewer history items.

[PratikDJagtap]

Interesting - dataset collection has worked well in our hands. It will be good to exchange notes as we proceed.

Pratik Jagtap,

[chambm]

Ping @tjgriff1

[bgruening]

Galaxy's job failure handling just isn't robust enough (yet) There is a fix in latest dev that makes this more robust if I recall correctly.

[chambm]

Pratik and Tim: the Omicron version of CustomProDB and PSM2SAM depend on a few Galaxy data managers to download the genome FASTA, index it with Bowtie, and download gene annotations from UCSC Table Browser (done inside the CustomProDB R script).

Do you want to keep this architecture or look at some alternate implementation? Currently, the user needs to run these data manager "workflows" first, and refresh the local data tables manually because Galaxy doesn't update it automatically yet. I can see either moving all this reference data into the user's history so it can be a proper part of the workflow, or doing these steps in some kind of initialization step.

The Omicron approach uses Docker to get a custom Galaxy flavor running instantly, but it can't include the reference data without making the Docker image gigantic. So I let the user download this reference data from within Galaxy using the data managers, rather than with an initialization script which runs when the Docker container runs. With the script alternative, it will take hours before the Galaxy flavor is running, but when it does run, it's directly ready for the workflow.

[PratikDJagtap]

Hello Matt,

I am copying @getiria-onsongo, @jj-umn, @tmcgowan on this so that they can comment on which of the approaches would work best - or if any alternative would help to make this process more user-friendly.

Regards, Pratik

On Mon, Nov 7, 2016 at 4:53 PM, Matt Chambers notifications@github.com wrote:

Pratik and Tim: the Omicron version of CustomProDB and PSM2SAM depend on a few Galaxy data managers to download the genome FASTA, index it with Bowtie, and download gene annotations from UCSC Table Browser (done inside the CustomProDB R script).

Do you want to keep this architecture or look at some alternate implementation? Currently, the user needs to run these data manager "workflows" first, and refresh the local data tables manually because Galaxy doesn't update it automatically yet. I can see either moving all this reference data into the user's history so it can be a proper part of the workflow, or doing these steps in some kind of initialization step.

The Omicron approach uses Docker to get a custom Galaxy flavor running instantly, but it can't include the reference data without making the Docker image gigantic. So I let the user download this reference data from within Galaxy using the data managers, rather than with an initialization script which runs when the Docker container runs. With the script alternative, it will take hours before the Galaxy flavor is running, but when it does run, it's directly ready for the workflow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70#issuecomment-258989632, or mute the thread https://github.com/notifications/unsubscribe-auth/AKL0Oq4-lB6VGaBMCH0C6nCWz5tcXvYOks5q76vWgaJpZM4KngwS .

[tjgriff1]

Hi - here's my thoughts. When using the Docker implementation, it seems reasonable to continue to do this the way you have in the past -- having the user download the reference data needed after the instance is up and running. It seems more reasonable to do it this way and keep the Docker image more agile. As long as we are able to document the process users need to follow to download the reference data and get it processed correctly for DB generation it should be an OK way to approach this.

I'm also envisioning that not all users will be dependent on the Docker image to gain access to these workflows. Some users may have their own Galaxy instance already running or be utilizing cloud infrastructure where instances are already in place (e.g. Jetstream). For these users, we can share the workflows they need to make the pipeline run. These workflows can include the steps needed to create the DB via CustomProDB.

In both cases it will be important to have the workflow documented so they know how and what they need to do to make it work.

Make sense?

• TG

On Mon, Nov 7, 2016 at 5:01 PM, Pratik Jagtap notifications@github.com wrote:

Hello Matt,

I am copying @getiria-onsongo, @jj-umn, @tmcgowan on this so that they can comment on which of the approaches would work best - or if any alternative would help to make this process more user-friendly.

Regards, Pratik

Pratik Jagtap,

Research Assistant Professor, Department of Biochemistry, Molecular Biology and Biophysics,

University of Minnesota.

Address: 7-166 MCB, 420 Washington Ave SE, Minneapolis, MN 55455

Phone: 612-624-0381 http://612-624-0381/ Email: pjagtap@umn.edu pjagtap@umn.edu

Twitter: twitter.com/pratikomics http://twitter.com/pratikomics Google Scholar: z.umn.edu/pjgs http://z.umn.edu/pjgs

http://z.umn.edu/galaxypreferences

On Mon, Nov 7, 2016 at 4:53 PM, Matt Chambers notifications@github.com wrote:

Pratik and Tim: the Omicron version of CustomProDB and PSM2SAM depend on a few Galaxy data managers to download the genome FASTA, index it with Bowtie, and download gene annotations from UCSC Table Browser (done inside the CustomProDB R script).

Do you want to keep this architecture or look at some alternate implementation? Currently, the user needs to run these data manager "workflows" first, and refresh the local data tables manually because Galaxy doesn't update it automatically yet. I can see either moving all this reference data into the user's history so it can be a proper part of the workflow, or doing these steps in some kind of initialization step.

The Omicron approach uses Docker to get a custom Galaxy flavor running instantly, but it can't include the reference data without making the Docker image gigantic. So I let the user download this reference data from within Galaxy using the data managers, rather than with an initialization script which runs when the Docker container runs. With the script alternative, it will take hours before the Galaxy flavor is running, but when it does run, it's directly ready for the workflow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70# issuecomment-258989632, or mute the thread https://github.com/notifications/unsubscribe-auth/AKL0Oq4- lB6VGaBMCH0C6nCWz5tcXvYOks5q76vWgaJpZM4KngwS .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70#issuecomment-258991527, or mute the thread https://github.com/notifications/unsubscribe-auth/AKMY3pakdI6_kczomRqlKIrKpIjtC5DQks5q7629gaJpZM4KngwS .

Tim Griffin, Ph.D. Professor, and Director, Center for Mass Spectrometry and Proteomics University of Minnesota Dept. of Biochemistry, Molecular Biology and Biophysics 6-155 Jackson Hall 321 Church Street SE Minneapolis, MN 55455 USA

Office: 7-144 Molecular Cellular Biology (MCB)

Tel: 612-624-5249 Fax: 612-624-0432 Email: tgriffin@umn.edu

https://www.cbs.umn.edu/bmbb/contacts/timothy-j-griffin Center for Mass Spectrometry and Proteomics website: http://www.cbs.umn.edu/msp/

[PratikDJagtap]

My preference would be for the tool to have the option to use prebuilt indices and Shared Data Libraries for reference data.

That would be the model that makes the most sense for institutions such as galaxy main or MSI that run persistent Galaxy instances.

Retaining the option to build these on the fly makes sense when one is using the docker model and only using galaxy as a workflow engine, rather than as a collaboration and data sharing environment.

JJ

[chambm]

Does Galaxy have the necessary data types for indices to be able to share them in data libraries?

[jj-umn]

No. We use Data managers for the index files: fai,2bit,bowtie,bwa,hisat,etc. which get added to the .loc files. We symlink the annotation files: GTF, VCF, etc. for those references in as Shared Data Libraries. These are admin tasks.

[tjgriff1]

Good point JJ - for the persistent instances it would be really good to have necessary files in the shared data library. Much easier on the user than downloading and/or uploading large files.

• TG

On Tue, Nov 8, 2016 at 9:45 AM, Pratik Jagtap notifications@github.com wrote:

On 11/7/16 5:01 PM, Pratik Jagtap wrote:

Hello Matt,

I am copying @getiria-onsongo, @jj-umn, @tmcgowan on this so that they can comment on which of the approaches would work best - or if any alternative would help to make this process more user-friendly.

Regards, Pratik

Pratik Jagtap,

Research Assistant Professor, Department of Biochemistry, Molecular Biology and Biophysics,

University of Minnesota.

/Address: 7-166 MCB, 420 Washington Ave SE, Minneapolis, MN 55455/

/_Phone:_612-624-0381 http://612-624-0381/*Email: *pjagtap@umn.edu mailto:pjagtap@umn.edu /

/Twitter: twitter.com/pratikomics http://twitter.com/pratikomics _Google Scholar:_z.umn.edu/pjgs http://z.umn.edu/pjgs/

http://z.umn.edu/galaxypreferences

On Mon, Nov 7, 2016 at 4:53 PM, Matt Chambers <notifications@github.com mailto:notifications@github.com> wrote:

Pratik and Tim: the Omicron version of CustomProDB and PSM2SAM depend on a few Galaxy data managers to download the genome FASTA, index it with Bowtie, and download gene annotations from UCSC Table Browser (done inside the CustomProDB R script).

Do you want to keep this architecture or look at some alternate implementation? Currently, the user needs to run these data manager "workflows" first, and refresh the local data tables manually because Galaxy doesn't update it automatically yet. I can see either moving all this reference data into the user's history so it can be a proper part of the workflow, or doing these steps in some kind of initialization step.

The Omicron approach uses Docker to get a custom Galaxy flavor running instantly, but it can't include the reference data without making the Docker image gigantic. So I let the user download this reference data from within Galaxy using the data managers, rather than with an initialization script which runs when the Docker container runs. With the script alternative, it will take hours before the Galaxy flavor is running, but when it does run, it's directly ready for the workflow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ galaxyproteomics/tools-galaxyp/issues/70#issuecomment-258989632, or mute the thread https://github.com/notifications/unsubscribe-auth/AKL0Oq4- lB6VGaBMCH0C6nCWz5tcXvYOks5q76vWgaJpZM4KngwS.

My preference would be for the tool to have the option to use prebuilt indices and Shared Data Libraries for reference data.

That would be the model that makes the most sense for institutions such as galaxy main or MSI that run persistent Galaxy instances.

Retaining the option to build these on the fly makes sense when one is using the docker model and only using galaxy as a workflow engine,

rather than as a collaboration and data sharing environment.

JJ

James E. Johnson Minnesota Supercomputing Institute University of Minnesota

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70#issuecomment-259172340, or mute the thread https://github.com/notifications/unsubscribe-auth/AKMY3jyiAmAwrYCUxcml-cT_jkiMFYG4ks5q8JkwgaJpZM4KngwS .

Tim Griffin, Ph.D. Professor, and Director, Center for Mass Spectrometry and Proteomics University of Minnesota Dept. of Biochemistry, Molecular Biology and Biophysics 6-155 Jackson Hall 321 Church Street SE Minneapolis, MN 55455 USA

Office: 7-144 Molecular Cellular Biology (MCB)

Tel: 612-624-5249 Fax: 612-624-0432 Email: tgriffin@umn.edu

https://www.cbs.umn.edu/bmbb/contacts/timothy-j-griffin Center for Mass Spectrometry and Proteomics website: http://www.cbs.umn.edu/msp/

[chambm]

How coupled is SearchGUI/PeptideShaker to the MGF format? If we could keep things as mzML or mz5, preserving nativeID would be a big advantage for tracing downstream ids back to their source spectra.

[PratikDJagtap]

Hello Matt,

Currently MGF is the only input format that SearchGUI seems to support. We can generate a github issue on the SearchGUI and PeptideShaker github site for this.

Regards, Pratik

[chambm]

@jj-umn In the GCC2016 workflow, there is a Select step filtering for ^\d+\tpr.B[^\t,]*(, pr.B[^\t,]*)*\t.*$ - This appears to be related to the Mouse pre-pro-B and pro-B FASTQ files, but the proteomic data I have is named like Mo_Tai_iTRAQ_f5. It looks like it's trying to filter on the protein column in the Peptide Shaker output, but my protein column looks like:

1 0, 1.503, 1.5436, 49.5419, 5.2189, 5.2784, 5.2833 GLLLYGPPGTGK 2 731.5188 SIYYITGESK

So I guess it has something to do with the input FASTA. Is it supposed to select only the non-reference accessions for each PSM?

[PratikDJagtap]

Hello Matt,

If this is a filtering step for the PSM Report output from PeptideShaker - then we used the accession numbers from protein FASTA file (pre-pro-B and pro-B) to parse out the peptides from RNAseq data. Do you have accession numbers associated with these peptide identifications?

@jj-umn https://github.com/jj-umn might be able to provide more information.

Regards, Pratik

Pratik Jagtap,

Research Assistant Professor, Department of Biochemistry, Molecular Biology and Biophysics, University of Minnesota.

Address: 7-166 MCB, 420 Washington Ave SE, Minneapolis, MN 55455

*Phone: 612-624-0381 http://612-624-0381/ Email: pjagtap@umn.edu pjagtap@umn.edu *Twitter: twitter.com/pratikomics http://twitter.com/pratikomics

Google Scholar: z.umn.edu/pjgs http://z.umn.edu/pjgs

How Are You – and How’s Your Microbiome? http://z.umn.edu/hayahym

On Tue, Dec 13, 2016 at 1:25 PM, Matt Chambers notifications@github.com wrote:

@jj-umn https://github.com/jj-umn In the GCC2016 workflow, there is a Select step filtering for ^\d+\tpr.B[^\t,](, pr.B[^\t,])\t.$ - This appears to be related to the Mouse pre-pro-B and pro-B FASTQ files, but the proteomic data I have is named like Mo_Tai_iTRAQ_f5. It looks like it's trying to filter on the protein column in the Peptide Shaker output, but my protein column looks like:

1 0, 1.503, 1.5436, 49.5419, 5.2189, 5.2784, 5.2833 GLLLYGPPGTGK 2 731.5188 SIYYITGESK

So I guess it has something to do with the input FASTA. Is it supposed to select only the non-reference accessions for each PSM?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproteomics/tools-galaxyp/issues/70#issuecomment-266836059, or mute the thread https://github.com/notifications/unsubscribe-auth/AKL0OkyLpI3LLbvlBUxQhGgqzOTzU4Rkks5rHvFBgaJpZM4KngwS .

[chambm]

@jj-umn Is the regex above for selecting only PSMs that map to ONLY non-reference protein sequences?