Conversion of FragPipe workflow file into philosopher pipeline config file

tobiasko commented 3 years ago

Describe the issue or question:

https://github.com/Nesvilab/philosopher/discussions/276#discussioncomment-1302874

I would be really helpful if one could develop a FragPipe workflow using the GUI and afterwards turn it into a command line workflow that is executed using philosopher pipeline. The main obstacle: Both softwares use different ways of logging parameters (workflow files vs. config files) and some central parts of the FragPipe haven't been integrated into Philosopher (Ionquant, Percolator, ...). Are you planing to bridge these two frameworks?

Upload your log file (If a log file hasn't been generated, go to the 'Run' tab in FragPipe, click 'Export Log', zip the resulting "log_[date_time].txt" file to avoid truncation, then attach the zipped file by drag & drop here.)

fcyu commented 3 years ago

Yes, Guo Ci @guoci and I briefly discussed about it. The basic idea is making FragPipe support command line (no GUI). Then, using workflow file, fasta file, and data manifest file as input. We haven't looked into the technical details to see if it is easy to implement, but will keep you posted if there is any progress.

Best,

Fengchao

jjGG commented 3 years ago

Hei @Fengchao and @guoci,

Why not optionally also write the bash-file (similar to "print commands") but one that is actually working? (If you copy-paste out from the "print commands" there is still a lot of things that are not written out and it keeps changing for every new version?

fcyu commented 3 years ago

First of all, all paths in the bash file would be not correct if FragPipe was running in one computer but the bash file was supposed to be run in another computer. Users need to modify them manually. Second, there are some processes that are not reflected in the printed commands.

Best,

Fengchao

anesvi commented 3 years ago

Yes should do that Maybe with an easy way for the user to edit paths to files and programs on a different system that they intent to run the script

Sent from my iPhone

On Sep 14, 2021, at 9:37 AM, Grossmann @.***> wrote:

External Email - Use Caution

Hei @fengchaohttps://github.com/fengchao and @guocihttps://github.com/guoci,

Why not optionally also write the bash-file (similar to "print commands") but one that is actually working? (If you copy-paste out from the "print commands" there is still a lot of things that are not written out and it keeps changing for every new version?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/FragPipe/issues/462#issuecomment-919158595, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIIMM6ZEG5IUA45JXACACP3UB5FXNANCNFSM5EADBOQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues

tobiasko commented 3 years ago

A FragPipe CLI! That would be the premium solution! Thanks for considering!

anesvi commented 3 years ago

Are you guys running FragPipe on linux clusters, with a lot of files etc.? Or do you have an issue with GUI on your system? Can you help us understand why FragPipe CLI would be beneficial in your case? Best, Alexey

From: Tobias Kockmann @.> Sent: Tuesday, September 14, 2021 10:42 AM To: Nesvilab/FragPipe @.> Cc: Nesvizhskii, Alexey @.>; Comment @.> Subject: Re: [Nesvilab/FragPipe] Conversion of FragPipe workflow file into philosopher pipeline config file (#462)

External Email - Use Caution

A FragPipe CLI! That would be the premium solution! Thanks for considering!

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/FragPipe/issues/462#issuecomment-919218255, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIIMM6ZBK345NFCNC7RNGALUB5NL3ANCNFSM5EADBOQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues

tobiasko commented 3 years ago

I will try: We are already running a FragPipe-like analysis workflow (using shell scripts and philosopher pipeline) on a computing cluster that resembles the LFQ-MBR for IMS data from the FragPipe GUI. The jobs are actually submitted through our coreLIMS system (termed b-fabric) that is accessible via a web interface. The workflow management and job distribution of b-fabric is still a mystery to me. What i know is that b-fabric FragPipe makes use of slurm. We basically copy the raw files to the cluster nodes from our central raw file repository (DB) and do the processing (by running a shell script) on a single node having many cores and plenty of memory (not user accessible). Once that is done, result files are copied back to the DB (registered in b-fabric) and again made available to user/employees via the web interface.

Does this help?

fcyu commented 3 years ago

Is it possible to use X-window forwarding in slurm? I can image that it would be hard, but just want to confirm.

Thanks,

Fengchao

tobiasko commented 3 years ago

So a typical development cycle for a new cluster workflow is: a) Use a FragPipe GUI instance to test which workflow and parameters are needed to get best results on a small scale test case. b) build a command line script that does the same on the cluster using data registered in the coreLIMS.

tobiasko commented 3 years ago

Is it possible to use X-window forwarding in slurm? I can image that it would be hard, but just want to confirm.

Thanks,

Fengchao

I have no clue. But how would that help?

fcyu commented 3 years ago

You can use FragPipe GUI in a remote Linux using X-window forwarding: https://fragpipe.nesvilab.org/docs/tutorial_setup_x_forwarding.html

tobiasko commented 3 years ago

I know, but that is still interactive usage. Our system is more like a Galaxy server, with distributed resources in the back that are not directly exposed to users.

fcyu commented 3 years ago

Thanks. That explains the need of having a FragPipe CLI.

chhh commented 3 years ago

@fcyu @guoci A hint, you have everything you need for a CLI version in the form of FragpipeRun::configureTaskGraph(). With minor refactoring UI dependency can be abstracted away - make an interface for each UI panel that is used in graph config and provide a file-based impl of those interfaces for the CLI version. All the file-parameter-reading is also already there in the form of workflow files.

tobiasko commented 3 years ago

Should you need any help during testing, we would surely support you! Just let us know! We might also be able to provide benchmark data should that be of interest. Currently, we routinely run LFQ jobs through MaxQuant on the cluster, but especially for ddaPASEF data this becomes tedious. We are also looking into combining FragPipe with Dia-NN following your manuscript https://www.biorxiv.org/content/10.1101/2021.03.08.434385v1

tobiasko commented 3 years ago

Something that came to my mind while reading this: https://www.nature.com/articles/s41592-021-01254-9

Why not go one step further and turn FragPipe into a managed pipeline (for instance nextflow) and publish it using a curated repository (like https://nf-co.re/)??? I personally have little experience using any of these tools (apart from Galaxy), but it could be something that makes reusability for other cores much easier and could even be published.

What do you think?

fcyu commented 3 years ago

Thanks for your suggestion. I also have no experience of developing nextflow pipeline, but we will discuss internally.

Best,

Fengchao

fabianegli commented 3 years ago

This is on short notice, but there is a nf-core Hackathon next week. That would be an opportunity to get started with such a pipeline, or at least get some steps wrapped in modules. I will partially attend the nf-core Hackathon - October 2021 and be happy to work on a nf-core pipeline reflecting FragPipe.

fabianegli commented 3 years ago

My thoughts about a potential FragPipe-Nextflow workflow:

Automation (as in this issue and #451 )
Runs on any OS that does have Docker or Singularity installed ( #171 )
Scalability
Better reproducibility (e.g. the Python installation is not being documented for FragPipe runs, if I am not mistaken)
CI

There is already a proteomics pipeline on nf-core that is based on openMS. Here is the website for proteomicslfq and here the GitHub repository. It would make sense to discuss the generation of a new pipeline on the nf-core Slack in the channel proteomicslfq to consolidate the nf-core proteomics community.

NB: nf-core pipelines are required to be MIT licensed.

fcyu commented 3 years ago

Sounds interesting!

Good luck and have fun!

Fengchao

anesvi commented 3 years ago

Can you contact me directly? Easier to discuss this way. nesvi at med.umich.edu Best Alexey

From: Fabian Egli @.> Sent: Tuesday, October 19, 2021 4:35 PM To: Nesvilab/FragPipe @.> Cc: Nesvizhskii, Alexey @.>; Comment @.> Subject: Re: [Nesvilab/FragPipe] Conversion of FragPipe workflow file into philosopher pipeline config file (#462)

External Email - Use Caution

My thoughts about a potential FragPipe-Nextflow workflow:

Automation (as in this issue and #451https://github.com/Nesvilab/FragPipe/issues/451 )
Runs on any OS that does have Docker or Singularity installed ( #171https://github.com/Nesvilab/FragPipe/issues/171 )
Scalability
Better reproducibility (e.g. the Python installation is not being documented for FragPipe runs, if I am not mistaken)
CI

There is already a proteomics pipeline on nf-core that is based on openMS. Here is the website for proteomicslfqhttps://nf-co.re/proteomicslfq and here the GitHub repositoryhttps://github.com/nf-core/proteomicslfq. It would make sense to discuss the generation of a new pipeline on the nf-core Slackhttps://nfcore.slack.com in the channel proteomicslfq to consolidate the nf-core proteomics community.

NB: nf-core pipelines are required to be MIT licensed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/FragPipe/issues/462#issuecomment-947084211, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIIMM67BDALMY2DNUZWEW53UHXI5RANCNFSM5EADBOQQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues

guoci commented 3 years ago

@fabianegli has anything been done for the FragPipe-Nextflow workflow? I have started working on a headless version of FragPipe. The license of FragPipe is GPLv3 and some of the software in the pipeline are closed sourced.

fabianegli commented 3 years ago

@guoci Great to hear that. This will eventually solve this issue. Let's move the Nextflow related discussion into a new issue (#518) to try and keep topics separated.

fcyu commented 2 years ago

Hi all,

The beta version of FragPipe headless is ready. Here (https://www.dropbox.com/s/uyc01xb8ytj998k/FragPipe-17.2-build28.zip?dl=1) is the link to download it. It is actually part of FragPipe. To trigger the headless mode, type fragpipe.bat in Windows or fragpipe in Linux followed by the options:

Running without GUI. Usage:
        Windows: fragpipe.bat --headless --workflow <path to workflow file> --manifest <path to manifest file> --workdir <path to result directory>
        Linux: fragpipe --headless --workflow <path to workflow file> --manifest <path to manifest file> --workdir <path to result directory>
Options:
        -h
        --help                          # Print this help message.
        --headless                      # Running in headless mode.
        --workflow <string>             # Specify path to workflow file.
        --manifest <string>             # Specify path to manifest file.
        --workdir <string>              # Specify the result directory.
        --dry-run                       # (optional) Dry run, not really run FragPipe.
        --ram <integer>                 # (optional) Specify the maximum allowed memory size. Set it to 0 to let FragPipe decide. Default = 0
        --threads <integer>             # (optional) Specify the number of threads. Default = core number - 1
        --config-msfragger <string>     # (optional) specify the location of the MSFragger jar file. If not specified, using the one in the cache.
        --config-philosopher <string>   # (optional) specify the location of the Philosopher binary file. If not specified, using the one in the cache.
        --config-python <string>        # (optional) specify the location of the Python directory. If not specified, using the one in the cache.

For the first time using it, we recommend loading the spectral files in FragPipe GUI, saving the workflow and manifest files. Then, update the database.db-path parameter at the beginning of the workflow file. Run FragPipe headless using the workflow and manifest files.

Please feel free to let us know if you have any questions or find any bugs.

Merry Christmas and happy new year,

Fengchao

cpanse commented 2 years ago

@fabianegli has anything been done for the FragPipe-Nextflow workflow? I have started working on a headless version of FragPipe. The license of FragPipe is GPLv3 and some of the software in the pipeline are closed sourced.

@fabianegli has anything been done for the FragPipe-Nextflow workflow? I have started working on a headless version of FragPipe. The license of FragPipe is GPLv3 and some of the software in the pipeline are closed sourced.

@guoci Can you please point me to the FragPipe source code?

fcyu commented 2 years ago

https://github.com/Nesvilab/FragPipe

fstein commented 2 years ago

Is this link still referencing to the latest version? https://www.dropbox.com/s/uyc01xb8ytj998k/FragPipe-17.2-build28.zip?dl=1

fcyu commented 2 years ago

@fstein , No, here (https://www.dropbox.com/s/0o3508muem55vmb/FragPipe-17.2-build44.zip?dl=1) is the latest pre-release.

Best,

Fengchao

fcyu commented 2 years ago

There is no difference in terms of code. Please check your java installation. You need to have java 9+.

Best,

Fengchao

On Tue, May 10, 2022 at 3:52 AM fstein @.***> wrote:

Hi Fengchao,

unfortunately, I cannot open the FragPipe GUI. I get the following error (Java virtual machine Launcher: Error: A JNI error has occurred, please check your installation and try again): [image: image] https://user-images.githubusercontent.com/4124421/167577028-44cdd156-97ca-40d1-ab65-0e5a96ced42f.png

I do not get this error with the normal version of FragPipe.

Best,

Frank

— Reply to this email directly, view it on GitHub https://github.com/Nesvilab/FragPipe/issues/462#issuecomment-1122049517, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU27W36Q2EH2RCU6K74WHTVJIIRXANCNFSM5EADBOQQ . You are receiving this because you modified the open/close state.Message ID: @.***>

-- Dr. Fengchao Yu Research Investigator University of Michigan

fstein commented 2 years ago

Hi Fengchao,

I forgot to include the java directory into the folders. I figured it out 5 minutes after my post and therefore deleted my post. But thanks for getting back to me :-)

Best,

Frank

Nesvilab / FragPipe

Conversion of FragPipe workflow file into philosopher pipeline config file #462