search

broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
996 stars 361 forks source link

putting everything to one zip archive #2410

Open antonkulaga opened 7 years ago

antonkulaga commented 7 years ago

It is very inconvenient to provide a lot of separate files via cromwell REST or console and it often leads to many mistakes when one has to run many pipelines. It would be much better to be able just zip whole folder with cromwell project and give one json config file that gives all pathes inside zip (where is subworkflows folder, where are options and where is the main workflow).

katevoss commented 7 years ago

@antonkulaga can you tell me more about your use case for zipping all of your files together?

antonkulaga commented 7 years ago

We have RNA-Seq and de-novo assembly pipelines with many sub-workflows, forgetting to include a file or two is a common mistake that people make in the lab. Another inconvenience is that we have sub workflows in subfolders, and we do not know how to both keep references (incl. subfolders) to sub-workflows in the main WDL script (so IntelliJ idea WDL plugin can check that they are correct) but at the same time - send them as a lot of files via REST. I think being able to pack everything into one zip archive(preserving subfolder structure) and send it to REST API will allow tracking relative paths properly. And the last argument is that sending only one file (instead of 5-15) to REST API is way more convenient for users!