search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.61k stars 606 forks source link

Remove 64KB size limit on scripts #5025

Open bentsherman opened 1 month ago

bentsherman commented 1 month ago

Since DSL2, Nextflow scripts have been limited to 64KB. I can't remember the exact reason but I recall that it has to do with the Groovy engine that is used to evaluate scripts. GroovyShell is used for scripts and Grengine is used for config files. However, I think we should be able to use the Groovy CompilationUnit directly instead, and I think that would remove the limit. Might be something to implement in #4613 and #4744 .

Also, the TaskRun uses Grengine to (1) evaluate the task script and (2) evaluate global variables for the task hash. I think we can address both with the custom parser. For (1) we can just use a closure, and the parser can handle the shell: syntax. For (2), I don't think global variables will be allowed by the strict parser so it won't be needed.

bentsherman commented 1 month ago

Extending Groovy's CompilationUnit will also allow us to further customize the compilation pipeline, such as by removing phases that we don't need or replacing them with more optimized versions.