Closed kba closed 2 years ago
Why do you come up with a new workflow format? We have already agreed that we will use the Nextflow syntax. As users, they will submit the whole Nextflow file (.nf
) to the Workflow Server for the execution.
Or do you have any opinion against that approach?
In our last meeting on Friday, I mentioned providing a simpler workflow format which I can then translate to a Nextflow script. There were no objections. So I thought we are going to offer two ways of providing workflows: 1) A basic workflow format - for basic workflows 2) A Nextflow script file (.nf) - for more advanced workflows
Update after today's planning: Since we agreed to use the Nextflow format in general, providing a basic workflow format is not required.
Example Nextflow workflows are available here.
Soon, I will provide Nextflow scripts for the recommended OCR-D workflows here.
Questions the documentation should answer (not exhaustive):
ocrd process
workflows we reference to NF?Should https://github.com/OCR-D/zenhub/issues/72 (Proof of concept) be part of the documentation here or should that be somewhere else?
I think it should be part of the Reference Implementation, https://github.com/OCR-D/zenhub/issues/100
This post contains brief answers to the initially requested questions on Nextflow (NF). I will regularly edit/update this post as more feedback is obtained. I will also extend the answers depending on the further questions/needs. I'm reserving my right to answer with a documentation link when the topic to be covered requires more details. PS: I am not a Nextflow expert. I am myself using Nextflow for a few months and still learning.
Nextflow is a workflow framework that allows the integration of various scripting languages into a single cohesive pipeline. Nextflow also has its own Domain Specific Language (DSL) that extends Groovy (extension of Java).
We choose it due to its rich set of features:
The NF script contains the following structures:
Check this source code example: seq_ocrd_wf_many.nf
TODO: I will provide more structure-related details here based on the example above.
The minimally used features for local runs are the parameters, processes, process decorators, and workflows. I will provide further answers to any following questions related to this main question. I am not sure what else to cover here for now.
A Nextflow workflow script contains several processes. Processes are executed independently and are isolated from each other (i.e. they do not have a shared memory space). Communication between the processes is possible only through data channels (similar to the pipes model in Unix). These channels are basically asynchronous FIFO queues. Any process can define one or more channels as input and output. The order of interaction between these processes, and ultimately the order of workflow execution depends on the communication channel dependencies between processes. For example, if process A writes data to channel A and process B reads data from channel A, then Nextflow knows that process A must be executed before process B.
Check this source code example: seq_ocrd_wf_many.nf
TODO: I will provide more parallelization details here based on the example above.
There is still no running processing server. More details will be announced once there is more to talk about. The interaction will most probably happen with curl through a bash script inside the Nextflow process. Of course, if it is integrated inside the OCR-D core, then no direct interactions will be needed from inside the Nextflow script.
There is still no running METS server. More details will be announced once there is more to talk about. The interaction will most probably happen with curl through a bash script inside the Nextflow process. Of course, if it is integrated inside the OCR-D core, then no direct interactions will be needed from inside the Nextflow script.
I have written an OtoN (OCR-D to Nextflow) converter which converts basic OCR-D process workflows to Nextflow workflow scripts. Check here: OtoN
Currently, there are no known issues or bugs. Feel free to report any bugs, errors, or lack of errors (when an error is expected). The tool will probably be a part of the OCR-D software in the future when it is stable enough for general use.
Depends on the use case. Detailed instructions for local executions and example Nextflow workflow scripts can be found here: Nextflow
I will provide further answers to any following questions related to this main question.
Try to stick to the structure provided in point 2 when writing Nextflow scripts. You can also check the Nextflow examples provided in point 8. The naming conventions for variables, function names, process names, and workflow names are encouraged to follow the snake case. I will provide further answers to any following questions related to this main question.
Superseded by #216
My initial idea for the workflow format is as follows:
So, here is a brief example of an OCR-D workflow:
Notes:
Of course, a more advanced variation is also possible. However, the more complex it gets, the harder it would be for the general user. For example, having separate blocks for different processing categories. Consider the following format:
Notes: