nested make flow

[mlap-t]

Dear makeflow experts, Could you provide an example of a nested makeflow (using jx). I saw it in the manual:

output-files: input-files MAKEFLOW makeflow-file [working-dir]

but was not able to implement it. thanks a lot

[btovar]

Sure! I'll add this example to the manual:

FILE: main.jx

{ "rules": [ { "workflow": "sub.jx", "args": {"my_var": "hello world"}, "outputs": [ { "dag_name": "output" } ] } ] }

FILE: sub.jx

{ "rules": [ { "command": "echo " + my_var + " > output", "outputs": [ "output" ] } ] }

$ makeflow --jx main.jx

The workflow in main.jx uses sub.jx as a nested workflow. Note that the outputs of sub.jx are listed as the outputs in the corresponding rules in main.jx. (The same would go for the inputs). In this example main.jx is passing the value of "my_var" to the nested workflow.

Ben

On Mon, Oct 5, 2020 at 6:26 AM mlap-t notifications@github.com wrote:

Dear makeflow experts, Could you provide an example of a nested makeflow (using jx). I saw it in the manual:

output-files: input-files MAKEFLOW makeflow-file [working-dir]

but was not able to implement it. thanks a lot

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cooperative-computing-lab/makeflow-examples/issues/39, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXQMYQHUQJ4RS3UZB3B5UDSJGNMFANCNFSM4SEQA35A .

[mlap-t]

Thanks a lot. I tried but did not succeed (yet...). The situation is: I have a set of makeflow files each in a subdirectory. These files are stand alone: I just run them doing makeflow --jx myWF.jx.

So my main worflow is simply:

{
    "define": {
            "vars"  : ["dm2F", "dcpF", "th13F","th23F","spectralF","cthSlopeF","nueanueF","numuanumuF","nuenumuF","escaleEmF","escaleHadF","normNCF","Tracks","Middle","Showers","taunorm_nc_fit"]
    },
    "rules": [
    {
        "workflow" : format("%s/errorScan.jx",v)
    } for v in vars
    ]
}

and for reference you can see all the code here: https://www.cppm.in2p3.fr/~mperrint/tmp/Err_nmh-y3.0-F49-orca_GENOA_DUSJ-2020-10-06_1519/

The main is : https://www.cppm.in2p3.fr/~mperrint/tmp/Err_nmh-y3.0-F49-orca_GENOA_DUSJ-2020-10-06_1519/errorScanAll.jx

and the sub-wf are in the subfolder, for example: https://www.cppm.in2p3.fr/~mperrint/tmp/Err_nmh-y3.0-F49-orca_GENOA_DUSJ-2020-10-06_1519/dm2F/errorScan.jx

When I run makeflow --jx errorScanAll.jx I get:

parsing errorScanAll.jx...
makeflow: line 0: rule neither defines a command nor a submakeflow.makeflow: line 0: error parsing the rule.2020/10/06 16:31:00.87 makeflow[256285] fatal: makeflow: couldn't load errorScanAll.jx: Invalid argument
Terminated

thanks in advance for your help

mathieu

[btovar]

mathieu,

As it is, we are able to parse the errorScanAll.jx. The error you are getting seems to only appear if "command" or "workflow" are missing. Which version of makeflow are you using? (makeflow --version)

That said, it seems that nested workflows may not be the tool you need. When using nested workflows, the sub-workflows become part of the main one, which means the inputs, and outputs are computed from the point of view of the main workflow.

Given that all your sub-workflows are independent of each other, I would instead try something like this:

{
    "define": {
            "vars"  : ["dm2F", "dcpF", "th13F","th23F","spectralF","cthSlopeF","nueanueF","numuanumuF","nuenumuF","escaleEmF","escaleHadF","normNCF","Tracks","Middle","Showers","taunorm_nc_fit"]
    },
    "rules": [
    {
        "command" : format("cd %s && makeflow errorScan.jx",v)
    } for v in vars
    ]
}

[mlap-t]

Thanks Ben, I run locally on a machine which has 48 cores and I wanted to optimised the usage of the ressources, this is why I try the nested workflow. I am not sure I will have this optimisation with the solution you suggest, do I?

My makeflow version is: makeflow version 7.0.18 FINAL from source (released 2019-09-24 11:52:32 -0400) Built by btovar@disc01.crc.nd.edu on 2019-09-24 11:52:32 -0400 System: Linux disc01.crc.nd.edu 3.10.0-1062.1.1.el7.x86_64 #1 SMP Tue Aug 13 18:39:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Configuration: --strict --build-label from source --build-date --tcp-low-port 9000 --sge-parameter -pe smp $cores --strict --prefix /var/condor/execute/dir_20139/cctools-10231a88-x86_64-centos7

[btovar]

mathieu, yes, you should be able to use all the 48 cores. The only thing that is really changing is that you won't need to define the inputs and outputs of every sub-workflow in errorScanAll.jx.

[mlap-t]

Ben, I just tried your solution but I end up with 480 jobs running on my 48 cpu (so each job forced to use only 1% of a CPU). I would prefer running only 42 jobs at a time.

[btovar]

One thing you can try is to limit how many subworkflows and rules per subworkflow run at each time. For example, to run a maximum of 6 subworkflows at the same time, you can try:

$ makeflow --jx errorScanAll.jx -j 6

and then to make sure that at no time you run more than 42 jobs, you can modify the "command" line to:

"command" : format("cd %s && makeflow errorScan.jx  -j 7",v)

That said, I would like to talk with our team because I think we can modify makeflow to serve your use-case better.

[mlap-t]

Thanks Ben, although it is not optimal, I think this is a working solution :+1:

[btovar]

Yes, the issue is that by design each subworkflow becomes a blackbox, which means we cannot do a finer resource allocation. Once resources for a subworkflow are set, they can't be modified.

I think a slightly better version is something like this:

    "define": {
            "cores_per_workflow": 4,
            "vars"  : ["dm2F", "dcpF", "th13F","th23F","spectralF","cthSlopeF","nueanueF","numuanumuF","nuenumuF","escaleEmF","escaleHadF","normNCF","Tracks","Middle","Showers","taunorm_nc_fit"]
    },
    "categories":
    {
        "default":
        {
            "resources": {"cores":cores_per_workflow}
        }
    },  

    "rules": [
    {
        "command" : format("cd %s && makeflow --local-cores %d errorScan.jx",v, cores_per_workflow)
    } for v in vars
    ]
}

and then run makeflow normally with: $ makeflow --jx errorScanAll.jx

In this way, each subworkflow won't use more than 4 cores, and the total number of cores used won't be more than the ones the machine has. (Of course you can modify the 4 cores to some other value, and setting a maximum total number of cores used to some value N with $ makeflow --jx errorScanAll.jx --local-cores N)

[mlap-t]

Thanks Ben, although it is not optimal, I think this is a working solution +1

It is indeed working :) Thanks again

[btovar]

Thanks for letting us know!