WilhelmusLab / ice-floe-tracker-pipeline

Processing pipeline for IceFloeTracker.jl
1 stars 1 forks source link

Populate --cpus-per-task based on number of images in set #82

Open tdivoll opened 11 months ago

tdivoll commented 11 months ago

In theory, this is possible using cylc broadcast.

See this thread.

I tried setting a family set_cores_num with this:

[[set_cores]] 
        # From the broadcast above, this family can be used to dynamically set the number of cores
        [[[environment]]]
            num_images1=( $(find $fetchdata_dir -maxdepth 2 -type f | wc -l) )
            num_images2=( $(echo $((($(date +%s -d ${enddate_array[0]}) - $(date -d +%s ${startdate_array[0]}))/86400)) ) )
        script = """
            cylc broadcast \
            "${CYLC_WORKFLOW_ID}" \
            -n set_cores_num
            -p "${CYLC_TASK_CYCLE_POINT}" \
            -s '[directives]--cpus-per-task=${num_images}' \
            -s '[directives]--mem=60G' \
            -s '[environment]num_images=${num_images}'
        """
[[set_cores_num]]
[[preprocess<param_set>]]
        inherit = set_cores_num

TO TRY: * num_images1 => runs fine but does not add the directives to preprocessing task as expected