caracal-pipeline / stimela

Stimela 2.0
GNU General Public License v2.0
5 stars 4 forks source link

For-loops and step level assignments fail prevalidation #282

Open JSKenyon opened 7 months ago

JSKenyon commented 7 months ago

As discussed, when adding a simple for loop to my recipe, I ran into the following error.

────────────────────────────────── detailed error report follows ───────────────────────────────────
        ⚠ pre-validation of recipe '1gc' failed                                                     
        └── ValidationError: Value 'Unresolved(recipe.nnodes)' of type 'scabha.basetypes.UNSET'     
            could not be converted to Integer                                                       
                full_key: opts.backend.kube.dask_cluster.num_workers                                
                reference_type=Optional[DaskCluster]                                                
                object_type=DaskCluster  

This seems to stem from using the loop variable inside a step level assignment. The following is a stripped-down config which demonstrates the currently bugged pattern.

_include:
  - kubeconfig.yaml
  - (cultcargo)quartical.yml

1gc:
  name: CROSSCAL

  for_loop:
    var: nnodes
    over: [1,2,3]

  steps:
    gkb1:
      cab: quartical
      assign:
        config.opts.backend.kube.dask_cluster.num_workers: =recipe.nnodes

The current workaround (courtesy of @o-smirnov) is to first do a dummy assignment at the recipe level as follows:

_include:
  - kubeconfig.yaml
  - (cultcargo)quartical.yml

1gc:
  name: CROSSCAL

  assign:
    nnodes: 0

  for_loop:
    var: nnodes
    over: [1,2,3]

  steps:
    gkb1:
      cab: quartical
      assign:
        config.opts.backend.kube.dask_cluster.num_workers: =recipe.nnodes