Specifying the change of namelist options in the config file

[agoscinski]

Objective

From the last discussion on PR https://github.com/C2SM/ETHIOPIA/pull/20 we established that passing namelists needs extra logic that cannot be handled by this PR in a user friendly for w&c workflows. There needs to be a way in workflow config file to update the namelist options during the workflow without creating an extra script that does it for the user. In addition certain updates of the namelist need to be automatized by workflow engine as start and end dates.

Proposition

I think it is important that everything that is specified in the config file in a certain way also always behaves in the certain way. Therefore I would like to highlight namelists in the config file differently than regular inputs to signify to the user that they are also handled differently internally. I also think it is important that in principle the update of start and end dates in the namelist can be expressed in the config file. This way it can be adapted by the user if needed and it is easier to explain to the user in the doc that we are just adding this to the config file when the plugin is icon.

Here is the suggestion

cycles:
- icon_cycle:
    period: 3PM
    tasks:
    - icon_task:
        # for namelist updates
        namelists:
        # because multiple namelists can be changed here 
        # we first specify the data node name of the name list
        - master_namelist:
            port_name: icon_master.namelist # aiida plugin socket/port
            options:
              master_model_nml:
                model_min_rank: 2
              # This is implicitely done by the workflow engine
              # and does not need to be done by the user
              # but can be changed by the user if needed
              master_time_control_nml:
                 # macros need to be also discussed, but lets not dive too much
                 # into this here, this is just some example for possible usage
                 # in the future to get a feeling
                experimentstartdate: '%CURRENT_DATE'
                experimentstopdate: '%CURRENT_DATE + %CYCLE_PERIOD'
        - model_namelist:
            port_name: model.namelist

The namelists are now put under a new key namelists. Alternatively one could also mark inputs as namelists, but I had the feeling this would become to nested once one wants to update the options. This should be then also compatible with the global inheritance that we need to decide in issue #26.

A dummy namelist from aiida-icon as reference https://github.com/aiida-icon/aiida-icon/blob/main/tests/data/restarts_present/inputs/icon_master.namelist.

[leclairm]

I think that, by default, everything should be handled by the tool without any explicit mention of the changes in the yaml file. I think it would clutter the yaml file too much without bringing much possibility to the user. Typically, changing the dates is not a procedure that the user would do differently than what is done internally. Actually, expressing the modification of the restart option (are we in a fresh start or a restart) would require some sort of conditional statement in the yaml file.

Still, I've always been in favor of having the possibility to override namelist parameters from the yaml file, but that should first belong to the task description, not the cycles. If we can think of a use case where modifications are cycle dependent we could also expose it there but for now we don't need anything besides dates handling and restart mechanism.

Regarding implementation, it would anyways make sense to work as if we had to expose these mechanisms later on. Like using the namelist modification from aiida-icon modify_master_nml

[agoscinski]

I think that, by default, everything should be handled by the tool without any explicit mention of the changes in the yaml file. I think it would clutter the yaml file too much without bringing much possibility to the user. Typically, changing the dates is not a procedure that the user would do differently than what is done internally. Actually, expressing the modification of the restart option (are we in a fresh start or a restart) would require some sort of conditional statement in the yaml file.

Yes that I don't disagree with. The suggestion is just to make it possible in principle to change this default behavior so it can be clearer communicated what is happening with the namelist between the steps and it can be changed in principle.

Still, I've always been in favor of having the possibility to override namelist parameters from the yaml file, but that should first belong to the task description, not the cycles. If we can think of a use case where modifications are cycle dependent we could also expose it there but for now we don't need anything besides dates handling and restart mechanism.

Specifying it in the task will be also possible, but I did not present it here because it depends how we solve #26.

Regarding implementation, it would anyways make sense to work as if we had to expose these mechanisms later on. Like using the namelist modification from aiida-icon modify_master_nml

That means putting it into the inputs.

- icon_task:
    inputs:
    - master_namelist:
        port_name: icon_master.namelist # aiida plugin socket/port
        modify_nml:
            model_min_rank: 2

I was afraid that this becomes to nested, but it looks okay.

[agoscinski]

We had a discussion on zoom and I wanted to post here a summary of some points.

We decided that the namelist is not part of the inputs but rather a config option of the task and we want to try to support these two ways to specifying it

tasks:
- icon_task
    icon_master_namelist:
      src: /path/to/icon/namelist
      modify:  # modifies namelist
          model_min_rank: 2

and

tasks:
- icon_task
    icon_master_namelist: /path/to/icon/namelist

[leclairm]

This should go into a PR dealing with all the specifics of an ICON task.