Pointing to correct global.cylc

[cwhitlock-NOAA]

The FMS conda env currently has an empty global.cylc file. When loading a conda environment to run a local copy of fre-cli, this can lead to the following behavior/error:

2024-06-14T15:29:56-04:00 INFO - Extracting job.sh to /home/cew/cylc-run/c384L65_am5f7b12r0_pdclim1850F__gfdl.ncrc5-deploy__prod-
    openmp/.service/etc/job.sh
 <...snip...>
    _status_subtool()
TypeError: _status_subtool() missing 3 required positional arguments: 'experiment', 'platform', and 'target'

Loading the cylc module after the conda env activation does not solve the issue.

Ian pointed out that the reason this does not trigger for all developers is that most developers have a file ~/.cylc/flow/global.cylc that either mirrors the module global.cylc or has enough configuration present that cylc will run. ~/.cylc/flow/global.cylc is prioritized by cylc if it is present; cylc does not notice or use the empty config file.

Reference: https://cylc.github.io/cylc-doc/stable/html/reference/config/global.html

With that, there seem to be two levels of solution: short-term fix: Document that anyone wanting to run fre-cli from a conda environment should have a file ~/.cylc/flow/global.cylc that mirrors the one in the cylc module; you can do that with the following:

   mkdir --parents ~/.cylc/flow && touch ~/.cylc/flow/global.cylc && module load cylc && cylc config > ~/.cylc/flow/global.cylc

long-term fix: The default global.cylc in fre-cli should mirror the one in the cylc module. We could add this as part of the push scripts in git-hooks (every time that code is pushed, module load cylc && cylc config > $GIT_ROOT/global.cylc triggers) or we might be able to alter something in system-settings

[ceblanton]

A configured global.cylc is needed for GFDL use (https://gitlab.gfdl.noaa.gov/fre2/system-settings/-/blob/main/flow/global.cylc):

• run the cylc daemon service processes on the workflow1 host (default is to run the scheduler on the host you're running cylc play from)
• use /xtmp for share and work directories, and when the workflow ends, delete the share and work directories (default is setting share and work in $HOME, and not removing work directories afterwards)
• if the workflow requests the "ppan" platform, use the Slurm job scheduler and submit jobs to the PP/AN batch nodes
• if the workflow requests the "ppan_test" platform, use the "ppan" settings plus scrape performance metrics for EPMT

But for non-GFDL use, this global.cylc file isn't needed, or a different one may be needed. @singhd789 and @Ciheim run Cylc workflows outside GFDL and may have the most experience with non-GFDL global.cylc files.

I'm not sure I agree with storing the GFDL global.cylc here in fre-cli, but I agree it has to go somewhere, and it's a little confusing now.

Also, how does not using the GFDL global.cylc result in the error you see? I would think that not using the GFDL global.cylc would result in running the Cylc scheduler where you are and would cause an error if the workflow definition (flow.cylc) uses the "ppan" or "ppan_test" platforms.

missing 3 required positional arguments: 'experiment', 'platform', and 'target'

[ilaflott]

@ceblanton at GFDL, if one does conda activate fre-cli to use fre pp <subtool> etc., then the necessary global.cylc is not pulled in automatically. I am not convinced the error message above has anything to do with those global cylc settings, necessarily.

One can see this will be an issue with a cylc config giving no output, after you hide any ~/.cylc/flow/global.cylc files you have lying around from past development work. What this means is a new user on GFDL whom e.g. tries to configure a workflow, and run via fre pp configure/install/run, will not be running with the necessary platform definitions within the file you linked.

Ergo, when the conda env is installed/built/updated at GFDL, it would be nice if there were something to cover the above missing piece. I agree that GFDL's global.cylc doesn't belong in this repo, but a procedure for placing the global.cylc or figuring out if it's needed at all seems like it would cover this base.