Runtime introduction tutorial doesn't work out-of-the-box for conda installation

Description

I was following the cylc tutorial and was unable to make the runtime-introduction practical work.

On my first try, the job did run (the log shows the python code was executed and the csvs are done), but it seems it can't communicate back to the scheduler.

Reproducible Example

Test 1 - just follow the basic instructions

mamba create -n cylc cylc-flow cylc-uiserver
mamba activate cylc
cylc get-resources tutorial/runtime-introduction
cd ~/cylc-src/runtime-introduction
cylc validate .
cylc install
cylc play runtime-introduction

In the GUI, I see that all jobs are submitted, but none ever go beyond that stage. Looking at a log I see:

# cat ~/cylc-run/runtime-introduction/run1/log/job/20000101T0000Z/get_observations_aldergrove/01/job.err
bash: cylc: command not found
bash: cylc: command not found

Interestingly, I note that some PATH manipulation happens in ~/cylc-src/runtime-introduction/etc/python-job.settings, resulting in the following function added to ~/cylc-run/runtime-introduction/run1/log/job/20000101T0000Z/get_observations_aldergrove/01/job :

cylc__job__inst__user_env() {
    # TASK RUNTIME ENVIRONMENT:
    export PYTHONPATH PATH
    PYTHONPATH="/exec/prod2/.conda/cylc/bin:/exec/prod2/.conda/cylc/lib/python39.zip:/exec/prod2/.conda/cylc/lib/python3.9:/exec/prod2/.conda/cylc/lib/python3.9/lib-dynload:/exec/prod2/.conda/cylc/lib/python3.9/site-packages"
    PATH="$(dirname /exec/prod2/.conda/cylc/bin/python):$PATH"
}

(/exec/prod2/.conda is where my conda envs are installed)

However, cylc__job__inst__user_env() seems to never be called anywhere. I.e. output of grep -r cylc__job__inst__user_env ~/cylc-run/runtime-introduction/ only shows the declaration of that function in the generated job files).

Test 2 - `cylc` wrapper

After that try, I realized I hadn't read the Managing environments section of the installation doc (because it was under "Advanced installation"). And I started from the beginning again:

rm -r ~/cylc-*
cylc get-resources cylc ~/.local/bin
chmod +x ~/.local/bin/cylc
# Then edited ~/.local/bin/cylc with : CYLC_HOME_ROOT_ALT=/path/to/my/conda/envs
ln -s /path/to/my/conda/envs/cylc /path/to/my/conda/envs/cylc-8.2.3

# Then get the tutorial
cylc get-resources tutorial/runtime-introduction
cd ~/cylc-src/runtime-introduction
cylc validate .
cylc install
cylc play runtime-introduction

And this time it worked!

Conclusion

I was able to make it run, but I believe my experience raises the following issues:

Tutorial run seems to depend on some "Advanced configuration" that is not evident from reading the doc, at least for some setups (conda).
The python-job.settings "hack" looks like it does nothing ? Shouldn't it have added the correct cylc to the path in my first try ?
Naming the conda env with the cylc version in this kind of setup is kind of awkward as I don't know what version I'm going to get when calling mamba create. Should I override with CYLC_ENV_NAME, even though this is not recommended ?

cylc / cylc-flow