Debug Cylc crashing during `job.sh` run on Oscar

[tdivoll]

I have been getting an error trying to use Cylc that suggests cylc is not found half way through the job.sh, specifically lines 130 and 164 of the job.sh.

The workflow:

• connect to Oscar via VSCode
• deactivate Conda base env
• activate ift-env Conda env that contains Cylc install
• generate the cylc_hpc/flow.cylc using the sample params csv file and the flow_generator.py
• cylc install && cylc play && cylc tui
• It hangs on the first task: global_setup
• check `workflow/cylc_run//run1/log/job//01/job.err:

• /users/tdivoll/cylc-run/july-test/run1/.service/etc/job.sh: line 130: cylc: command not found /users/tdivoll/cylc-run/july-test/run1/.service/etc/job.sh: line 164: cylc: command not found

• check `workflow/cylc-run//run1/.service/etc/job.sh for line 130:

  # Send task started message
  cylc message -- "${CYLC_WORKFLOW_ID}" "${CYLC_TASK_JOB}" 'started' &
  CYLC_TASK_MESSAGE_STARTED_PID=$!

• check the Cylc forum to find this issue

Things I have tried:

1. install the cylc wrapper script as suggested in the linked thread and set the env var for Cylc version
2. add CYLC_HOME_ROOT="${CYLC_HOME_ROOT:-/users/tdivoll/.conda/envs/ift-cylc/bin/cylc}" to .bash_profile as a temp hack to see if it works
3. check that I don't have conda config --set auto_activate_base=true in .bashrc, .bash_profile, or .zshrc. See this thread for context.

[tdivoll]

SOLUTION:

.bashrc was being sourced on bash_profile! After I deleted this block from .bash_profile, Cylc works.

# Get the aliases and functions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

I suspect it was the Conda init block on the .bashrc that gets added by default when using Conda on Oscar. Essentially it would either activate the base environment inside the ift-env or just switch to base from ift-env (I'm still not sure which case is true), which then didn't have Cylc.

[tdivoll]

@hollandjg, for context as you move into using Cylc on Oscar