Issue in `radical-utils-env.sh`

[mtitov]

Issue caught in bootstrap_0.out

...
which: no radical-utils-env.sh in (/home/carns/working/src/recup/rct-example-2023-04/rct/rp_install/bin:/home/carns/working/src/recup/rct-example-2023-04/rct/bin:/home/carns/working/src/recup/rct-example-2023-04/rct/bin:/soft/datascience/conda/2022-09-08/mconda3/condabin:/soft/compilers/cudatoolkit/cuda-11.6.2/bin:/soft/libraries/nccl/nccl_2.14.3-1+cuda11.6_x86_64/include:/opt/cray/pe/pals/1.1.7/bin:/opt/cray/pe/craype/2.7.15/bin:/opt/cray/pe/gcc/11.2.0/bin:/opt/cray/pe/perftools/22.05.0/bin:/opt/cray/pe/papi/6.0.0.14/bin:/opt/cray/libfabric/1.11.0.4.125/bin:/opt/clmgr/sbin:/opt/clmgr/bin:/opt/sgi/sbin:/opt/sgi/bin:/home/carns/bin:/usr/local/bin:/usr/bin:/bin:/opt/c3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/pbs/bin:/sbin:/opt/cray/pe/bin)
radical-utils-env.sh: line 221: printf: missing unicode digit for \u
...

radical.utils version: 1.23.0-v1.22.0-1-g630a552@devel

env.rp.error.txt

[andre-merzky]

This is the culprit it seems:

BASH_FUNC___conda_activate%%=() {  if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then\n PS1="$CONDA_PS1_BACKUP";\n \unset CONDA_PS1_BACKUP;\n fi;\n \local ask_conda;\n ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return;\n \eval "$ask_conda";\n __conda_hashr\n}

There is an \unset and later a \return and eval. On the shell level that is acceptable (the backslash ensures expansion with $PATH but ignores aliases and functions IIRC), but in the context we use it it looks like unicode characters - some quoting is missing (should be\return` etc).

@mtitov : can you please check how conda_activate is defined in the environment?

[mtitov]

@andre-merzky for that platform, we do have in pre_bootstrap_0 conda loading (module load conda)

specifically about this run - as I understood that venv environment was active on the client side, and that what pilot tried to use. At the client side there were two environments activated (conda and venv) and RCT packages were installed within conda env, thus they were available for the client application, but not for the pilot side (you can see it in attached bootstrap_0.out)

Thus, this issue with radical-utils-env.sh seems didn't cause the failure, but kinda would be expected it to handle such cases

bootstrap_0.out.txt

[andre-merzky]

@mtitov : I have trouble seeing what is going on here. Would you mind adding a set -x near the top of the bootstrapper, run again, and send me the (lengthy) out and err files? Thanks!

[andre-merzky]

Pondering a bit further, I think this line

gsub(/\n/,"\\n",v);

should be changed to

gsub(/\/,"\\",v);

Would you mind giving this a try?

[mtitov]

with this change it gives an error

awk: cmd. line:4:       gsub(/\/,"\\",v);
awk: cmd. line:4:             ^ unterminated regexp
awk: cmd. line:5:       gsub(/\/,"\\",v);
awk: cmd. line:5:                        ^ unexpected newline or end of string
awk: cmd. line:5:             print k"="v;
awk: cmd. line:5:                        ^ syntax error

p.s. I've also attached bootstrap_0.out with the original error message and set -x

bootstrap0.out-x.txt

[andre-merzky]

Thanks, the output was helpful, I can reproduce this locally now.