robocorp / rcc

Repeatable, movable and isolated Python environments for your automation. šŸš€
https://robocorp.com/docs/
Apache License 2.0
453 stars 92 forks source link

No interactive shell in a chroot based environment, because PS1= is set to empty value #36

Closed datakurre closed 2 years ago

datakurre commented 2 years ago

I don't believe that this is a bug in RCC, but a glitch caused by my environment.

I have a reason to run RCC in a chroot based sandbox. While everything else I haved tried so far has been working (e.g. even a Firefox installed by RCC in a robot execution works), interactive shell does not.

rcc task shell --trace

Executes shell, but without input or output.

06.092052.803 [D] about to run command - [/nix/store/4wbh6d4sxqc81lmn3slcqy90j23awhzr-bash-interactive-5.1-p16/bin/bash --noprofile --norc -i]
06.092052.803 [T] Execute "/nix/store/4wbh6d4sxqc81lmn3slcqy90j23awhzr-bash-interactive-5.1-p16/bin/bash" with arguments ["--noprofile" "--norc" "-i"]
06.092052.803 [D] PID #16490 is "/nix/store/4wbh6d4sxqc81lmn3slcqy90j23awhzr-bash-interactive-5.1-p16/bin/bash --noprofile --norc -i".

Effectively, the command waits until shell process gets killed from the outside.

06.092713.075 [D] PID #16490 finished: signal: killed.
06.092713.233 [D] live "/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_9fcd2534" diagnosis: did not change during run [ef509423402cfa1cc265c3c7479ce00af65dfc0f63321fa826b5f25230aa99e1]
06.092713.233 [N] rcc shell lasted 382.492
06.092713.233 [D] Marked "/home/vagrant/.robocorp/temp/4d65822107fcbd30" for recycling.
06.092713.233 [N] Error: signal: killed
06.092713.233 [D] wait telemetry to complete
06.092713.233 [D] telemetry sending completed

Running the same shell command manually outside RCC works.

Any tips, how to debug this more and get RCC shell to work in my weird environment? For example, maybe my chroot based env is just missing some environment variables required for interactive shell through RCC.

$ rcc version
v11.25.0
$ rcc configuration diagnostics
Diagnostics:
 - RCC_VERBOSE_ENVIRONMENT_BUILDING      ...  "false"
 - ROBOCORP_HOME                         ...  "/home/vagrant/.robocorp"
 - ROBOCORP_OVERRIDE_SYSTEM_REQUIREMENTS ...  "false"
 - config-active-profile                 ...  "default"
 - config-http-proxy                     ...  ""
 - config-https-proxy                    ...  ""
 - config-micromambarc-used              ...  "false"
 - config-piprc-used                     ...  "false"
 - config-settings-yaml-used             ...  "false"
 - config-ssl-no-revoke                  ...  "false"
 - config-ssl-verify                     ...  "true"
 - controller                            ...  "rcc.user"
 - cpus                                  ...  "4"
 - executable                            ...  "/nix/store/4s1mmhh1bznfhrpsyg17pl5ddbww57xl-rcc-v11.25.0/bin/rcc"
 - hololib-catalog-location              ...  "/home/vagrant/.robocorp/hololib/catalog"
 - hololib-library-location              ...  "/home/vagrant/.robocorp/hololib/library"
 - hololib-location                      ...  "/home/vagrant/.robocorp/hololib"
 - holotree-location                     ...  "/home/vagrant/.robocorp/holotree"
 - holotree-shared                       ...  "false"
 - holotree-user-id                      ...  "72de99c"
 - micromamba                            ...  "0.25.1"
 - micromamba.bin                        ...  "/home/vagrant/.robocorp/bin/micromamba"
 - no-build                              ...  "false"
 - os                                    ...  "linux_amd64"
 - os-holo-location                      ...  "/opt/robocorp/ht"
 - rcc                                   ...  "v11.25.0"
 - rcc.bin                               ...  "/nix/store/4s1mmhh1bznfhrpsyg17pl5ddbww57xl-rcc-v11.25.0/bin/rcc"
 - stats                                 ...  "6 requests, 0 merges, 0 hits, 0 dirty, 1 misses, 4 failures | 686dfaed-eeab-527e-f8d0-c5dbc70a9d2e"
 - telemetry-enabled                     ...  "false"
 - tempdir                               ...  "/tmp"
 - uid:gid                               ...  "1000:100"
 - user-agent                            ...  "rcc/v11.25.0 (linux amd64) rcc.user"
 - user-cache-dir                        ...  "/home/vagrant/.cache"
 - user-config-dir                       ...  "/home/vagrant/.config"
 - user-home-dir                         ...  "/home/vagrant"
 - when                                  ...  "2022-09-06T09:35:22+02:00 (CEST)"
 - working-dir                           ...  "/home/vagrant/Desktop/RCC"

Checks:
 - RPA      ok       ROBOCORP_HOME (/home/vagrant/.robocorp) is good enough.
 - OS       ok       PYTHONPATH is not set, which is good.
 - OS       ok       PLAYWRIGHT_BROWSERS_PATH is not set, which is good.
 - OS       ok       NODE_OPTIONS is not set, which is good.
 - OS       ok       NODE_PATH is not set, which is good.
 - OS       ok       Supports long enough paths.
 - network  ok       api.eu1.robocorp.com found: [34.255.74.152 52.215.73.48 54.77.77.237]
 - network  ok       cloud.robocorp.com found: [34.249.245.73 34.254.80.145 99.80.113.73]
 - network  ok       conda.anaconda.org found: [104.17.92.24 104.17.93.24 2606:4700::6811:5d18 2606:4700::6811:5c18]
 - network  ok       downloads.robocorp.com found: [108.156.22.86 108.156.22.55 108.156.22.128 108.156.22.67 2600:9000:2368:5e00:18:cd0d:3c00:93a1 2600:9000:2368:d800:18:cd0d:3c00:93a1 2600:9000:2368:f800:18:cd0d:3c00:93a1 2600:9000:2368:fe00:18:cd0d:3c00:93a1 2600:9000:2368:c000:18:cd0d:3c00:93a1 2600:9000:2368:9c00:18:cd0d:3c00:93a1 2600:9000:2368:ea00:18:cd0d:3c00:93a1 2600:9000:2368:7c00:18:cd0d:3c00:93a1]
 - network  ok       files.pythonhosted.org found: [151.101.1.63 151.101.129.63 151.101.193.63 151.101.65.63 2a04:4e42::319 2a04:4e42:200::319 2a04:4e42:400::319 2a04:4e42:600::319]
 - network  ok       github.com found: [140.82.121.4]
 - network  ok       id.robocorp.com found: [13.32.123.117 13.32.123.4 13.32.123.12 13.32.123.72 2600:9000:2375:7e00:8:29ad:bc00:93a1 2600:9000:2375:a400:8:29ad:bc00:93a1 2600:9000:2375:3000:8:29ad:bc00:93a1 2600:9000:2375:a000:8:29ad:bc00:93a1 2600:9000:2375:9e00:8:29ad:bc00:93a1 2600:9000:2375:0:8:29ad:bc00:93a1 2600:9000:2375:c600:8:29ad:bc00:93a1 2600:9000:2375:ca00:8:29ad:bc00:93a1]
 - network  ok       pypi.org found: [151.101.192.223 151.101.0.223 151.101.64.223 151.101.128.223 2a04:4e42:400::223 2a04:4e42::223 2a04:4e42:600::223 2a04:4e42:200::223]
 - network  ok       robocorp.com found: [172.67.7.153 104.22.40.65 104.22.41.65 2606:4700:10::6816:2841 2606:4700:10::6816:2941 2606:4700:10::ac43:799]
 - network  ok       telemetry.robocorp.com found: [18.200.53.250 52.51.244.219]
 - network  ok       Canary download successful: https://downloads.robocorp.com/canary.txt
 - network  ok       PyPI canary download successful: https://pypi.org/jupyterlab-pygments/
 - network  ok       Conda canary download successful: https://conda.anaconda.org/conda-forge/linux-64/repodata.json
 - Settings ok       Toplevel settings are ok.
OK.
vjmp commented 2 years ago

@datakurre

Can you share how you are setting up that chroot shell, so that we are able to repeat this. And I hope it does not require nix environment.

Could you also try to give path to robot.yaml from commandline: rcc task shell --robot path/to/robot.yaml and see if that makes difference?

Other thing to try out is this: https://github.com/robocorp/rcc/blob/master/docs/recipes.md#how-to-activate-holotree-environment

datakurre commented 2 years ago

@vjmp

Thanks for the tips!

There's too much nix in the chroot sandbox, but rcc holotree variables did show the reason for broken shell:

$ rcc holotree variables --space mine /home/vagrant/Desktop/RCC/conda.yaml
export PYTHON_EXE=/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/bin/python3
export CONDA_DEFAULT_ENV=rcc
export CONDA_PREFIX=/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4
export CONDA_PROMPT_MODIFIER=(rcc) 
export CONDA_SHLVL=1
export PYTHONHOME=
export PYTHONSTARTUP=
export PYTHONEXECUTABLE=
export VIRTUAL_ENV=/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4
export PYTHONNOUSERSITE=1
export PYTHONDONTWRITEBYTECODE=x
export PYTHONPYCACHEPREFIX=/home/vagrant/.robocorp/temp/4d65822107fc5f74
export ROBOCORP_HOME=/home/vagrant/.robocorp
export RCC_ENVIRONMENT_HASH=fed4177fee2e86eb
export RCC_INSTALLATION_ID=686dfaed-eeab-527e-f8d0-c5dbc70a9d2e
export RCC_TRACKING_ALLOWED=false
export RCC_EXE=/nix/store/2i789311n1z5rnazand7zavxkbg6p78n-rcc-v11.25.0/bin/rcc
export RCC_VERSION=v11.25.0
export TEMP=/home/vagrant/.robocorp/temp/4d65822107fc5f74
export TMP=/home/vagrant/.robocorp/temp/4d65822107fc5f74
export PATH=/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/bin:/run/wrappers/bin:/usr/bin:/usr/sbin:/run/wrappers/bin:/home/vagrant/.nix-profile/bin:/etc/profiles/per-user/vagrant/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
export PS1=
export SHLVL=4
export _=/nix/store/4s1mmhh1bznfhrpsyg17pl5ddbww57xl-rcc-v11.25.0/bin/rcc
export CONDA_BACKUP_GOROOT=
export GOROOT=/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/go
export MAMBA_ROOT_PREFIX=/home/vagrant/.robocorp
export MAMBA_SSL_VERIFY=false
export PATH=/home/vagrant/Desktop/RCC/bin:/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/bin:/run/wrappers/bin:/usr/bin:/usr/sbin:/run/wrappers/bin:/home/vagrant/.nix-profile/bin:/etc/profiles/per-user/vagrant/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
export PYTHONPATH=
export ROBOT_ROOT=/home/vagrant/Desktop/RCC
export ROBOT_ARTIFACTS=/home/vagrant/Desktop/RCC/output

The shell is broken, because of line export PS1= even that variable is not empty in the parent shell calling RCC. I'm not sure, where RCC inherits that from, but if I patch RCC to set PS1 to any value, shell starts working.

datakurre commented 2 years ago

I can now patch this to work for me, and can close this issue.

I have no idea, where go gets that PS1=, but feel free to re-open, if you want to try to figure that out.

datakurre commented 2 years ago

Oh, it is coming from the holotree

$ cat /home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/rcc_activate.json 
{
  "CONDA_BACKUP_GOROOT": "",
  "GOROOT": "/home/vagrant/.robocorp/holotree/72de99c_5a1fac3_0404cee4/go",
  "MAMBA_ROOT_PREFIX": "/home/vagrant/.robocorp",
  "PS1": "",
  "SHLVL": "4",
  "_": "/nix/store/4s1mmhh1bznfhrpsyg17pl5ddbww57xl-rcc-v11.25.0/bin/rcc"
}
vjmp commented 2 years ago

@datakurre

If it is that, then that is coming from some dependency, that is setting it. What does your conda.yaml looks like?

That rcc_activate.json is just "diff" of "vanilla" and activated environment ... (the addition ...)

datakurre commented 2 years ago

@vjmp

The content of conda.yaml does not seem to matter.

I tried to debug the creation of rcc_activate.json:

  1. PS1 is present at the captured output before the activation
  2. PS1 is no longer present at the capture output after the activation
  3. therefore the diff is to unset PS1 by setting it to empty string in rcc_activate.json, breaking shell

And the difference in 1. and 2. is probably because 2. is generated through a rcc_activate.sh script, and bash may not export PS1 in non-interactive shell like scripts.

I can "reproduce" this myself with my bash by echo $PS1 echoing from prompt, but not from a script (unlike other exported environment variables, which always echo).

But why, oh why, this happens only on my bash + rcc :flushed:

(But as said, I have now patched this for myself šŸ˜Š)

datakurre commented 2 years ago

@vjmp You were correct. I just misinterpreted emptied PS1= environment variable as broken shell. I retried now, and shell was properly activated. Just with empty prompt. I thought it was broken, because empty enter` did not add new line, but never tried entering commands :see_no_evil:

Thanks again.