RP doesn't work on Debian 11 and above with default (localhost) configuration | Cause -> python not found to create server-agent env

From the radical.pilot.sandbox/rp.session.debian-x.workstation.019030.0005/pilot.0000/bootstrap_0.out

./bootstrap_0.sh: line 476: -c: command not found
PYTHON            : 
PYTHON_VERSION    : 
PIP               :  -m pip
./bootstrap_0.sh: line 514: -m: command not found
PIP version       : 
1644192983.9382,ve_setup_start,bootstrap_0,MainThread,pilot.0000,pilot_state,
virtenv_create    : TRUE
virtenv_update    : FALSE
rp install sources:  radical.gtod-1.6.7/ radical.utils-1.11.0/ radical.saga-1.11.1/ radical.pilot-1.11.2/
rp install target : SANDBOX
rp install lock   : FALSE
rp lock for virtenv create
obtained lock /home/workstation/radical.pilot.sandbox/ve.local.localhost.1.11.2.lock
1644192984.0048,ve_create_start,bootstrap_0,MainThread,pilot.0000,pilot_state,

# -------------------------------------------------------------------
#
# Download virtualenv tgz
# cmd: curl -1 -k -L -O 'https://files.pythonhosted.org/packages/66/f0/6867af06d2e2f511e4e1d7094ff663acdebc4f15d4a0cb0fed1007395124/virtualenv-16.7.5.tar.gz'
#
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0 4992k    0 41365    0     0  42349      0  0:02:00 --:--:--  0:02:00 42338^M100 4992k  100 4992k    0     0  2986k      0  0:00:01  0:00:01 --:--:-- 2987k
#
# SUCCESS
#
# -------------------------------------------------------------------

# -------------------------------------------------------------------
#
# unpacking virtualenv tgz
# cmd: tar zxmf 'virtualenv-16.7.5.tar.gz'
#
#
# SUCCESS
#
# -------------------------------------------------------------------

# -------------------------------------------------------------------
#
# Create virtualenv
# cmd:  virtualenv-16.7.5/virtualenv.py -p python3 /home/workstation/radical.pilot.sandbox/ve.local.localhost.1.11.2
#
/usr/bin/env: ‘python’: No such file or directory
#
# ERROR
# no fallback command available
#

Essentially RP assumes that python would be on the path for creating a virtual env in the sandbox. Problem: Debian 11 doesn't has a python instead has only python3 (i.e. /usr/bin/python3) Quoting from https://wiki.debian.org/Python

NOTE: Debian 11 (bullseye) has removed the "python" package and the '/usr/bin/python' symlink due to the deprecation of Python 2. No packaged scripts should depend on the existence of '/usr/bin/python': if they do, that is a bug that should be reported to Debian. You can use the 'python-is-python3' or 'python-is-python2' packages to restore an appropriate '/usr/bin/python' symlink for third-party or legacy scripts.

Ofcourse when the package python-is-python3 is installed or I manually create a symlink of python to python3, it works fine.

My Local Environment - Debian 11 but currently on the testing branch System's python - 3.9.10 which gets used on sandbox (server-agent) Tested on the master branch virtualenv with Python 3.6.15 on client-side resource_local.json -> localhost

Questions ->

What OS do we usually deploy on, also intend to support in the future? Is Debian-based OS's even there?
If we test on any debian derivative like Ubuntu, then what version of ubuntu?

I have not seen Debian breaking things in the past this way, so ideally we should not fix anything. But IMO, since Debian did it, sooner other distros (especially Debian family ones) might eventually do the same.

Actionables? -

Documenting the issue at least somewhere, maybe https://radicalpilot.readthedocs.io/en/stable/faq.html. If yes, where?
Does doing any changes to the deployment process makes sense or worth it?

This was the reason I was getting KeyboardInterrupt on the client side.

worksta+ 1010848  890558  6 19:16 pts/12   00:00:01  |   \_ python examples/00_getting_started.py                                                                                                                   
worksta+ 1011848 1010848  0 19:16 pts/6    00:00:00  |       \_ /bin/bash -i
worksta+ 1011958 1010848  0 19:16 pts/10   00:00:00  |       \_ /bin/sh -i                                                                                                                                          
Traceback (most recent call last):                                                                                                                                                                                  
File "examples/00_getting_started.py", line 100, in <module>           
tmgr.wait_tasks()                                                               
File "/home/workstation/.local/share/virtualenvs/radical-pilot-env/lib/python3.6/site-packages/radical/pilot/task_manager.py", line 1015, in wait_tasks
time.sleep (0.1)                                                                                        
KeyboardInterrupt

radical-cybertools / radical.pilot

RP doesn't work on Debian 11 and above with default (localhost) configuration | Cause -> python not found to create server-agent env #2498