venv: site-packages path is incorrect

[nihilox]

root@f310d0f1dfc9:/tmp/demo# python3.10 -m venv venv
root@f310d0f1dfc9:/tmp/demo# source venv/bin/activate

(venv) root@f310d0f1dfc9:/tmp/demo# python -I -m site
sys.path = [
    '/tmp/.mount_pythonCEi1JB/opt/python3.10/lib/python310.zip',
    '/tmp/.mount_pythonCEi1JB/opt/python3.10/lib/python3.10',
    '/tmp/.mount_pythonCEi1JB/opt/python3.10/lib/python3.10/lib-dynload',
    '/tmp/.mount_pythonCEi1JB/opt/python3.10/lib/python3.10/site-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.10/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

(venv) root@f310d0f1dfc9:/tmp/demo# python -m site
sys.path = [
    '/tmp/demo',
    '/tmp/.mount_pythonUgmxt7/opt/python3.10/lib/python310.zip',
    '/tmp/.mount_pythonUgmxt7/opt/python3.10/lib/python3.10',
    '/tmp/.mount_pythonUgmxt7/opt/python3.10/lib/python3.10/lib-dynload',
    '/tmp/demo/venv/lib/python3.10/site-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.10/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

Which should always be /tmp/demo/venv/lib/python3.10/site-packages.

[niess]

Hello @nihilox,

Thank you for reporting this issue.

Unfortunately I see no good solution right now in order to recover 100% native Python/venv behaviour with AppImages. However, there is a simple workaround. The problem is actually related to -E (i.e. ignoring PYTHON* environment variables). If isolation is done manually, e.g. as:

PYTHONPATH= python -s -m site

then sys.path is properly set.

The difficulty is that venv seems to rely on symbolic links, at least on Linux. My understanding is that venv site-packages location is determined from the link origin (i.e. sys.executable). But, when using an AppImage, Python is not called directly, it is mounted at AppImage start to a temporary directory, and then it is called through an AppRun wrapper script. In order to mimic a direct call through a symbolic link, I use the exec command in the AppRun script, e.g. as

(exec -a "$origin" "$target" "$@")

where $target is the actual Python runtime, inside the mounted AppImage. This properly sets sys.executable to $origin. However, when sys.executable differs from $target, then Python reads sys.base_prefix from its (hardcoded) initial installation location, e.g. /usr, which is no more valid once relocated, e.g. when using an AppImage. Therefore, I instead use the following command:

(PYTHONHOME="$base_prefix" exec -a "$origin" "$target" "$@")

where $base_prefix points to the mounted AppImage, i.e. $target = $base_prefix/bin/pythonX.Y. This trick seems to work fine in most cases. But, if the -E option is used, then it obviously fails since PYTHONHOME is ignored. This case is detected by the AppRun wrapper script. When -E is used, then the wrapper directly calls the Python executable inside the AppImage, as

"$target" "$@"

Thus, sys.executable points to $target. This is OK for native calls. However, it fails when running inside venv. Then, sys.path is initialised according to $target, instead of $origin.

As stated initialy, I do not have any good solution for now. The hacks that I have implemented for venv have some unsatisfactory side effects (e.g. spoiling PYTHONHOME), and anyway they do not work at 100%. On the other side, I like the fact that sys.executable points to the AppImage executable (or to a corresponding symbolic link), rather than to a temporary mount point.

[niess]

@nihilox ,

I have updated Python AppImages over the WE. They should now behave properly w.r.t. venv. Therefore, I am closing this issue for now. But, please feel free to re-open it if you still observe malfunctioning.

Technically speaking, I now patch sys.executable directly at Python runtime. This happens after getpath has initialised sys.prefix, etc., but before the execution of site.main. The latter sets sys.path according to the result of getpath, and depending on venv. With this new method, I don't need to trick Python anymore with the exec command and using PYTHONHOME. Thus, Python AppImage should now behave close to native Python. However, in order to do so, I had to slightly modify core Python.

Note that as a side effect, it also patches the issue with pip not being initialised at venv creation. Now it should work properly.