on windows + venv, the dll search path modification is broken

mattip commented 1 year ago

Solution to issue cannot be found in the documentation.

[X] I checked the documentation.

Issue

Using python -m venv destination and then running destination\script\python -c "import sqlite3" fails. Turning on set CONDA_DLL_SEARCH_MODIFICATION_DEBUG=1 show why: the paths inside the venv are added instead of the ones from the base python

> set CONDA_DLL_SEARCH_MODIFICATION_DEBUG=1
> destination\script\python -c "import sqlite3"
CondaEcosystemModifyDllSearchPath_Init() :: os.add_dll_directory(destination\Scripts\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath_Init() :: os.add_dll_directory(destination\Scripts\Scripts - ExePrefix)
CondaEcosystemModifyDllSearchPath_Init() :: os.add_dll_directory(destination\Scripts\Library\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath_Init() :: os.add_dll_directory(destination\Scripts\Library\usr\bin - ExePrefix)
CondaEcosystemModifyDllSearchPath_Init() :: os.add_dll_directory(destination\Scripts\Library\mingw-w64\bin - ExePrefix)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "d:\miniconda\envs\pypy38\Lib\sqlite3\__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "d:\miniconda\envs\pypy38\Lib\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
  File "d:\miniconda\envs\pypy38\Lib\_sqlite3.py", line 53, in <module>
    from _sqlite3_cffi import ffi as _ffi, lib as _lib
ImportError: The specified module could not be found

### Installed packages

```shell
not relevant

Environment info

not relevant

saaketp commented 1 year ago

hopefully, it should be as simple as using sys.base_prefix here instead of sys.prefix

https://github.com/conda-forge/pypy3.6-feedstock/blob/5a6375058645441284cef023c25957b40b812506/recipe/patches/0014-Add-CondaEcosystemModifyDllSearchPath.patch#L17-L21

mattip commented 1 year ago

The parallel cpython patch looks for the directory of the python.dll loaded into the process. For PyPy, since we are doing this in app-level python that translation bakes into the executable, sys.base_prefix would be easier.

mattip commented 1 year ago

The whole problem is a bit more complicated.

A venv from a conda env needs the system PATH to include the Library\bin from the conda env, otherwise PyPy cannot start up (it is missing libexpat.dll, libbz2.dll, libffi-8.ddl). I think this is done on CPython with a py.exe launcher.
Somehow the conda environment set up by the build is different from the one shipped, so the test that passes on a shipped PyPy fails in PR #95 during the build. The diffferentiating factor is where the stdlib files are found (PyPy searches for site.py)

mattip commented 1 year ago

Perhaps the first problem should be handled by the activate.bat script, which could modify the PATH

mattip commented 1 year ago

the second point was wrong: there was a bug in the PR that was fixed by a subsequent change (thanks @isuruf). While the new version works, the first point still holds. You can only use the virtualenv while the original conda environment is still activated.

conda-forge / pypy3.6-feedstock