Open sdebionne opened 2 months ago
sdebionne
commented
2 months ago Here is the repro that I had in mind, but unfortunately it does not repro. The real project that shows the issue depends on many third party dependencies so the dynamic of the installation is different.
sdebionne
commented
2 months ago Since I cannot provide a repro easily (but it still shows the structure of the project), I can provide a real world repro:
$ conda --version
conda 23.11.0
$ conda create -n test lima-camera-simulator-tango
...
$ ll $CONDA_PREFIX/lib/python3.10/site-packages/Lima/Server/camera
total 32
-rw-rw-r-- 2 Jul 1 08:42 __init__.py
-rw-rw-r-- 2 Oct 19 2022 Simulator.py <------ PLUGIN OK$ conda --version
conda 24.4.0
$ conda create -n test lima-camera-simulator-tango
...
$ ll $CONDA_PREFIX/lib/python3.10/site-packages/Lima/Server/camera
total 32
-rw-rw-r-- 2 1676 Jul 1 08:42 __init__.py
<------ MISSING PLUGINEnvironments are identicals with both versions of Conda.
sdebionne
commented
2 months ago Investigating further, with Conda >= 24, the plugin file (here Simulator.py) is actually installed in the wrong folder:
$CONDA_PREFIX/site-packages/Lima/Server/camera/Simulator.pywhile it should be
$CONDA_PREFIX/lib/python3.10/site-packages/Lima/Server/camera/Simulator.pye.g. the site-packages folder is not properly prefixed.
travishathaway
commented
2 months ago @sdebionne,
Is there anyway you could link to the project you are having difficulties with? This would help us debug what is currently happening.
From what I have read so far, I think that this is less of a problem with conda and more of a problem with the way this program is being packaged. For example, it's bad practice for one package to modify the contents of another like this. Each individual package, even if they are plugins, should be contained to themselves.
The reason for this is that it helps ensure package integrity. This can be important for security reasons because you would want to know when a package has been tampered with. For example, after adding new files to a package, the checksum value of the package at the time it was built would no longer match.
Regardless, it would really help to see the recipes for these packages to see exactly what's going on.
sdebionne
commented
2 months ago Thank you for your feedback @travishathaway,
You are right, we should improve our plugin discovery mechanism, so that plugin should not need to be installed in the tree of the main project. In the other hand there is clearly a change of behavior from conda 23 to 24, where our packages stopped to install properly.
While investigating, I found out that the same package does not install the same way with conda 24: it misses the site-packages prefix so that site-packages get copied directly in $CONDA_PREFIX and not in lib\python<version>
While trying to find a workaround, I noticed that switching from a CMake install (that would basically copy a single python file to the proper folder in site-packages) to a pip/setuptools install, fixed the issue. The main difference I see is a direct (build) dependency to python but maybe setuptools generates metadata that are now mandatory to get a "true" noarch python package.
The old package was plugin-project-1.10.0-0.tar.bz2 while the new one is (mind the py) plugin-project-1.10.0-py_1.tar.bz2. So I wonder if, while both recipes use noarch: python, the one without python as direct dependency, did not actually build correctly.
sdebionne
commented
2 months ago The plugin project and recipe is in https://github.com/esrf-bliss/Lima-camera-simulator/blob/develop/conda/tango , but I am afraid it is a bit "opaque"...
travishathaway
commented
2 months ago Thanks for the link to your project.
At this point, I am open to the possibility that a regression may have been introduced in the 24.1.0 release. But, considering that you have found a work around, I will not make this issue a priority.
One change that could come out of this though is to improve the conda-build documentation to warn against the method you were initially using.
Let's leave this discussion open though in case others in the community are facing similar issues.
jaimergp
commented
1 month ago Looking at the recipe, you definitely need python in that meta.yaml requirements. It's good practice to be explicit about the direct dependencies. I'd add a host section with python and pip (and lower bounds >=x.y as required by your project), and also add python >=x.y to run.
Shared namespace packages are supported and you can see some examples by examining the backports family, but that CMake config is maybe a bit non standard to follow the assumptions conda-build makes to support them?
jaimergp
commented
1 month ago Do you have a link to the artifacts themselves? Maybe there's something wrong with their "noarchness", because the plugin is being extracted its "noarch path" (starts with site-packages) instead of the "expanded path" (lib/pythonx.y/site-packages).
sdebionne
commented
1 month ago Sorry for the late answer, here is a link to the artifacts. I am trying to add the python as direct dependency to check whether that helps with the "noarch-pythoness".
jaimergp
commented
1 month ago I took a look and these are my findings. I can't reproduce your issue with latest conda.
This is what I did, assuming the artifacts have been downloaded to a local lima-channel directory.
$ conda create -n lima python=3.10 --platform=linux-64
$ conda activate lima
$ conda install lima-channel/linux-64/lima-camera-simulator-1.10.0-py310h2bc3f7f_0.tar.bz2 lima-channel/noarch/lima-camera-simulator-tango-1.10.0-0.tar.bz2
$ tree $CONDA_PREFIX/lib/python3.10/site-packages/Lima/
/opt/conda/envs/lima/lib/python3.10/site-packages/Lima/
├── Server
│ └── camera
│ └── Simulator.py # <------- this is the file we are after, right?
└── Simulator
├── Simulator.py
├── __init__.py
└── __pycache__
├── Simulator.cpython-310.pyc
└── __init__.cpython-310.pyc
4 directories, 5 filesThis looks good indeed. With the same artifact uploaded on Anaconda, why would I get a different results:
$ conda --version
conda 24.4.0
$ conda create -n lima-tmp
$ conda activate lima-tmp
$ conda install -c esrf-bcu lima-camera-simulator-tango=1.10.0=0
$ tree $CONDA_PREFIX/lib/python3.10/site-packages/Lima/ # edited output for better readability
/miniconda3/envs/lima-tmp/lib/python3.10/site-packages/Lima
├── Server
│ ├── camera
│ │ ├── __init__.py # <------- file is missing
│ │ └── __pycache__
│ │ └── __init__.cpython-310.pyc
$ tree $CONDA_PREFIX/site-packages/
/miniconda3/envs/lima-tmp/site-packages/
└── Lima
└── Server
└── camera
├── __pycache__
│ └── Simulator.cpython-310.pyc
└── Simulator.py # <------- found it but not in expended pathI intentionally did not install python in the env as it is an (indirect) dependency of the package.
sdebionne
commented
1 month ago Same result with conda 24.5.0
jaimergp
commented
1 month ago Yes, I understand why now.
The direct path/URL works because the solver doesn't have to be involved and conda parses the internal JSON metadata directly, which correctly points out that it's noarch: python.
With conda-libmamba-solver and libmamba v1, we lose the noarch information on they way out from the solver, and have to infer it from the subdir. But there are two types of noarch: python and generic. We rely on python being present in the run dependencies (as it should) to identify noarch: python. But since your package doesn't have it, it misses that, and considers it generic. See code here.
This won't be a problem with libmamba v2, because we do get the full noarch information without having to infer anything, but for now we need to be a bit more strict with the run requirements.
Sorry it took so long for me to realize it was about that!
sdebionne
commented
1 month ago Sorry it took so long for me to realize it was about that!
Awesome support, thank you!
jaimergp
commented
1 month ago I'll move this to conda-libmamba-solver because it's a "bug" there, not in conda. --solver=classic should work, btw.
Checklist
What happened?
We have packages that are actually plugins of another package. Up to conda < 24, installing the main project and the plugin worked fine. With conda >=24, we have the feeling that there is a race condition and that the plugin folder get recreated/overwritten by the main project
With two package containing:
project:site-packages/project/plugins/__init__.pyproject-bar-plugin:site-packages/project/plugins/bar.pyRunning
conda install project project-bar-pluginreports not error but thesite-packages/project/plugins/does not containsbar.py.I'll attach a repro ASAP.
Conda Info
Conda Config
Conda list
Additional Context
No response