Publish to conda-forge

[alexus37]

Hi, thanks for this useful project. I was wondering if you could also publish the package to conda-forge so it is possible to use it in conda_env.yaml files.

name: my_env

channels:
  - pytorch
  - conda-forge

dependencies:
  - python=3.7
  - pip
  - cmake_build_extension

Thanks!

[diegoferigo]

Hi @alexus37, thanks for opening this issue! In the past few months in the organization I'm working we started being more and more interested in conda and conda-forge, and we started (mainly @traversaro) to package our software for conda usage. My knowledge on packaging is still limited, but I'm getting acquainted with the process and in many cases is quite straightforward, especially for pure-Python projects like this one.

Though, for this particular case, I'm a bit skeptical. The aim of this setuptools extension is smoothing the experience of hybrid C++ / Python projects of integrating them with tools that comply with PEP517 and PEP518 (like pip and pypa/build). The package generated by these tools are installed in the active site-packages folder (that could be system, virtual environment, or also a conda environment), but the exported CMake project is bundled in the same site-packages/<package_name> folder. In order for another package to import the CMake project and consume the targets, either it has to use similarly cmake-build-extension with the cmake_depends_on option, or the user has to manually add an entry to CMAKE_PREFIX_PATH=[...]:<path/to/site-packages>/package_name.

Conda, AFAIK, has a different approach. The installation prefix of hybrid C++ / Python projects is different for the two languages. C++ resources are installed in $CONDA_PREFIX/lib where all the other environment libraries are installed, and CMake's find_package resources are active by default. Python resources, instead, go similarly in $CONDA_PREFIX/lib/pythonX.Y/site-packages.

This being said, it seems to me that using cmake-build-extension for a conda package kind of defeats the purpose of conda itself. Maybe I'm not expert enough with the conda ecosystem (I'm definitely not), but can you elaborate a bit more on your use case? My fear is to trigger a pattern that, in my limited knowledge, does not really comply well with the approach that should be used. Maybe here an opinion of @traversaro could be enlightening.

I'm not against a priori to bring cmake-build-extensions to conda-forge, but since then I'd also have to maintain it (regardless of the nice automation they have in place), I'd like first to understand if there's actually a reason for that effort. For instance, we do maintain hybrid projects both in PyPI and conda-forge (or custom conda channel), and we have the need of cmake-build-extension only in the former case. But again, maybe it's my knowledge that is still too limited. This discussion is definitely interesting, and I'd like to hear your opinion.

[alexus37]

Hi, thanks for the elaborate answer. First I want to state that I am not a packaging expert and I decided to fork the lib and modify it to my needs. However, my initial plan was the following:

1. I am using conda to create an env for my project. To do this I created a YAML file

   
   name: myEnv

channels:

• conda-forge

dependencies:

• python=3.7
• pip
• pybind11
• cmake_build_extension
• pip:
  • -r requirements.txt

2. This installs all dependencies and runs pip afterwards. In the requirements.txt I have a local submodule that would require cmake_build_extension and is installed as a local link.

-e lib/my_submodule

Since my my_submodule requires cmake_build_extension to be installed I thought it makes sense to install it via conda forge first.

conda env create -f conda_env.yaml

However, there might be better ways to achieve this.

Feel free to close the issue

[diegoferigo]

Ok I see, thanks for the additional details. The main problem to me does not seem the lack of cmake_build_extension in conda, rather the usage of -e in your requirements.txt. In fact, editable installation do not pull any dependency of the local submodule even if they are listed either in your setup.py or your setup.cfg.

Without changing your pattern, what I would suggest is to use the YAML file to prepare the environment in which you then install your module in editable mode. This means:

1. Removing the -e lib/my_submodule from requirements.txt
2. Add cmake-build-extension to requirements.txt
3. After creating the conda environment, run pip install -e .

---

I'm not a big fan of requirements.txt and I never used it personally (I believe that a well structured setup.cfg is a better pattern). I'm not sure if the order of packages is kept. If yes, alternatively to what I proposed above, you maybe could update it to something similar:

cmake-build-extension
# [...] other packages that may or may not require cmake-build-extension
-e lib/submodule

[diegoferigo]

What discussed above remains valid, though I might have found an interesting use case to have cmake_build_extension in conda-forge #25. Therefore, I reopen the issue.

Edit: also, packages that use our context manager to fix Windows imports must have cmake-build-extension installed in their system. Therefore, it's no longer a build-only requirement (that could be handled directly by pip during conda packaging) but it's a runtime dependency. In this case, having it in conda-forge would be necessary.

[diegoferigo]

Forgot to comment here. After https://github.com/conda-forge/staged-recipes/pull/16572, cmake_build_extension is available in conda-forge :rocket:

Check the resulting feedstock for more details: https://github.com/conda-forge/cmake-build-extension-feedstock/