search

KhiopsML / khiops-python

The Python library of the Khiops AutoML suite
https://khiops.org
BSD 3-Clause Clear License
10 stars 1 forks source link

Separate conda package in two #168

Closed folmos-at-orange closed 7 months ago

folmos-at-orange commented 8 months ago

Description

Currently, the conda package contains the python code and the Khiops binaries. This makes building times very long as it recompiles khiops for each python version even though it is not necessary. Separating the binaries would speed up the tests and the release procedure.

Questions/Ideas

popescu-v commented 8 months ago

Yes, the khiops-core package would be pushed to the relevant Conda channels (dev or release) and "pulled" by the khiops (-python) Conda package upon install, as a dependency.

Thus, instead of using the Git tag of the khiops repository for controlling which version of the Khiops binaries will be used, we would use the relevant version of the khiops-core package. This entails the minor hassle of making sure a relevant version of the khiops-core package (sometimes based on the dev branch of the Khiops Git repository) is pushed to the khiops-dev Conda channel.

folmos-at-orange commented 8 months ago

Further simplifications: The LearningTest infra will only test 2 conda packages (Linux and Windows).

sgouache commented 8 months ago

Great initiative! But are you sure we won't end up having NxM khiops-core packages ? (N=number_of_os_versions, M=number_of_python_versions) See ref

folmos-at-orange commented 8 months ago

Great initiative! But are you sure we won't end up having NxM khiops-core packages ? (N=number_of_os_versions, M=number_of_python_versions) See ref

This should resolve the issue of having to test different "python versions" for binaries that do not use any of the Python API.

sgouache commented 8 months ago

I know we're not testing any python. What I'm concerned about is the availability of the MPI(CH) shared library for each python version. As we've experienced, depending on the chosen channel, not all python versions have the same MPICH versions available. Unless we can depend on a MPI conda package that is "variant-free", I guess it will look like: