KhiopsML / khiops-python

The Python library of the Khiops AutoML suite
https://khiops.org
BSD 3-Clause Clear License
8 stars 1 forks source link

Handle OpenMPI / MPICH switching within the Khiops local runner #183

Closed popescu-v closed 2 weeks ago

popescu-v commented 1 month ago

Description

Currently, the KhiopsLocalRunner assumes the usage of MPICH / MSMPI with its specific command arguments. However, Khiops (the binaries) recently added OpenMPI support for non-Conda environments (see https://github.com/KhiopsML/khiops/pull/252). Thus, khiops-python should enable switching between MPICH and OpenMPI, according to the MPI library the MODL* binaries have been compiled with / for.

Questions/Ideas

popescu-v commented 1 month ago

After discussion, the first solution would be chosen in the short run. The third solution could be investigated for later, as we need to make sure it also works on Windows, as well as on Conda-like environments in all OSes.

popescu-v commented 3 weeks ago

We also need to update the khiopspydev Docker images to install and use OpenMPI instead of MPICH.

popescu-v commented 3 weeks ago

Detailed implementation plan:

  1. Change MPICH to OpenMPI in the Ubuntu and Rocky khiopspydev Dockers.
  2. Replace local compilation of Khiops MODL* binaries with wget from the pre-releases (KHIOPS_REVISION must be a Git tag)
  3. Add distinct Conda environments to these Dockers: for each Python version, there will be 2 Conda environments:
    • one for the current tests (based on system-wide Khiops + khiops-python repo clone)
    • one for new tests, based on Conda-installed khiops-core (on KHIOPS_REVISION) + khiops-python repo clone
  4. Add "pure" Conda tests in unit-tests.yml:
    • keep current tests on Conda environments which use system-wide Khiops (+ system-wide MPI - will be OpenMPI - see 1.)
    • add tests on the Conda environments which contain khiops-core installed.