Scenarios' `MaxCoreNumber`, `MemoryLimit` and `TemporaryDirectoryName` `AnalysisSpec.SystemParameters` Are Set via Environment-set Values

[popescu-v]

Description

Currently, Khiops scenarios's MaxCoreNumber, MemoryLimit and TemporaryDirectoryName SystemParameters are set either from a custom GeneralOptions instance, or from KhiopsRunner's general_options attribute. But, the GeneralOptions's attributes max_cores, max_memory_mb and khiops_temp_dir are initialized from khiops_env KHIOPS_PROC_NUMBER, KHIOPS_MEMORY_LIMIT and KHIOPS_TMP_DIR attributes, respectively.

However:

• KHIOPS_PROC_NUMBER is just an internal variable of khiops_env which is used for setting the KHIOPS_MPI_COMMAND variable; thus, this sets the upper bound of the number of CPU cores that Khiops would use in the current session;
• KHIOPS_MEMORY_LIMIT is relayed to the MODL* binaries and sets the upper bound of the number of MBs that Khiops would use in the current session;
• KHIOPS_TMP_DIR is relayed to the MODL* binaries and sets the default temporary directory that Khiops would use in the current session.

Hence, these are global settings which set bounds for any processing done by Khiops in the current working session.

However, Khiops scenarios' parameters SystemParameters.{MaxCoreNumber,MemoryLimit,TemporaryDirecytoryName} are local to each processing (i.e. to each Core API call), hence:

• they should be set via the Khiops Core API calls
• the expectation is that their values are lower than or equal to the global settings.

Questions/Ideas

• All Khiops Core API functions defined in khiops/core/api.py should take three new keyword arguments:
  • max_core_number: int
  • memory_limit: int
  • tmp_dir: str
• the KhiopsRunner._write_scenario_prologue should not write into AnalysisSpec.SystemParameters.{MaxCoreNumber,MemoryLimit,TemporaryDirectoryName} from the general_options anymore, but from task_args, as these are "task"-level arguments.

[folmos-at-orange]

So, these parameters will be part of the official API ? Not only for Python ?

[folmos-at-orange]

One detail: When executing a large quantity small tasks is often convenient to execute with just 1 CPU because the overhead coming from the MPI process creation dominates the execution time. This overhead is worse on machines with many CPUs.

So when an API with max_cores set to 1 or 2 I think MPI should not be used on execution.