KhiopsML / khiops

Khiops is an AutoML suite for supervised and unsupervised learning
https://khiops.org
BSD 3-Clause Clear License
34 stars 4 forks source link

Fix slice memory estimation #369

Closed folmos-at-orange closed 2 months ago

folmos-at-orange commented 2 months ago

La méthode ComputeGlobalSlaveNecessaryMemory, dont le deuxième paramètre a changé radicalement de valeur, est utilisée plusieurs fois dans le code, avec une variation de paramètre (MemSegmentByteSize, BufferedFile::nDefaultBufferSize).

Et réciproquement, elle utilise pour son implémentation d'autres méthodes internes (ComputeRecodingObjectsNecessaryMemory, ComputeSliceSetTotalReadBufferNecessaryMemory, ComputeSliceSetTotalReadBufferNecessaryMemory).

Il faudrait vérifier si le changement proposé ne doit pas être propagé à d'autres méthodes.

The function it is used in two places:

In MasterInitializeDataTableBinarySliceSet it was called with MemSegmentByteSize but this increases the necessary memory with respect to BufferedFile::nDefaultBufferSize (see the comment in the code) . So it must be called with the BufferedFile:nDefaultBufferSize so we reach the lower bound (SetMin) of the requested memory.

folmos-at-orange commented 2 months ago

I ran LearningTest:

folmos-at-orange commented 2 months ago

Added after offline discussion: Silence the removal of temporary dictionary files