elastic / ml-cpp

Machine learning C++ code
Other
149 stars 62 forks source link

[ML] Investigate/fix reported memory usage #2664

Closed jan-elastic closed 1 month ago

jan-elastic commented 3 months ago

Anomaly detection uses more memory than is reported by the system (at least on MacOS).

E.g. for a job from the QA test suite, the anomaly detection system reports:

        "model_bytes": 257363572,
        "peak_model_bytes": 257363572,

while MacOS reports (via activity monitor / top / ps) that the autodetect process uses up to approximately 1GB.

Additionally, ECK shows a gradual increase in memory consumed by pods containing the autodetect process.

Screenshot from 2024-06-04 12-51-40