worksforme model training task is failing (probably OOM) (hashtag4095)

[vikramsubramanian]

)

[mayil-ai[bot]]

Possible Solution

• Increase the workerType to a larger instance that has more memory for the worksforme_model_training task in the infra/data-pipeline.yml file.
• If the workerType is already at the maximum available, optimize the model training code to use less memory, for example by reducing the batch size or using memory-efficient data structures and algorithms.
• Check for memory leaks in the worksforme_model_training function and fix them if any are found.
• Implement checkpointing and restore capabilities in the training process to recover from intermediate states in case of failure due to OOM.
• If applicable, enable swap memory on the worker to provide a buffer for memory spikes, but be aware that this can lead to performance degradation.
• Monitor memory usage during the training process to identify when and where the OOM issue occurs, and adjust the training parameters accordingly.

Code snippets to check

infra → data-pipeline.yml

This snippet defines the task for training the test failure model, which is likely related to the issue as it specifies the worker type and resources that could be causing the OOM failure. https://github.com/Mayil-AI/bugbug-21dec23/blob/0acd00da46afbb37bd047c0bce06ce7cfad21568/infra/data-pipeline.yml#L1087-L1508