search

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
8 stars 3 forks source link

Associations upload memory leak? #710

Closed ddobie closed 2 months ago

ddobie commented 7 months ago

During a large pipeline run (https://dev.pipeline.vast-survey.org/piperuns/70) I managed to crash vast-data at the association upload step (https://github.com/askap-vast/vast-pipeline/blob/159816940e7afe745bbd4d39e9c808ca8e8f727e/vast_pipeline/pipeline/loading.py#L223)

2024-02-16 16:44:52,155 finalise DEBUG related_df memory after upload: 17.64984MB
2024-02-16 16:44:52,156 finalise DEBUG Current memory usage: 15.524GB
2024-02-16 16:45:06,674 finalise DEBUG sources_df memory after srcs_df merge: 11053.352438MB
2024-02-16 16:45:06,674 finalise DEBUG Current memory usage: 21.095GB
2024-02-16 16:45:06,674 loading INFO Upload associations...
2024-02-16 16:45:09,239 loading DEBUG associations_df memory usage: 11053.352438MB
2024-02-16 16:45:09,240 loading DEBUG Current memory usage: 21.099GB
2024-02-16 16:45:09,240 model_generator DEBUG Building 20815521 association generators

Grafana memory usage for the run is here - things go pear shaped almost immediately.

ddobie commented 7 months ago

Possible solutions here?

https://medium.com/@rui.jorge.rei/today-i-learned-django-memory-leak-and-the-sql-query-cache-1c152f62f64