search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Pipeline test test_data_flow (and more) uses at least 3GB of memory #249

Closed whohensee closed 1 month ago

whohensee commented 2 months ago

On my terrible laptop which only has a small amount of memory available before culling the tests, we noticed that this test (at least) balloons up to 3+GB before being culled by the out-of-memory killer.

This was brought up to at least be aware of due to the architecture of perlmutter having 64 or 128 cores and 512GB ram, therefore if the memory usage goes much higher than this we may be limited on the number of instances we can run at once.

tests that I have had it fail on (among others): pipeline/test_pipeline.py::test_data_flow improc/test_alignment.py::test_warp_decam

guynir42 commented 2 months ago

Let's go into the test_data_flow and either track the memory use manually or using psutil to get the memory usage.

guynir42 commented 1 month ago

I think we understand this well enough to close this issue.