search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Memory free tests are flaky #263

Open guynir42 opened 1 month ago

guynir42 commented 1 month ago

I often get failures in the various test_free tests that verify memory being freed. In all the cases I've seen, the memory is exactly unchanged before and after freeing. I think this may have something to do with a race condition (i.e., the garbage collector didn't have time to do its work). Maybe adding a short sleep after each statement would make it happen.

It could be something else, as garbage collection is sort of voodoo, but this is worth a try.

rknop commented 1 month ago

Agreed it's worth a try, but I fear that the voodoo factor is going to be pretty big. It probably has various algorithms and heuristics as to when it's worth really freeing the memory vs. caching it aside for future python allocations. Unless there's a way to tell the garbage collector "release all non-allocated memory back to the system" (not just "run garbage collection"), I predict that this is going to be a very frustrating problem.

guynir42 commented 2 weeks ago

I have marked all these tests as "skip" because it is slowing down the development. But we should think more on this, and maybe read more about the garbage collection process and figure out something more robust.