memory leak when loading images and requesting spectral profiles repeatedly

[veggiesaurus]

Not sure if this is possibly related to https://github.com/CARTAvis/carta/issues/83.

I wrote a code snippet to load the same file 500 times, and make a region each time. If you run the snippet with a spectral profiler widget open, the backend leaks memory.

The code snippet:

const N = 500;
// Adjust delay so that full profile is calculated each iteration
const delayTime = 2500;

for (let i = 0; i < N; i++) {
    const frame = await app.openFile("casa", "supermosaic.10", "")
    const region = frame.regionSet.addEllipticalRegion({x: 2260, y: 805}, 700, 700);
    await app.delay(20);
    frame.regionSet.selectRegion(region);
    await app.delay(delayTime);
}

I tried the same image with CASA, FITS and HDF5, and got the following results (not exactly apples-to-apples, because I stopped each after a few hundred iterations).

CASA:

FITS:

HDF5:

[jolopezl]

I am using macOS 11, and it looks like I can't reproduce the memory behavior.

CASA: FITS: HDF5:

@veggiesaurus, do you remember on which platform you observed this leak? I tested this with the current status of the backend.

If the leak is gone, that's great. But, in the meantime, I found another problem. For the case of the FITS file, after ~100 seconds of running, the loop of the program ends with:

libc++abi: Pure virtual function called!
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

While, after much fewer iterations with the HDF5 file, it crashes with:

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Instead of posting too much information about this case here, I would open another issue if you or someone else can confirm this also happens. The steps to reproduce this are the same.

[veggiesaurus]

Please open an issue. I'll check on my MacBook on Monday

[jolopezl]

Despite the unexpected crash, for the purpose of this issue, I managed to get a longer loop for the FITS file (starting carta_backend without using Clion):

[veggiesaurus]

@jolopezl just confirming that the leak is still present in dev. Are you sure you're generating spetral profiles? Does your frontend layout include a spectral profiler widget set to the active region?

This is with a CASA file:

and this is the same file, but without the spectral profiler widget:

[jolopezl]

@jolopezl just confirming that the leak is still present in dev. Are you sure you're generating spectral profiles? Does your frontend layout include a spectral profiler widget set to the active region?

Thanks for pointing out the missing widget. I verified that this is actually happening on several platforms.

[jolopezl]

No evident leaks are pointing to problems with carta-backend at the moment, as far as I can tell. I ran Valgrind with memcheck and massif tools to check for potential issues. I ran the JS code snippet 500 times, 750 times, and 1000 times using an Ubuntu 20.04 docker container. These tools didn't show any evident problems. The program ended all times successfully. I ran the same loop with address sanitization enabled 100 and 250 times, and no problems triggered. In both cases, psrecord shows increasing memory usage. We could attribute these problems to casacore, but that’s hard to tell without addressing all issues we have spotted in casacore (remember that we suppress problems from casacore). That’s a major effort.