Segmentation fault with large image segmentation

[jonhsussman]

Hello,

I am using Mesmer to segment large images from CODEX multiplexed imaging. This has worked well for all of the data, but since beginning to generate images of larger size, we are running into a segmentation fault error on images with sizes greater than about 33k X 33k, which occurs after a couple of hours of running. It is hard to difficult to pinpoint the exact size of image that begins to cause this error, but it works for images of about 31k X 31k just with a very long runtime (~8 hours). We are running app.predict(image, image_mpp=0.5). Considering the possibility that this could be an issue of RAM, the issue still persists after doubling the allotted RAM to 900 GB.

First, this is the output log for a successful run:

Then, this is the output for run which causes the error:

Has anyone else encountered this error on larger images?

[ngreenwald]

Hey John, For an image that large, I would expect you would have a much easier time processing it in smaller crops of say 10k x 10k or 20k x 20k pixels. You wouldn't have very many cells at the boundary to worry about overlaps, so you would likely get a performance speedup without sacrificing much, if any, accuracy. Maybe @msschwartz21 has some ideas about the error you're seeing so that we can continue to expand the image size supported with the model.