Compute Error due to Memory Error

[jesusdpa1]

Good Afternoon,

In the last run I have try with cytokit, I started observing the following error

I try modifying the parameter target_shape: [504, 672] #1024, 1344 and changing the:

tile_height: 1007 tile_width: 1344 tile_overlap_x: 576 tile_overlap_y: 433

Without any luck

experiment_2.txt

pipeline_execution_2.txt

Thanks in advance for your help,

Kind Regards

[eric-czech]

The only thing I can say with confidence is that the issue isn't related to GPU memory exhaustion (it's RAM exhaustion). If you can figure out and share these, then maybe I can help:

1. How much memory does the machine have?
2. How much is available when you're running that script (are other processes using most of it)?
3. How much memory do you see the process use before it fails?

By some quick calculations based on your config, one single tile image should occupy ~2G of memory which is not abnormally large in my experience.

[jesusdpa1]

Hi Eric, thanks for your fast reply,

How much memory does the machine have?

it has 32Gb

How much is available when you're running that script (are other processes using most of it)?

around 28Gb

How much memory do you see the process use before it fails?

before it crashes is around 4Gb for the GPU

this is the output just when it crashes:

[eric-czech]

Hm that's odd. Are you running this with docker or did you install it directly? If Docker, are there limits on container memory usage in your configuration? If installed directly, are there any ulimits in place (i.e. does ulimit -a not say "unlimited" for max memory usage) or is this perhaps using some 32-bit python install instead of 64-bit?

I'm not sure what it could be beyond that, though it's not inconceivable that it's still related to the process using more than 28G (you're not likely to catch it trying with free). You could change the config to say that there are say 10 or maybe 5 z-planes instead of 20 and see if that works.

[jesusdpa1]

The last one got to the problem. Apparently 20 planes is a lot for a 32Gb computer.

I installed cytokit through docker container, and the output for ulimit is the following

Thank you!

[eric-czech]

Ah well that's too bad. Some final suggestions that might help then:

• Assuming your images are currently 16-bit, dropping down to 8-bit (i.e. uint8) should save a lot of memory (not half, but close to it)
• You could remove the last 4 channels (DAPI-009, Blank*3) from the config and cytokit would then just ignore the related image files (I assume they're for background/autofluorescence correction but there's nothing in this pipeline that would use them)
• Since it's only a 5x5 grid, chances are that some range of 10 z-planes likely captures the in-focus objects across most or all of the grid (given a 1.5um step size) and if that range ends up being say 5-15, there is a way to remap the z planes to that range. In other words, you can have the config say there are 10 z planes and then ignore any images for z planes 1-5 and 16-20 (i.e. only use the middle planes). Happy to provide more details if you want to go that route.

Lastly, most of our experiments are processed on a 128G box if that's a helpful reference for what would definitely circumvent memory issues on an experiment of the size you're running.

[jesusdpa1]

For the first option, should I modify the images or how can I set the read to be uint8?

The second option didn't gave a positive result, but I'll keep the changes since we are not using the last one for any analysis

For the third option, how I could set the algorithm to the planes 6-15?

Thanks for the last tip, I'll look on to this for a future acquisition

[eric-czech]

For the first option, should I modify the images or how can I set the read to be uint8?

If you're using a keyence, then you can change that for future experiments in the driver but for this data I would write a python script that loads the images in and then writes them back out as uint8.

The second option didn't gave a positive result, but I'll keep the changes since we are not using the last one for any analysis

Sounds good. Removing 4 channels shouldn't save much but worth a shot.

For the third option, how I could set the algorithm to the planes 6-15?

In the config, you can add properties like this:

environment:
   # 1-based index re-mappings can be used to ignore parts of the raw data;
   # can be used on "z", "cycle", "channel" and "region" properties
   index_symlinks:
     # This will say for example for z plane 1, to go find an image on disk with "Z006" in it
     z: {1: 6, 2: 7, 3: 8, 4: 9, 5: 10, 6: 11, 7: 12, 8: 13, 9: 14, 10: 15}
acquisition:
  num_z_planes: 10

[jesusdpa1]

Thank you Eric!

The last option work perfectly,

[eric-czech]

Awesome! I'll close this out then.