Efficient intensity measurements

[dchaley]

For the ~140M pixel image we detect ~160k cells. Adding QuPath intensity measurement takes about 2 minutes to process ~1600 cells (1%). That means it would take ~3hr for all cells. This is on my M3 laptop, a decently powerful machine.

On GCP Batch, this took 12.5 hours (‼️)

Getting this data is a key part of this workflow:

From Brenna:

The intensity measurements are the main output from QuPath for downstream analysis, so unfortunately we have to run that super long step 😫

So the question is: how do we run this thing more efficiently? And how is it run today? Do users start the process then walk away overnight...? And where is this running in the first place today?

[dchaley]

Help us @bnovotny wan Kenobi, you're our only hope 🙏🏻 Please see issue description for more questions re: adding image intensity measurements. Trying to wrap my head around how long this takes.

How many pixels are in your typical image, plus, how many detected compartments? Is ~160k "detection objects" a typical, small, large amount?

[dchaley]

We ran the Kotlin profiler to see where it was spending time. As suspected it's all in intensity measurements. We were surprised that the bulk of the work is actually in decompression. The OME.TIFF is compressed, ~60MB vs the ~400MB uncompressed raw pixel values. (There are ~140M pixels)

It's not clear how much it's decompressing each time, could we speed it up by doing larger chunks, etc etc. It's most likely done this way to save on memory and process pieces of the image at once. However we seem to be paying an inordinate cost in decompressing "something" over and over and over and over again.

[dchaley]

We confirmed that it's at least issuing load requests in tiny, tiny tiles: 1 request for detected object. For example, this request is for a ~40 sq px area:

There's gotta be a way to do this better– maybe preloading the image or who knows– but if we're decompressing even chunks of the file over and over no wonder it's taking so long.

[dchaley]

This adds a new question for you Brenna: are you dealing with compressed or uncompressed OMETIFFs ?

[bnovotny]

Wow this is a lot of info!

Today we are running this either on our remote server or local laptop, and yep, the only strategy I know of is to just let it run. Definitely not unusual for exporting measurements to take multiple hours running on my laptop. It's much faster on the big server, but the files I need are often not located there.

That being said, 160K cells and 140M pixels is a pretty big image compared to what I am used to working with. Raymond is working on some larger images, but the range for most of my projects have been 2k-50k cells, 4M-30M pixels per image.

I checked with Raymond, and apparently the compression status depends on the platform. CellDive is always compressed but the other platforms vary. We could standardize someday, but we're not sure whether that would be in the direction of "everything compressed" or "nothing compressed".

I hope that's helpful, let me know if I can clarify anything more!

[dchaley]

Thanks Brenna! So this tells me: it can run "anywhere", it just needs to get the measurements generated before a human looks at it somewhere.

Good to know re: typical image size. I'll look for something more in that ballpark to test. Actually. You showed us that open-source QuPath project: what size did that have, can I just use that data for testing?

I'm still curious what's going on here with performance. I smell an optimization :suspect: 🤔

[bnovotny]

Here is the public dataset I was using; I think the images should be more reasonably sized! https://zenodo.org/records/6004986

The ones you need should be in the tiffs/ folder: *_full.tiff for the channels and *_cellmask.tiff for the masks.

I'm always in favor of optimization, but either way, thanks so much for working on this as always!!

[dchaley]

Thanks @bnovotny ! QQ: what's the difference between tiffs/X_cellmask and cpout/X_cellmask ? I assume the cpout one is like DeepCell output from CellProfiler. In that case … what's the tiffs one, that's called "initial cell mask" ?

I also assume I need to convert these into npz formats. I ... might or might not do that 😅

I do need a smaller data set, I wonder if we can use the older TIFF files we have that aren't ome-tiff … didn't work before but that was before we got anything working … mumbling to myself here.

[dchaley]

Also are you looking at the files in rna or the ones in 12plex_validation/HeLa ? (the "measurements" directories in there)

[dchaley]

Flamegraph BEFORE: for 140M px image

Nearly all time spent adding intensity measurements:

Spending 99.9% of time reading the image:

[dchaley]

Flamegraph AFTER:

We're still spending the vast majority of time reading the image … however … now it only takes ~14s …