run for project freezes intermittently

[mezwick]

Hi

I have noticed that when I run qupath-cellpose script for project, sometimes imy batch run just gets stuck on one image and cannot proceed.

At this point, my only options is to kill qupath and start again. Not sure how to troubleshoot the source of the issue, or if others run into the same issue?

Here is my script (which uses imageops to run cellpose on hematoxylin deconv channel)...

import qupath.ext.biop.cellpose.Cellpose2D

// Specify the model name (cyto, nuc, cyto2 or a path to your custom model)
def model = "my_model.598691"
def modeldir = 'K:\\path\\to\\model\\directory'
def pathModel = [modeldir, model].join('\\')

// Get info for stain deconvolution
def imageData = getCurrentImageData()
def stains = imageData.getColorDeconvolutionStains()

def cellpose = Cellpose2D.builder(pathModel) // Can choose "None" if you want to train from scratch
                .preprocess(
                    ImageOps.Channels.deconvolve(stains),
                    ImageOps.Channels.extract(0, 1) // 0 for HTX and 1 for DAB
                )
                . cellposeChannels(2, 1) // can specify 2 channels with .cellposeChannels(2, 1), or one with .cellposeChannels(1), First channels used for cyto, second for nuclei
//                .channels("Blue")  // or use work with .cellposeChannels( channel1, channel2 ) and follow the cellpose way
//                .preprocess(ImageOps.Filters.gaussianBlur(1)) // Optional preprocessing QuPath Ops 
//                .epochs(500)             // Optional: will default to 500
//                .learningRate(0.2)       // Optional: Will default to 0.2
//                .batchSize(8)            // Optional: Will default to 8
//                .minTrainMasks(5)        // Optional: Will default to 5
//                .pixelSize(0.1725)              // Resolution for detection, defaults to full resolution
                .diameter(0.0)                   // Median object diameter. Set to 0.0 for the `bact_omni` model or for automatic computation
                .cellExpansion(1.0)            // Approximate cells based upon nucleus expansion                
                .measureShape()                // Add shape measurements
                .measureIntensity()            // Add cell measurements (in all compartments) 
//                .maskThreshold(0.1)        // Probability threshold to accept between 0 and 1
//                .classify("MyDetections")     // PathClass to give newly created objects    
                .setOverlap(60)                // Overlap between tiles (in pixels) that the QuPath Cellpose Extension will extract. Defaults to 2x the diameter or 60 px if the diameter is set to 0 
                .tileSize(2048)                // If your GPU can take it, make larger tiles to process fewer of them. Useful for Omnipose    
                .useGPU()                 // Optional: Use the GPU if configured, defaults to CPU only
//                .modelDirectory( new File("Cellpose_models")) // Optional place to store resulting model. Will default to QuPath project root, and make a 'models' folder
//                .saveBuilder("My Builder") // Optional: Will save a builder json file that can be reloaded with Cellpose2D.builder(File builderFile)
                .build() 

// Run detection for the selected objects
selectObjectsByClassification("Tissue")
def pathObjects = getSelectedObjects()

if (pathObjects.isEmpty()) {
    Dialogs.showErrorMessage("Cellpose", "Please select a parent object!")
    return
}
cellpose.detectObjects(imageData, pathObjects)

[romainGuiet]

Hi @mezwick, sorry to read that you encounter an issue with the extension.

My first thought would be to "check" that you have "Tissue" annotation(s) in all the images you try to process.

My guess is that the code :

if (pathObjects.isEmpty()) {
    Dialogs.showErrorMessage("Cellpose", "Please select a parent object!")
    return
}

might block the processing ...

Alternatively you could try to replace it by :

if ( !pathObjects.isEmpty()) {
     cellpose.detectObjects(imageData, pathObjects)
}

Best

Romain

[NicoKiaru]

@mezwick another thing to look at:

• is the script failing at the same image ?
• Is the CPU active and or GPU active ?
• You can keep the log console open and this if an error message pops up

Hard to tell without more information or ways to reproduce the issue

[mezwick]

Thanks for tips, I will keep the log open and report back if it crops up again.

Each image definitely has at least 1 'Tissue' annotation. I have no issue running the project on single images. Or restarting the script on images where it has stalled.

The only trend so far is, it seems to fail on particularly large whole slide images, which often contain multiple 'Tissue' annotations on which to cell segment.

Will report back with log info if I catch it again.

[NicoKiaru]

What is the file format you use ? If it's vsi if might be related to a known issue where a part of the image on the edge might be missing.

[mezwick]

Hi Nico. Indeed it is .vsi images! Would that make sense though when i say, the images will run, when i rerun them?

[NicoKiaru]

If that's the bug I imagine, the same image will consistently fail. Re-running won't work.

The only way you can avoid the bug, is to restrict your annotations and make sure to keep a margin with the edge of the image. What's super annoying is that it's not possible to know how large should be the margin. In many cases, you don't need a margin, But sometimes you need a thousand pixel.

Alternatively, if you perform your analysis on the highest resolution, I think no pixel is missing, so you should be safe, but it'll take a lot of time.

If you want to see the underlying problem , it is explained here:

https://forum.image.sc/t/qupath-omero-weird-pyramid-levels/65484/6

(ping @dgault and @petebankhead)

[mezwick]

Ok, so i had a script freeze on a the 2nd image of a batch run this morining. I have just restarted the run for project on the image and it is already proceeding.

The issue seems to be at the tile writing stage (i noticed only a few tiles were being written previously). In this case, if i look in cellpose-temp i see that only 5 tiles were written. The log showed that the script did not proceed to writing beyond the fifth tile

INFO: Starting script at Tue Dec 06 08:14:15 GMT 2022
INFO: Provided model K:\path\to\model\directory\my_model.598691 is a file. Assuming custom model
INFO: If tiling is necessary, 60 pixels overlap will be taken between tiles
INFO: Folder creation of \\path\to\qupath\qupath_project\cellpose-temp was interrupted. Either the folder exists or there was a problem.
INFO: Saving images for 1 tiles
INFO: Saving images for 259 tiles
INFO: Saving images for 202 tiles
INFO: Saving images for 21 tiles
INFO: Saving images for 2 tiles
INFO: Saving to \\path\to\qupath\qupath_project\cellpose-temp\Temp_38912_36000_z0_t0.tif
INFO: Saving to \\path\to\qupath\qupath_project\cellpose-temp\Temp_28468_18176_z0_t0.tif
INFO: Saving to \\path\to\qupath\qupath_project\cellpose-temp\Temp_38197_55156_z0_t0.tif
INFO: Saving to \\path\to\qupath\qupath_project\cellpose-temp\Temp_38197_53108_z0_t0.tif
INFO: Saving to \\path\to\qupath\qupath_project\cellpose-temp\Temp_13621_40820_z0_t0.tif

[NicoKiaru]

Ping @lacan .

I think to be able able to fix the issue, we need a way to reproduce the error (data, model, script).

(Maybe one last thing to test: if you have a file that consistently fails, try exporting it to OME-TIFF in QuPath and re-run the detection to see if that changes anything.)

[lacan]

The issue might just be that for some reason we are using too many threads and something is stalling in QuPath when it's trying to write too many images in parallel. I would ask if you can test the newly released v0.6.0 and get back in case it keeps hanging.

If that is tha case, I will make a test version where we limit the number of threads for saving and re-reading data.

It could eventually also be that the issue will creep up somewhere else. As after saving all the tiles, it will need to load ALL of them into memory and compute all the statistics for each cell, which might end up murdering the PC.

What are the specs of the machine you are using?

[lacan]

Maybe another comment... I would sincerely check what is the effective pixel size you need for your analysis. You are not giving it a pixel size, meaning you are sending the full size image data to Cellpose, which generally expects objects to be around 30 pixels. Fixing the pixelSize() and diameter() could already help you have 1) smaller images and b) much faster cellpose segmentation as it does not need to guesstimate the diameters of your cells... Or is that not the case with your custom model?

[lacan]

If you use the latest shapshot you have an nThreads() method in the builder. You could set it to nThreads(1) and see if the issue persists. Considering it needs to save so many tiles, maybe it is a read-write issue. qupath-extension-cellpose-0.6.2-SNAPSHOT.jar.zip

[lacan]

Closing from lack of updates nor reproducibility

[mezwick]

I think this may have been caused by my running QuPath projects on a shared drive, rather than a local disk.

That or it was solved in a cellpose update, as it no longer seems to be an issue for me (i do make sure to write cellpose temp files locally with the update in 0.8 now though)