AICAN-Research / FAST-Pathology

⚡ Open-source software for deep learning-based digital pathology
BSD 2-Clause "Simplified" License
122 stars 26 forks source link

fastpathology crashes upon running a segmentation model #58

Closed SoroushOskouei closed 1 year ago

SoroushOskouei commented 1 year ago

I am trying to run a segmentation model on fastpathology with the following pipeline:

PipelineName "Lung Tumor Segmentation" PipelineDescription "Segmentation of the tumors of the lung" PipelineInputData WSI "Whole-slide image" PipelineOutputData segmentation stitcher 0 Attribute classes "Background;Tumor"

### Processing chain
ProcessObject tissueSeg TissueSegmentation
Attribute threshold 85
Input 0 WSI

ProcessObject patch PatchGenerator
Attribute patch-size 224 224
Attribute patch-magnification 5
Attribute patch-overlap 0.0
Attribute mask-threshold 0.05
Input 0 WSI
Input 1 tissueSeg 0

ProcessObject network SegmentationNetwork
Attribute scale-factor 0.00392156862
#Attribute inference-engine TensorRT
Attribute inference-engine OpenVINO
Attribute model "/home/soroush47/Work/KaggleSeg/onnxModel/SegModel1.onnx"
Input 0 patch 0

ProcessObject stitcher PatchStitcher
Input 0 network 0

### Renderers
Renderer imgRenderer ImagePyramidRenderer
Input 0 WSI

Renderer segRenderer SegmentationRenderer
Attribute opacity 0.5
Attribute border-opacity 1.0
Input 0 stitcher 0

When I run the pipeline I get the following errors in the terminal and the program crashes.

parsing
E: [ncAPI] [    504790] [QThread] ncDeviceOpen:764  global mutex initialization failed
QObject::~QObject: Timers cannot be stopped from another thread
smistad commented 1 year ago

This is an openvino error. There should be a file on the machine called /tmp/mvnc.mutex, which should be deleted. https://www.intel.com/content/www/us/en/support/articles/000033390/boards-and-kits.html

andreped commented 1 year ago

Just observed the same trying to run the BC seg model FPL through pyFAST.

I assume the file was not deleted yet, @smistad, as I guess @SoroushOskouei does not have sudo on the server?

Shall I delete it and make a test?


EDIT: At least I found it where you said it would be /tmp/mvnc.mutex.

I just deleted it, and everything works fine. You can now give it a go, @SoroushOskouei.

Is there a way to handle this bug in FAST, @smistad, or do we need to wait until OpenVINO fixes this? Quite annoying for end users if this happens with FP.

Also, @mhoibo, if you observe anything similar, let us know, such that we know how we can reproduce the issue :]

smistad commented 1 year ago

I have added a fix to fast, so it should be part of next release

smistad commented 1 year ago

Fixed in latest build