runPipeline --datahub not working anymore

[andreped]

Describe the bug After a complete reset of FAST on my macbook, where I deleted the FAST directory and reinstalled pyFAST, I observed that running a simple pipeline in the datahub through the runPipeline CLI no longer worked. From the error log, it seems like FAST cannot find the FPL, and after checking the expected location, I was surprised to find that it was not there.

To Reproduce

runPipeline --datahub nuclei-segmentation-model --verbose --file /path/to/some/image.svs

System:

• OS and version: [e.g. Windows 10/Ubuntu 18.04] macOS 12 Monterey
• GPU and OpenCL vendor (NVIDIA, AMD, Intel) Intel HD Graphics 530
• pyFAST version: 4.7.1

Error logs:

INFO [0x107317600] Processing token --opencl-platform
INFO [0x107317600] Processing token --file
INFO [0x107317600] Processing token /Users/andreped/Downloads/CMU-1.svs
INFO [0x107317600] CommandLineParser: Assigning variable file with value /Users/andreped/Downloads/CMU-1.svs
INFO [0x107317600] Loaded configuration file: /usr/local/lib/python3.8/site-packages/fast/bin/fast_configuration.txt
INFO [0x107317600] Test data path: /Users/andreped/FAST/data/
INFO [0x107317600] Kernel source path: /usr/local/lib/python3.8/site-packages/fast/bin//..//kernels/
INFO [0x107317600] Kernel binary path: /Users/andreped/FAST//kernel_binaries/
INFO [0x107317600] Documentation path: /usr/local/lib/python3.8/site-packages/fast/bin//..//doc/
INFO [0x107317600] Pipeline path: /Users/andreped/FAST//pipelines/
INFO [0x107317600] Qt plugins path: /usr/local/lib/python3.8/site-packages/fast/bin//..//plugins/
INFO [0x107317600] Library path: /usr/local/lib/python3.8/site-packages/fast/bin//..//lib/
WARNING [0x107317600] Test data was NOT found at /Users/andreped/FAST/data/. Download the test data by running ./downloadTestData or fast.downloadTestDataIfNotExists() in Python.
INFO [0x107317600] Creating new QApp
INFO [0x107317600] Creating new GL context for computation thread
Retrieving 'Nuclei segmentation model' model from FAST Data Hub
License: CC BY 4.0
Downloading [============================================================================] 100% | 27MB / 27MB | ETA 0.0 mins
Complete.                                                                                                                   
ERROR [0x107317600] Terminated with unhandled exception: Unable to open file /Users/andreped/FAST/data//../datahub/nuclei-segmentation-model/pipeline.fpl
libc++abi: terminate_handler unexpectedly returned

---

EDIT: I also managed to reproduce the issue in Google CoLab (running Linux). Perhaps the FPL-file is just not downloaded anymore?

[smistad]

Try nuclei-segmentation instead of nuclei-segmentation-model: runPipeline --datahub nuclei-segmentation --verbose --file /path/to/some/image.svs

I realize we need to produce an error when trying to run data hub items which are not pipelines..

[andreped]

Aaah, thats true. I thought for a second that the pipeline name was linked to the ID as seen on the DataHub.

Would be helpful to know which FPL ID one should set in runPipeline from the DataHub model list.

[andreped]

After updating the terminal command, it resulted in a new issue.

TL;DR:

ERROR [0x102c80600] FAST exception caught in Qt event handler Could not open the file /Users/andreped/FAST/data//WSI/A05.svs

Full log below:

INFO [0x102c80600] Processing token --file
INFO [0x102c80600] Processing token /Users/andreped/Downloads/CMU-1.svs
INFO [0x102c80600] CommandLineParser: Assigning variable file with value /Users/andreped/Downloads/CMU-1.svs
INFO [0x102c80600] Loaded configuration file: /usr/local/lib/python3.8/site-packages/fast/bin/fast_configuration.txt
INFO [0x102c80600] Test data path: /Users/andreped/FAST/data/
INFO [0x102c80600] Kernel source path: /usr/local/lib/python3.8/site-packages/fast/bin//..//kernels/
INFO [0x102c80600] Kernel binary path: /Users/andreped/FAST//kernel_binaries/
INFO [0x102c80600] Documentation path: /usr/local/lib/python3.8/site-packages/fast/bin//..//doc/
INFO [0x102c80600] Pipeline path: /Users/andreped/FAST//pipelines/
INFO [0x102c80600] Qt plugins path: /usr/local/lib/python3.8/site-packages/fast/bin//..//plugins/
INFO [0x102c80600] Library path: /usr/local/lib/python3.8/site-packages/fast/bin//..//lib/
WARNING [0x102c80600] Test data was NOT found at /Users/andreped/FAST/data/. Download the test data by running ./downloadTestData or fast.downloadTestDataIfNotExists() in Python.
INFO [0x102c80600] Creating new QApp
INFO [0x102c80600] Creating new GL context for computation thread
Retrieving 'Nuclei segmentation' pipeline from FAST Data Hub
Retrieving 'Nuclei segmentation model' model from FAST Data Hub
License: CC BY 4.0
Downloading [============================================================================] 100% | 27MB / 27MB | ETA 0.0 mins
Complete.                                                                                                                   
WARNING [0x102c80600] Test data was NOT found at /Users/andreped/FAST/data/. Download the test data by running ./downloadTestData or fast.downloadTestDataIfNotExists() in Python.
WARNING [0x102c80600] Test data was NOT found at /Users/andreped/FAST/data/. Download the test data by running ./downloadTestData or fast.downloadTestDataIfNotExists() in Python.
WARNING [0x102c80600] Test data was NOT found at /Users/andreped/FAST/data/. Download the test data by running ./downloadTestData or fast.downloadTestDataIfNotExists() in Python.
INFO [0x102c80600] QApp already exists..
INFO [0x102c80600] Found 1 OpenCL platforms.
INFO [0x102c80600] 1 platforms selected for inspection.
INFO [0x102c80600] Platform 0: Apple
INFO [0x102c80600] This platform has 3 available devices in total
INFO [0x102c80600] Looking for GPU devices only.
INFO [0x102c80600] 2 selected.
INFO [0x102c80600] Inspecting device 0 with the name Intel(R) HD Graphics 530
INFO [0x102c80600] Device has OpenGL interop capability
INFO [0x102c80600] The device was accepted.
INFO [0x102c80600] Inspecting device 1 with the name AMD Radeon Pro 460 Compute Engine
INFO [0x102c80600] Device has NOT OpenGL interop capability
INFO [0x102c80600] The device was NOT accepted.
INFO [0x102c80600] A device-platform mismatch was detected.
INFO [0x102c80600] Intel(R) HD Graphics 530 has 24 compute units
INFO [0x102c80600] The device Intel(R) HD Graphics 530 got a score of 24
INFO [0x102c80600] The platform Apple was selected as the best platform.
INFO [0x102c80600] A total of 1 devices were selected for the context from this platform:
INFO [0x102c80600] The best device was: Intel(R) HD Graphics 530
INFO [0x102c80600] The following device was selected as main device: Intel(R) HD Graphics 530
INFO [0x102c80600] Writing directly to 3D textures/images is supported on main device
INFO [0x102c80600] Set attribute filename to /Users/andreped/FAST/data//WSI/A05.svs for object importer
INFO [0x102c80600] Added process object WholeSlideImageImporter with id importer
INFO [0x102c80600] Added process object TissueSegmentation with id tissueSeg
INFO [0x102c80600] Connected process object importer to tissueSeg
INFO [0x102c80600] Set attribute patch-size to 256 256 for object patch
INFO [0x102c80600] Set attribute patch-magnification to 20 for object patch
INFO [0x102c80600] Set attribute patch-overlap to 0.1 for object patch
INFO [0x102c80600] Added process object PatchGenerator with id patch
INFO [0x102c80600] Connected process object importer to patch
INFO [0x102c80600] Connected process object tissueSeg to patch
INFO [0x102c80600] Loading inference engines in folder /usr/local/lib/python3.8/site-packages/fast/bin//..//lib/
INFO [0x102c80600] Loading inference engine ONNXRuntime from shared library /usr/local/lib/python3.8/site-packages/fast/bin//..//lib//libInferenceEngineONNXRuntime.dylib
INFO [0x102c80600] Loading inference engine OpenVINO from shared library /usr/local/lib/python3.8/site-packages/fast/bin//..//lib//libInferenceEngineOpenVINO.dylib
INFO [0x102c80600] Loading inference engine TensorFlow from shared library /usr/local/lib/python3.8/site-packages/fast/bin//..//lib//libInferenceEngineTensorFlow.dylib
INFO [0x102c80600] Inference engine TensorFlow selected
INFO [0x102c80600] Set attribute scale-factor to 0.003921568627451 for object network
INFO [0x102c80600] Set attribute model to "/Users/andreped/FAST/data//../datahub/nuclei-segmentation//../nuclei-segmentation-model/high_res_nuclei_unet.onnx" for object network
INFO [0x102c80600] Model format ONNX was not supported by engine TensorFlow, auto selecting engine which supports this format...
INFO [0x102c80600] Selected OpenVINO as the best engine for format ONNX
INFO [0x102c80600] OpenVINO found device: CPU
INFO [0x102c80600] OpenVINO loaded model
INFO [0x102c80600] Found input node input_1:0 with shape [?,256,256,3] and index 0
INFO [0x102c80600] Guessed image ordering to be channel last as shape was -1 256 256 3 
INFO [0x102c80600] Found output node conv2d_26/truediv:0 with shape [?,256,256,2]
INFO [0x102c80600] OpenVINO successfully compiled model
INFO [0x102c80600] OpenVINO is running network on CPU
INFO [0x102c80600] Added process object SegmentationNetwork with id network
INFO [0x102c80600] Connected process object patch to network
INFO [0x102c80600] Added process object PatchStitcher with id stitcher
INFO [0x102c80600] Connected process object network to stitcher
INFO [0x102c80600] Added process object ImagePyramidRenderer with id imgRenderer
INFO [0x102c80600] Connected process object importer to renderer imgRenderer
INFO [0x102c80600] Added renderer ImagePyramidRenderer with id imgRenderer
INFO [0x102c80600] Set attribute border-opacity to 0.5 for object segRenderer
INFO [0x102c80600] Added process object SegmentationRenderer with id segRenderer
INFO [0x102c80600] Connected process object stitcher to renderer segRenderer
INFO [0x102c80600] Added renderer SegmentationRenderer with id segRenderer
INFO [0x102c80600] Added view with id view
INFO [0x102c80600] Added renderer imgRenderer to the view
INFO [0x102c80600] Added renderer segRenderer to the view
INFO [0x102c80600] Set attribute 2Dmode to true for object view
INFO [0x102c80600] Set attribute background-color to white for object view
Pipeline execution started.
INFO [0x102c80600] QApp already exists..
INFO [0x102c80600] Setting up pipeline..
INFO [0x102c80600] Finished setting up pipeline.
qt.qpa.window: <QNSWindow: 0x7fa6e8f67630; contentView=<QNSView: 0x7fa6e8f67370; QCocoaWindow(0x6000000a3c80, window=QWidgetWindow(0x6000010a26a0, name="QGLWidgetClassWindow"))>> has active key-value observers (KVO)! These will stop working now that the window is recreated, and will result in exceptions when the observers are removed. Break in QCocoaWindow::recreateWindowIfNeeded to debug.
INFO [0x102c80600] EXECUTING WholeSlideImageImporter because PO is modified.
ERROR [0x102c80600] FAST exception caught in Qt event handler Could not open the file /Users/andreped/FAST/data//WSI/A05.svs
INFO [0x102c80600] Resizing window to 512 512
INFO [0x102c80600] EXECUTING ImagePyramidRenderer because PO is modified.
INFO [0x102c80600] EXECUTING TissueSegmentation because PO is modified.

I checked the FAST/data/ directory which was empty, explaining this issue. However, I do not get why the test data is required at all to run this pipeline. Any ideas?

[smistad]

The pipeline defaults to the wsi in the test set. Maybe --file should be --filename? I don't remember. Should add some error messages for that as well..

[andreped]

Oh, found the issue. It should be --filename and not --file. My bad!

Here I guess I assume that in the FPL file of the nuclei segmentation model there exists a @@filename@@. This is not so obvious.

And then there is a bug in the H2G-Net README (see here), because we here write --file. Unless in that pipeline there is a @@file@@ variable instead?

If thats the case, it would be great if all FPLs in the DataHub that used a single input all had @@filename@@ as the variable to make the runPipeline --datahub API more standardized across FPLs.

[smistad]

Yes, I agree. Should also produce some sensible error messages

[andreped]

As the initial error has been resolved, this issue can be closed, but you may want to keep it open until you have added the discussed fixes to not forget about them.

[smistad]

@andreped What do you think the parameter name be, --file or --filename? --file is shorten :)

[andreped]

What do you think the parameter name be, --file or --filename?

Hmm, I am fine with both, but --file is shorter so probably that one, @smistad? Or could it not be obvious enough for people what "file" means? I assume the one that's chosen will be used for all existing and future FPLs when using runPipeline, right?

[smistad]

Yes, the goal should be to use the same for all. At least all in pathology.

[smistad]

I have now changed to --file for the other pipelines. Closing..