search

labsyspharm / mcmicro

Multiple-choice microscopy pipeline
https://mcmicro.org/
MIT License
104 stars 58 forks source link

Segmentation:Unmicst tensorflow error #537

Closed AhmedElshahedd closed 6 months ago

AhmedElshahedd commented 7 months ago

I have faced the following error while running the pipeline at the segmentation process:

2024-02-15 13:55:16 WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:111: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. 2024-02-15 13:55:16 Instructions for updating: 2024-02-15 13:55:16 non-resource variables are not supported in the long term 2024-02-15 13:55:16 sh: 1: nvidia-smi: not found 2024-02-15 13:55:18 2024-02-15 12:55:18.714128: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: UNKNOWN ERROR (34)

ArtemSokolov commented 7 months ago

Hi @AhmedElshahedd,

Can you please share the full content of the .command.log file in the corresponding work directory?

AhmedElshahedd commented 7 months ago

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/compat/v2_compat.py:111: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term sh: 1: nvidia-smi: not found 2024-02-19 14:42:50.709380: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: UNKNOWN ERROR (34) /app/UnMicst1-5.py:114: UserWarning: tf.layers.batch_normalization is deprecated and will be removed in a future version. Please use tf.keras.layers.BatchNormalization instead. In particular, tf.control_dependencies(tf.GraphKeys.UPDATE_OPS) should not be used (consult the tf.keras.layers.BatchNormalization documentation). bn = tf.nn.leaky_relu(tf.layers.batch_normalization(c00+shortcut, training=UNet2D.tfTraining)) /usr/local/lib/python3.8/dist-packages/keras/legacy_tf_layers/normalization.py:455: UserWarning: layer.apply is deprecated and will be removed in a future version. Please use layer.__call__ method instead. return layer.apply(inputs, training=training) /app/UnMicst1-5.py:136: UserWarning: tf.layers.batch_normalization is deprecated and will be removed in a future version. Please use tf.keras.layers.BatchNormalization instead. In particular, tf.control_dependencies(tf.GraphKeys.UPDATE_OPS) should not be used (consult the tf.keras.layers.BatchNormalization documentation). lbn = tf.nn.leaky_relu(tf.layers.batch_normalization( /app/UnMicst1-5.py:139: UserWarning: tf.layers.dropout is deprecated and will be removed in a future version. Please use tf.keras.layers.Dropout instead. return tf.layers.dropout(lbn, 0.35, training=UNet2D.tfTraining) /usr/local/lib/python3.8/dist-packages/keras/legacy_tf_layers/core.py:401: UserWarning: layer.apply is deprecated and will be removed in a future version. Please use layer.__call__ method instead. return layer.apply(inputs, training=training) /app/UnMicst1-5.py:199: UserWarning: tf.layers.batch_normalization is deprecated and will be removed in a future version. Please use tf.keras.layers.BatchNormalization instead. In particular, tf.control_dependencies(tf.GraphKeys.UPDATE_OPS) should not be used (consult the tf.keras.layers.BatchNormalization documentation). tf.layers.batch_normalization(tf.nn.conv2d(cc, luXWeights2, strides=[1, 1, 1, 1], padding='SAME'), /app/UnMicst1-5.py:220: UserWarning: tf.layers.batch_normalization is deprecated and will be removed in a future version. Please use tf.keras.layers.BatchNormalization instead. In particular, tf.control_dependencies(tf.GraphKeys.UPDATE_OPS) should not be used (consult the tf.keras.layers.BatchNormalization documentation). return tf.layers.batch_normalization( /app/UnMicst1-5.py:852: UserWarning: .//2_Probabilities_1.tif is a low contrast image skimage.io.imsave( /app/UnMicst1-5.py:856: UserWarning: .//2_Probabilities_1.tif is a low contrast image skimage.io.imsave( /app/UnMicst1-5.py:861: UserWarning: .//qc//2_Preview_1.tif is a low contrast image skimage.io.imsave(args.outputPath + '//qc//' + fileStem + 'Preview' + str(int(dapiChannel)+1) + '.tif', np.uint8(255 * PM), **save_kwargs) Using CPU loading data loading data loading data 0.34 0.25 Model restored. Using channel 1 Inference... Inference... Inference...

clarenceyapp commented 7 months ago

Hi @AhmedElshahedd, It appears that UnMICST finished. It writes Inference once for each of the 3 probability classes. The messages at the top are all warnings because UnMICST couldn't find a GPU so it resorted to using your CPU instead. Can you confirm whether any output files were saved in the 'probability maps' folder?

AhmedElshahedd commented 7 months ago

Hello,

I'm using a TMA which is de arrayed to 56 cores. However, UnMICST generated only 49 output files in the 'probability maps' folder, and it skips the rest then it starts S3segmenter which is producing a value error.

(base) student@Precision-3650:~/Downloads$ nextflow run labsyspharm/mcmicro --in ./new_test/ --params new_test/params.yml -resume N E X T F L O W ~ version 23.10.1 NOTE: Your local project version looks outdated - a different revision is available in the remote repository [0ce135cc6a] Launching https://github.com/labsyspharm/mcmicro [peaceful_baekeland] DSL2 - revision: 6f4e66f23c [master] executor > local (17) [- ] process > illumination - [- ] process > registration:ashlar - [- ] process > background:backsub - [- ] process > dearray:coreograph - [- ] process > dearray:roadie:runTask - [- ] process > segmentation:roadie:runTask - [3d/19aaf7] process > segmentation:worker (unmicst-47) [ 85%] 48 of 56, cached: 48 [2e/28a7a0] process > segmentation:s3seg (17) [ 0%] 0 of 48 [- ] process > quantification:mcquant - [- ] process > downstream:worker - [- ] process > viz:autominerva - ERROR ~ Error executing process > 'segmentation:s3seg (15)'

Caused by: Process segmentation:s3seg (15) terminated with an error exit status (1) executor > local (17) [- ] process > illumination - [- ] process > registration:ashlar - [- ] process > background:backsub - [- ] process > dearray:coreograph - [- ] process > dearray:roadie:runTask - [- ] process > segmentation:roadie:runTask - [3d/19aaf7] process > segmentation:worker (unmicst-47) [ 85%] 48 of 56, cached: 48 [50/1e7175] process > segmentation:s3seg (9) [ 2%] 1 of 46, failed: 1 [- ] process > quantification:mcquant - [- ] process > downstream:worker - [- ] process > viz:autominerva - ERROR ~ Error executing process > 'segmentation:s3seg (15)'

Caused by: Process segmentation:s3seg (15) terminated with an error exit status (1)

Command executed:

python /app/S3segmenter.py --crop dearray --maskPath mask.tif --imagePath 11.ome.tif --stackProbPath 11_Probabilities_1.tif --outputPath .

Command exit status: 1

Command output: Using first channel by default! 2024-02-23 13:43:19.764656 regionprops

Command error: Using first channel by default! /app/S3segmenter.py:18: DeprecationWarning: Please use uniform_filter from the scipy.ndimage namespace, the scipy.ndimage.filters namespace is deprecated. from scipy.ndimage.filters import uniform_filter 2024-02-23 13:43:19.764656 regionprops Traceback (most recent call last): File "/app/S3segmenter.py", line 512, in nucleiMask = S3NucleiSegmentationWatershed(nucleiPM,nucleiCrop,args.logSigma,TMAmask,args.nucleiFilter,args.nucleiRegion) File "/app/S3segmenter.py", line 245, in S3NucleiSegmentationWatershed MITh = threshold_otsu(mean_ints) File "/usr/local/lib/python3.8/site-packages/skimage/filters/thresholding.py", line 280, in threshold_otsu raise ValueError("threshold_otsu is expected to work with images " ValueError: thresholdotsu is expected to work with images having more than one color. The input image seems to have just one color 5.389121338912134. /app/S3segmenter.py:105: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:102: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return maxima.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.boolhere. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool) /app/S3segmenter.py:105: DeprecationWarning:np.boolis a deprecated alias for the builtinbool. To silence this warning, useboolby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.bool` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations return padded.astype(np.bool)

Work dir: /home/student/Downloads/work/78/6a028a94c8cd94c579866aa1ab628d

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

-- Check '.nextflow.log' file for details

ArtemSokolov commented 7 months ago

It seems there are a few things going on:

suggests that the probability maps are uniform. If you open that probability map in an image viewer (e.g., ImageJ), do you see anything that looks like cells, or is it just a single-color image? If latter, then UnMicst didn't detect any cells and you may need to tweak its parameters.

One other thing to check is to try running the pipeline with another segmenter (e.g., Mesmer or Cellpose) and see if you are running into the same issues. UnMicst is probably the most-resource intensive module in the pipeline right now, so it may be worth doing a run with something more lightweight to make sure it's not a resource allocation issue.

AhmedElshahedd commented 7 months ago

Hi @ArtemSokolov @clarenceyapp, It worked for me, thanks for your help.