Closed kushalkolar closed 4 years ago
Just to add onto this, there's a bug with tensorflow v1 w/regards to cuDNN and newer RTX cards. The following exception is raised when training or predicting:
2020-08-18 21:49:40.924336: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-08-18 21:49:40.949069: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
0%| | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/share/python-virtual-environments/nuset/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node model_U-Net/conv1-1/Conv2D}}]]
[[ToFloat/_105]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node model_U-Net/conv1-1/Conv2D}}]]
0 successful operations.
0 derived errors ignored.
This seems to be a well known issue that can be fixed by just setting an environment variable: export TF_FORCE_GPU_ALLOW_GROWTH=true
https://github.com/tensorflow/tensorflow/issues/24496
Hi,
This is a cool package! I had trouble getting it installed so I changed some things to make it easier for others.
I created a proper
setup.py
so that it can be easily installed viapip
. For now it can be installed usingpip install -e .
, but you'll need to modify some things to get it working from any arbitrary dir if you want to have it on pypi.About the installation:
scikit-image>0.15
do not have amin_size
kwarg forskimage.morphology.remove_small_holes()
which you use inutils.normalized.clean_image()
. So thesetup.py
is set to work with0.15.x
.__init__.py
tomodel_layers
andutils
so that they're proper sub-packages that can be identified throughsetuptools.find_packages()
MANIFEST.in
so that network model files etc. are included.scikit-image 0.15 has the
min_size
kwarg: https://scikit-image.org/docs/0.15.x/api/skimage.morphology.html#skimage.morphology.remove_small_holesit was removed in v0.16: https://scikit-image.org/docs/0.16.x/api/skimage.morphology.html?highlight=morphology%20remove_small_holes#remove-small-holes
Bug fixes:
usingCL
attribute toNuSet
class inNuSeT.py
. Set it toFalse
.0
intrain_gui.py
(line 189-196), otherwise line 272 will raise an exception. Not sure if this is the right way to fix this.