Closed RAHoebe closed 3 years ago
urakubo
commented
3 years ago Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
RAHoebe
commented
3 years ago Hi,
I tried running both CPU and GPU .88 versions but they do not start.
Best Ron
Op vr 6 nov. 2020 om 07:05 schreef urakubo notifications@github.com:
Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-722870015, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCJZYD7VVJRRLUGWQ4TWALSOOG2DANCNFSM4TCIOQHA .
urakubo
commented
3 years ago Hi, Ron:
One of my collaborators has confirmed the safe launch, and I have also confirmed it (CPU version) on my laptop pc that was not used for its build. This may be a machine-dependent problem. Please try to launch uni-em using the following procedure:
launch "cmd.exe"
cd {Unzipped folder}\UNI-EM0.88_CPU\dist\main main.exe
Even if you fail in the launch, you will see error messages in the cmd window. Please tell me what it says.
Bests, Hidetoshi
2020年11月9日(月) 17:03 Jooppie notifications@github.com:
Hi,
I tried running both CPU and GPU .88 versions but they do not start.
Best Ron
Op vr 6 nov. 2020 om 07:05 schreef urakubo notifications@github.com:
Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-722870015, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALCJZYD7VVJRRLUGWQ4TWALSOOG2DANCNFSM4TCIOQHA
.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723839458, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWRMX3POQHWDXLGWZBTFEDSO6O6JANCNFSM4TCIOQHA .
RAHoebe
commented
3 years ago Hi,
Looks like known numpy bug in Windows: https://developercommunity.visualstudio.com/content/problem/1207405/fmod-after-an-update-to-windows-2004-is-causing-a.html
Best Ron
E:\Ron\Downloads\UNI-EM0.88_CPU\UNI-EM0.88_CPU\dist\main>main
On entry to DGEBAL parameter number 3 had an illegal value
On entry to DGEHRD parameter number 2 had an illegal value
On entry to DORGHR DORGQR parameter number 2 had an illegal value
On entry to DHSEQR parameter number 4 had an illegal value
Traceback (most recent call last):
File "UNI-EM\main.py", line 15, in
Op ma 9 nov. 2020 om 12:54 schreef urakubo notifications@github.com:
Hi, Ron:
One of my collaborators has confirmed the safe launch, and I have also confirmed it (CPU version) on my laptop pc that was not used for its build. This may be a machine-dependent problem. Please try to launch uni-em using the following procedure:
launch "cmd.exe"
cd {Unzipped folder}\UNI-EM0.88_CPU\dist\main main.exe
Even if you fail in the launch, you will see error messages in the cmd window. Please tell me what it says.
Bests, Hidetoshi
2020年11月9日(月) 17:03 Jooppie notifications@github.com:
Hi,
I tried running both CPU and GPU .88 versions but they do not start.
Best Ron
Op vr 6 nov. 2020 om 07:05 schreef urakubo notifications@github.com:
Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-722870015, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ALCJZYD7VVJRRLUGWQ4TWALSOOG2DANCNFSM4TCIOQHA
.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723839458, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHWRMX3POQHWDXLGWZBTFEDSO6O6JANCNFSM4TCIOQHA
.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723966867, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCJZYC7ZDHKILHOBV2PYELSO7J65ANCNFSM4TCIOQHA .
urakubo
commented
3 years ago Hi Ron,
Thank you for the bug report! Indeed, the failure of launch was reproduced under our Windows10, Ver2004, and this problem was then solved using numpy==1.19.3. Please find the updated version from the following link. Please tell me if you find any further difficulty. I will formally publish ver 0.88 after some modification in the next week.
UNI-EM0.88_2_CPU (294MB) https://bit.ly/38FuDGb
UNI-EM0.88_2_GPU (723MB) https://bit.ly/2GLd1wV
Sincerely, Hidetoshi
2020年11月10日(火) 1:06 Jooppie notifications@github.com:
Hi,
Looks like known numpy bug in Windows: https://developercommunity.visualstudio.com/content/problem/1207405/fmod-after-an-update-to-windows-2004-is-causing-a.html
Best Ron
E:\Ron\Downloads\UNI-EM0.88_CPU\UNI-EM0.88_CPU\dist\main>main On entry to DGEBAL parameter number 3 had an illegal value On entry to DGEHRD parameter number 2 had an illegal value On entry to DORGHR DORGQR parameter number 2 had an illegal value On entry to DHSEQR parameter number 4 had an illegal value Traceback (most recent call last): File "UNI-EM\main.py", line 15, in
File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\system\MainWindow.py", line 32, in File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\annotator\Annotator\Annotator.py", line 14, in File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\annotator\Annotator\AnnotatorServer.py", line 8, in File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "lib\site-packages\numpy__init.py", line 305, in File "lib\site-packages\numpy__init__.py", line 302, in _win_os_check RuntimeError: The current Numpy installation ('E:\Ron\Downloads\UNI-EM0.88_CPU\UNI-EM0.88_CPU\dist\main\numpy\ init__.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86 [26048] Failed to execute script mainOp ma 9 nov. 2020 om 12:54 schreef urakubo notifications@github.com:
Hi, Ron:
One of my collaborators has confirmed the safe launch, and I have also confirmed it (CPU version) on my laptop pc that was not used for its build. This may be a machine-dependent problem. Please try to launch uni-em using the following procedure:
launch "cmd.exe"
cd {Unzipped folder}\UNI-EM0.88_CPU\dist\main main.exe
Even if you fail in the launch, you will see error messages in the cmd window. Please tell me what it says.
Bests, Hidetoshi
2020年11月9日(月) 17:03 Jooppie notifications@github.com:
Hi,
I tried running both CPU and GPU .88 versions but they do not start.
Best Ron
Op vr 6 nov. 2020 om 07:05 schreef urakubo notifications@github.com:
Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-722870015, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ALCJZYD7VVJRRLUGWQ4TWALSOOG2DANCNFSM4TCIOQHA
.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723839458, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHWRMX3POQHWDXLGWZBTFEDSO6O6JANCNFSM4TCIOQHA
.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723966867, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCJZYC7ZDHKILHOBV2PYELSO7J65ANCNFSM4TCIOQHA .
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
RAHoebe
commented
3 years ago Hi Hidetoshi,
The CPU version works fine now. The GPU version works OK on a Quadro P5000 (16GB) but fails (probably due to lack of memory) on a GTX1080 (8GB). The tensorboard implementation in your program crashes the program every now and then. When I start tensorboard separately (in conda environment) it works OK.
Thanks,
Best, Ron
Here the start of the P5000 and GTX1080 output:
Start training. batch_size = 4 beta1 = 0.5 checkpoint = None display_freq = 200 filter_depth = 64 filter_size = 3 fliplr = True flipud = True image_height = 256 image_width = 256 input_dir = F:\train\UNI-EM0.88_2_GPU\data\tmp\2D_CNN_paired loss = softmax lr = 0.0002 max_epochs = 2000 max_steps = None mode = train n_dense_blocks = 5 n_dense_layers = 5 n_highway_units = 9 n_res_blocks = 9 network = unet output_dir = F:\train\model patch_size = 256 progress_freq = 50 save_freq = 5000 seed = 835257852 summary_freq = 100 target_dir = None trace_freq = 0 transpose = True u_depth = 8 examples count = 3 (each A and B) 2020-11-10 10:01:04.354038: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2020-11-10 10:01:05.652221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335 pciBusID: 0000:03:00.0 totalMemory: 16.00GiB freeMemory: 13.40GiB 2020-11-10 10:01:05.671745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2020-11-10 10:05:39.135884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-10 10:05:39.149933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2020-11-10 10:05:39.159410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2020-11-10 10:05:39.167046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12972 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1) parameter_count = 36886144 progress epoch 49 step 1 image/sec 15.4 remaining 8 min loss 0.3080284 recording summary ......
2020-11-10 09:28:50.009603: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2020-11-10 09:28:50.316524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.8475 pciBusID: 0000:03:00.0 totalMemory: 8.00GiB freeMemory: 7.04GiB 2020-11-10 09:28:50.317080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2020-11-10 09:28:54.806048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-10 09:28:54.806367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2020-11-10 09:28:54.809534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2020-11-10 09:28:54.811212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6791 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:03:00.0, compute capability: 6.1) parameter_count = 36886144 2020-11-10 09:29:02.088019: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2020-11-10 09:29:02.089002: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED Traceback (most recent call last): File "lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call File "lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn File "lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node unet/encoder_1/conv/Conv2D}} = Conv2DT=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "UNI-EM\segment_2D_DNN\translate.py", line 839, in
Caused by op 'unet/encoder_1/conv/Conv2D', defined at:
File "UNI-EM\segment_2D_DNN\translate.py", line 839, in
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node unet/encoder_1/conv/Conv2D (defined at UNI-EM\segment_2D_DNN\translate.py:134) = Conv2DT=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[12764] Failed to execute script translate
Op di 10 nov. 2020 om 03:24 schreef urakubo notifications@github.com:
Hi Ron,
Thank you for the bug report! Indeed, the failure of launch was reproduced under our Windows10, Ver2004, and this problem was then solved using numpy==1.19.3. Please find the updated version from the following link. Please tell me if you find any further difficulty. I will formally publish ver 0.88 after some modification in the next week.
UNI-EM0.88_2_CPU (294MB) https://bit.ly/38FuDGb
UNI-EM0.88_2_GPU (723MB) https://bit.ly/2GLd1wV
Sincerely, Hidetoshi
2020年11月10日(火) 1:06 Jooppie notifications@github.com:
Hi,
Looks like known numpy bug in Windows:
Best Ron
E:\Ron\Downloads\UNI-EM0.88_CPU\UNI-EM0.88_CPU\dist\main>main On entry to DGEBAL parameter number 3 had an illegal value On entry to DGEHRD parameter number 2 had an illegal value On entry to DORGHR DORGQR parameter number 2 had an illegal value On entry to DHSEQR parameter number 4 had an illegal value Traceback (most recent call last): File "UNI-EM\main.py", line 15, in
File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\system\MainWindow.py", line 32, in
File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\annotator\Annotator\Annotator.py", line 14, in
File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "UNI-EM\annotator\Annotator\AnnotatorServer.py", line 8, in
File " ", line 971, in _find_and_load File " ", line 955, in _find_and_load_unlocked File " ", line 665, in _load_unlocked File "c:\users\uraku\desktop\uni-em_related\uni-em-builder-master\uni-em-build\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "lib\site-packages\numpy__init.py", line 305, in
File "lib\site-packages\numpy\ init__.py", line 302, in _win_os_check RuntimeError: The current Numpy installation('E:\Ron\Downloads\UNI-EM0.88_CPU\UNI-EM0.88_CPU\dist\main\numpy\init.pyc') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86 [26048] Failed to execute script main
Op ma 9 nov. 2020 om 12:54 schreef urakubo notifications@github.com:
Hi, Ron:
One of my collaborators has confirmed the safe launch, and I have also confirmed it (CPU version) on my laptop pc that was not used for its build. This may be a machine-dependent problem. Please try to launch uni-em using the following procedure:
launch "cmd.exe"
cd {Unzipped folder}\UNI-EM0.88_CPU\dist\main main.exe
Even if you fail in the launch, you will see error messages in the cmd window. Please tell me what it says.
Bests, Hidetoshi
2020年11月9日(月) 17:03 Jooppie notifications@github.com:
Hi,
I tried running both CPU and GPU .88 versions but they do not start.
Best Ron
Op vr 6 nov. 2020 om 07:05 schreef urakubo <notifications@github.com :
Thank you for your comment! This is a serious bug, and we fixed the programs. Please use the following ones. We will publish them after some confirmation.
UNI-EM0.88_CPU (294MB) https://bit.ly/3p3YRs7
UNI-EM0.88_GPU (722MB) https://bit.ly/38hErWj
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/urakubo/UNI-EM/issues/35#issuecomment-722870015>, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ALCJZYD7VVJRRLUGWQ4TWALSOOG2DANCNFSM4TCIOQHA
.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723839458, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AHWRMX3POQHWDXLGWZBTFEDSO6O6JANCNFSM4TCIOQHA
.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-723966867, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ALCJZYC7ZDHKILHOBV2PYELSO7J65ANCNFSM4TCIOQHA
.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/urakubo/UNI-EM/issues/35#issuecomment-724407663, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALCJZYHPBMXWJFPT2CLAUTLSPCP6DANCNFSM4TCIOQHA .
urakubo
commented
3 years ago Thank you for comments. Many bugs are fixed, and some are updated in the version below,
UNI-EM0.88_4_CPU (363MB) https://bit.ly/3pPRG78
UNI-EM0.88_4_GPU (1,068MB) https://bit.ly/2Kzt6r5
urakubo
commented
3 years ago Further bug fix
UNI-EM0.88_5_CPU (363MB) https://bit.ly/3nSmTVu
UNI-EM0.88_5_GPU (1,068MB) https://bit.ly/39cwi62
train.zip
Hi,
I try to train a new network, however Images and Mask do not match (sorting error?). So it takes the wrong mask for the image. See zip file.
Best, Ron