search

BiaPyX / BiaPy

Open source Python library for building bioimage analysis pipelines
https://BiaPyX.github.io
MIT License
127 stars 28 forks source link

when I try to run the main.py error occured #19

Closed YDyd11 closed 2 years ago

YDyd11 commented 2 years ago

#####################

TRAIN THE MODEL

#####################

2022-06-29 13:59:43.604993: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2022-06-29 13:59:43.622141: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3400045000 Hz Epoch 1/360 2022-06-29 13:59:45.237194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2022-06-29 13:59:45.822237: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2022-06-29 13:59:46.034149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2022-06-29 13:59:46.034483: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at conv_ops.cc:1106 : Not found: No algorithm worked! Traceback (most recent call last): File "/data12T/ydaugust/code/EM_domain_adaptation-main/EM_Image_Segmentation-master/main.py", line 154, in trainer.train() File "/data12T/ydaugust/code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py", line 267, in train self.results = self.model.fit(self.train_generator, validation_data=self.val_generator, File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit tmp_logs = self.train_function(iterator) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 828, in call result = self._call(*args, *kwds) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call return self._stateless_fn(args, **kwds) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2942, in call return graph_function._call_flat( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call outputs = execute.execute( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[node model/conv2d/Conv2D (defined at /code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py:267) ]] [[Greater_1/_38]] (1) Not found: No algorithm worked! [[node model/conv2d/Conv2D (defined at /code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py:267) ]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_3792]

Function call stack: train_function -> train_function

2022-06-29 13:59:46.144834: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]

I don't know how to fix it. and I have followed the step that the author wrote on this link https://github.com/danifranco/EM_Image_Segmentation/blob/master/utils/env/environment.yml

danifranco commented 2 years ago

Hello,

I need a more detailed description of the steps you follow, type of problem (eg. segmentation, instance seg., clasification or detection), and the .yaml file you are using. Furthermore, which type of images are you using? dimensions? the more information you give me the easier it will be for me to help you.

Best,

Dani

danifranco commented 2 years ago

Close due to inactivity.

YDyd11 commented 2 years ago

I am sorry， I did not notice your reply. I was trying to segment, I used attention_unet_2d.yaml, the png image I was using, and the dimensions were 2d. patch size was the same as the original YAML used.

danifranco commented 2 years ago

Can you try the last version of the library please? Also, we need the full output of the workflow and not just the error. Try to run it again and send us all the output so we can help you better.

YDyd11 commented 2 years ago

sure. I have tried the lasted version,but it didn't work. all the output is as follow 2022-08-31 23:04:34.401681: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:36.499476: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-08-31 23:04:36.500241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2022-08-31 23:04:36.514812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:02:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:36.515825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:03:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:36.516808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:05:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:36.517933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: pciBusID: 0000:06:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:36.517955: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:36.519527: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2022-08-31 23:04:36.519570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2022-08-31 23:04:36.520926: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-08-31 23:04:36.521172: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-08-31 23:04:36.522749: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2022-08-31 23:04:36.523619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2022-08-31 23:04:36.526966: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2022-08-31 23:04:36.533792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3 2022-08-31 23:04:36.534314: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-08-31 23:04:36.534771: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-08-31 23:04:37.199867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:02:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:37.204060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:03:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:37.206271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:05:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:37.215504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: pciBusID: 0000:06:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:37.215556: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:37.215626: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2022-08-31 23:04:37.215663: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2022-08-31 23:04:37.215704: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-08-31 23:04:37.215740: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-08-31 23:04:37.215779: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2022-08-31 23:04:37.215819: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2022-08-31 23:04:37.215870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2022-08-31 23:04:37.225643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3 2022-08-31 23:04:37.225701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:39.135732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-08-31 23:04:39.135772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 3 2022-08-31 23:04:39.135782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N Y Y Y 2022-08-31 23:04:39.135790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: Y N Y Y 2022-08-31 23:04:39.135799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: Y Y N Y 2022-08-31 23:04:39.135807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 3: Y Y Y N 2022-08-31 23:04:39.139477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6098 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:02:00.0, compute capability: 6.1) 2022-08-31 23:04:39.141173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 6098 MB memory) -> physical GPU (device: 1, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1) 2022-08-31 23:04:39.142803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 6098 MB memory) -> physical GPU (device: 2, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:05:00.0, compute capability: 6.1) 2022-08-31 23:04:39.147240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 6097 MB memory) -> physical GPU (device: 3, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:06:00.0, compute capability: 6.1) 2022-08-31 23:04:39.148609: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-08-31 23:04:39.149729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:02:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.151640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:03:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.153505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:05:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.155338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: pciBusID: 0000:06:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.155449: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:39.155530: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2022-08-31 23:04:39.155579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2022-08-31 23:04:39.155624: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-08-31 23:04:39.155668: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-08-31 23:04:39.155713: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2022-08-31 23:04:39.155758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2022-08-31 23:04:39.155804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2022-08-31 23:04:39.164097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3 2022-08-31 23:04:39.164150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-08-31 23:04:39.164165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 3 2022-08-31 23:04:39.164176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N Y Y Y 2022-08-31 23:04:39.164186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: Y N Y Y 2022-08-31 23:04:39.164193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: Y Y N Y 2022-08-31 23:04:39.164199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 3: Y Y Y N 2022-08-31 23:04:39.167515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6098 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:02:00.0, compute capability: 6.1) 2022-08-31 23:04:39.168491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 6098 MB memory) -> physical GPU (device: 1, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1) 2022-08-31 23:04:39.169476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 6098 MB memory) -> physical GPU (device: 2, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:05:00.0, compute capability: 6.1) 2022-08-31 23:04:39.170457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 6097 MB memory) -> physical GPU (device: 3, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:06:00.0, compute capability: 6.1) 4 Physical GPUs, 4 Logical GPUs !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! test 2022-08-31 23:04:39.171386: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-08-31 23:04:39.171999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:02:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.172974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties: pciBusID: 0000:03:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.173936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 2 with properties: pciBusID: 0000:05:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.174905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 3 with properties: pciBusID: 0000:06:00.0 name: NVIDIA TITAN X (Pascal) computeCapability: 6.1 coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s 2022-08-31 23:04:39.174931: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 2022-08-31 23:04:39.174961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2022-08-31 23:04:39.174989: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10 2022-08-31 23:04:39.175016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2022-08-31 23:04:39.175043: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2022-08-31 23:04:39.175071: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2022-08-31 23:04:39.175098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2022-08-31 23:04:39.175124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2022-08-31 23:04:39.183217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1, 2, 3 2022-08-31 23:04:39.183324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-08-31 23:04:39.183350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 1 2 3 2022-08-31 23:04:39.183412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N Y Y Y 2022-08-31 23:04:39.183427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1: Y N Y Y 2022-08-31 23:04:39.183433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 2: Y Y N Y 2022-08-31 23:04:39.183439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 3: Y Y Y N 2022-08-31 23:04:39.186403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6098 MB memory) -> physical GPU (device: 0, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:02:00.0, compute capability: 6.1) 2022-08-31 23:04:39.187122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 6098 MB memory) -> physical GPU (device: 1, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1) 2022-08-31 23:04:39.187863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 6098 MB memory) -> physical GPU (device: 2, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:05:00.0, compute capability: 6.1) 2022-08-31 23:04:39.188599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 6097 MB memory) -> physical GPU (device: 3, name: NVIDIA TITAN X (Pascal), pci bus id: 0000:06:00.0, compute capability: 6.1) /data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/client/session.py:1761: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call InteractiveSession.close() to release resources held by the other session(s). warnings.warn('An interactive session is already active. This can ' Date: 2022-08-31 23:04:39 Arguments: Namespace(config='/data12T/ydaugust/code/EM_domain_adaptation-main/EM_Image_Segmentation-master/templates/attention_unet_2d.yaml', dataroot='/data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++', gpu='1', name='Lucchi2vnc', result_dir='/data12T/ydaugust/experiment/CUT/segmentation_result', run_id=1) Job: Lucchi2vnc_1 Python : 3.8.11 (default, Aug 3 2021, 15:09:35) Keras : 2.4.0 Tensorflow : 2.4.1 Configuration details: AUGMENTOR: AFFINE_MODE: constant AUG_NUM_SAMPLES: 10 AUG_SAMPLES: True BRIGHTNESS: False BRIGHTNESS_EM: False BRIGHTNESS_EM_FACTOR: (0.1, 0.3) BRIGHTNESS_EM_MODE: 3D BRIGHTNESS_FACTOR: (0.1, 0.3) BRIGHTNESS_MODE: 3D CBLUR_DOWN_RANGE: (2, 8) CBLUR_INSIDE: True CBLUR_SIZE: (0.2, 0.4) CHANNEL_SHUFFLE: False CMIX_SIZE: (0.2, 0.4) CNOISE_NB_ITERATIONS: (1, 3) CNOISE_SCALE: (0.05, 0.1) CNOISE_SIZE: (0.2, 0.4) CONTRAST: False CONTRAST_EM: False CONTRAST_EM_FACTOR: (0.1, 0.3) CONTRAST_EM_MODE: 3D CONTRAST_FACTOR: (0.1, 0.3) CONTRAST_MODE: 3D COUT_APPLY_TO_MASK: False COUT_CVAL: 0 COUT_NB_ITERATIONS: (1, 3) COUT_SIZE: (0.05, 0.3) CUTBLUR: False CUTMIX: False CUTNOISE: False CUTOUT: False DA_PROB: 0.5 DRAW_GRID: True DROPOUT: False DROP_RANGE: (0, 0.2) ELASTIC: False ENABLE: True E_ALPHA: (12, 16) E_MODE: constant E_SIGMA: 4 GAMMA_CONTRAST: False GC_GAMMA: (1.25, 1.75) GRAYSCALE: False GRIDMASK: False GRID_D_RANGE: (0.4, 1) GRID_INVERT: False GRID_RATIO: 0.6 GRID_ROTATE: 1 G_BLUR: False G_SIGMA: (1.0, 2.0) HFLIP: True MB_KERNEL: (3, 7) MEDIAN_BLUR: False MISALIGNMENT: False MISSING_PARTS: False MISSP_ITERATIONS: (10, 30) MOTB_K_RANGE: (8, 12) MOTION_BLUR: False MS_DISPLACEMENT: 16 MS_ROTATE_RATIO: 0.5 RANDOM_ROT: True RANDOM_ROT_RANGE: (-180, 180) ROT90: False SHEAR: False SHEAR_RANGE: (-20, 20) SHIFT: False SHIFT_RANGE: (0.1, 0.2) SHUFFLE_TRAIN_DATA_EACH_EPOCH: True SHUFFLE_VAL_DATA_EACH_EPOCH: False VFLIP: True ZFLIP: False ZOOM: False ZOOM_RANGE: (0.8, 1.2) DATA: CHANNELS: B CHANNEL_WEIGHTS: (1, 0.2) CHECK_GENERATORS: False CHECK_MW: True CONTOUR_MODE: thick EXTRACT_RANDOM_PATCH: False MW_OPTIMIZE_THS: False MW_TH1: 0.2 MW_TH2: 0.1 MW_TH3: 0.3 MW_TH4: 1.2 MW_TH5: 1.5 PATCH_SIZE: (256, 256, 1) PROBABILITY_MAP: False REFLECT_TO_COMPLETE_SHAPE: False REMOVE_BEFORE_MW: True REMOVE_SMALL_OBJ: 30 ROOT_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++ TEST: BINARY_MASKS: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In/../bin_mask INSTANCE_CHANNELS_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In_B_thick INSTANCE_CHANNELS_MASK_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out_B_thick IN_MEMORY: True LOAD_GT: True MASK_PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out MEDIAN_PADDING: False OVERLAP: (0, 0) PADDING: (0, 0) PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In RESOLUTION: (-1,) USE_VAL_AS_TEST: False TRAIN: CHECK_CROP: False INSTANCE_CHANNELS_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/lucchi(vnc_style)_B_thick INSTANCE_CHANNELS_MASK_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Train_Out_B_thick IN_MEMORY: True MASK_PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Train_Out OVERLAP: (0, 0) PADDING: (0, 0) PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/lucchi(vnc_style) REPLICATE: 0 RESOLUTION: (-1,) VAL: BINARY_MASKS: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In/../bin_mask CROSS_VAL: False CROSS_VAL_FOLD: 1 CROSS_VAL_NFOLD: 5 FROM_TRAIN: True INSTANCE_CHANNELS_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In_B_thick INSTANCE_CHANNELS_MASK_DIR: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out_B_thick IN_MEMORY: True MASK_PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out MEDIAN_PADDING: False OVERLAP: (0, 0) PADDING: (0, 0) PATH: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In RANDOM: True RESOLUTION: (-1,) SPLIT_TRAIN: 0.1 W_BACKGROUND: 0.06 W_FOREGROUND: 0.94 LOSS: TYPE: CE MODEL: ACTIVATION: elu ARCHITECTURE: attention_unet BATCH_NORMALIZATION: False DEPTH: 3 DROPOUT_VALUES: [0.1, 0.1, 0.2, 0.2, 0.3] FEATURE_MAPS: [16, 32, 64, 128, 256] KERNEL_INIT: he_normal LOAD_CHECKPOINT: False N_CLASSES: 1 OUT_CHANNELS: 1 SPATIAL_DROPOUT: False Z_DOWN: 1 PATHS: CHARTS: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/charts CHECKPOINT: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/h5_files CHECKPOINT_FILE: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/h5_files/model_weights_Lucchi2vnc_1.h5 CROP_CHECKS: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/check_crop DA_SAMPLES: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/aug GEN_CHECKS: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/gen_check GEN_MASK_CHECKS: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/gen_mask_check LOSS_WEIGHTS: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/loss_weights MAP_CODE_DIR: MAP_H5_DIR: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/mAP_h5_files PROB_MAP_DIR: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/prob_map PROB_MAP_FILENAME: prob_map.npy RESULT_DIR: DET_LOCAL_MAX_COORDS_CHECK: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/per_image_local_max_check FULL_IMAGE: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/full_image FULL_POST_PROCESSING: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/full_post_processing PATH: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1 PER_IMAGE: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/per_image PER_IMAGE_INSTANCES: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/per_image_instances PER_IMAGE_INST_VORONOI: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/per_image_instances_voronoi PER_IMAGE_POST_PROCESSING: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/per_image_post_processing TEST_FULL_GT_H5: /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out/h5 TEST_INSTANCE_CHANNELS_CHECK: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/test_instance_channels TRAIN_INSTANCE_CHANNELS_CHECK: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/train_instance_channels VAL_INSTANCE_CHANNELS_CHECK: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/val_instance_channels WATERSHED_DIR: /data12T/ydaugust/experiment/CUT/segmentation_result/Lucchi2vnc/results/Lucchi2vnc_1/watershed PROBLEM: NDIM: 2D TYPE: SEMANTIC_SEG SYSTEM: NUM_CPUS: 1 NUM_GPUS: 1 SEED: 0 TEST: APPLY_MASK: False AUGMENTATION: True DET_LOCAL_MAX_COORDS: False DET_MIN_TH_TO_BE_PEAK: 0.2 DET_TOLERANCE: 10 DET_VOXEL_SIZE: (1, 1, 1) ENABLE: True EVALUATE: True MAP: False MATCHING_SEGCOMPARE: False MATCHING_STATS: False MATCHING_STATS_THS: [0.3, 0.5, 0.75] POST_PROCESSING: BLENDING: False YZ_FILTERING: False YZ_FILTERING_SIZE: 5 Z_FILTERING: False Z_FILTERING_SIZE: 5 STATS: FULL_IMG: True MERGE_PATCHES: True PER_PATCH: True VERBOSE: True VORONOI_ON_MASK: False TRAIN: BATCH_SIZE: 6 CHECKPOINT_MONITOR: val_loss EARLYSTOPPING_MONITOR: val_loss ENABLE: True EPOCHS: 360 LR: 0.002 LR_SCHEDULER: ENABLE: False NAME: OPTIMIZER: SGD PATIENCE: 50 Python process limited to 1 thread ###################

SANITY CHECKS

###################

Checking wheter the images in /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Train_Out are binary . . . Checking wheter the images in /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out are binary . . . #################

LOAD DATA

#################

LOAD

0) Loading train images . . . Loading data from /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/lucchi(vnc_style) 100%|█████████████████████████████████████████| 165/165 [00:04<00:00, 35.03it/s] Loaded data shape is (1980, 256, 256, 3) 1) Loading train masks . . . Loading data from /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Train_Out 100%|████████████████████████████████████████| 165/165 [00:00<00:00, 210.35it/s] Loaded data shape is (1980, 256, 256, 1) Loaded train data shape is: (1782, 256, 256, 3) Loaded validation data shape is: (198, 256, 256, 3)

END LOAD

2) Loading test images . . . Loading data from /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_In 100%|████████████████████████████████████████| 165/165 [00:00<00:00, 189.34it/s] Loaded data shape is (165, 768, 1024, 1) 3) Loading test masks . . . Loading data from /data2T/ydaugust/data/EM_dataset/EM_domain_adaptation/Lucchi++/Test_Out 100%|████████████████████████████████████████| 165/165 [00:01<00:00, 156.46it/s] Loaded data shape is (165, 768, 1024, 4) ########################

PREPARE GENERATORS

########################

Initializing train data generator . . . Initializing val data generator . . . Creating generator samples . . . 0) Creating the examples of data augmentation . . . 100%|██████████████████████████████████████████| 10/10 [00:00<00:00, 203.73it/s]

END TR-SAMPLES

#################

BUILD MODEL

#################

Model: "model"

Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) [(None, None, None, 1)] 0

conv2d (Conv2D) (None, None, None, 16) 160 input_1[0][0]

activation (Activation) (None, None, None, 16) 0 conv2d[0][0]

dropout (Dropout) (None, None, None, 16) 0 activation[0][0]

conv2d_1 (Conv2D) (None, None, None, 16) 2320 dropout[0][0]

activation_1 (Activation) (None, None, None, 16) 0 conv2d_1[0][0]

max_pooling2d (MaxPooling2D) (None, None, None, 16) 0 activation_1[0][0]

conv2d_2 (Conv2D) (None, None, None, 32) 4640 max_pooling2d[0][0]

activation_2 (Activation) (None, None, None, 32) 0 conv2d_2[0][0]

dropout_1 (Dropout) (None, None, None, 32) 0 activation_2[0][0]

conv2d_3 (Conv2D) (None, None, None, 32) 9248 dropout_1[0][0]

activation_3 (Activation) (None, None, None, 32) 0 conv2d_3[0][0]

max_pooling2d_1 (MaxPooling2D) (None, None, None, 32) 0 activation_3[0][0]

conv2d_4 (Conv2D) (None, None, None, 64) 18496 max_pooling2d_1[0][0]

activation_4 (Activation) (None, None, None, 64) 0 conv2d_4[0][0]

dropout_2 (Dropout) (None, None, None, 64) 0 activation_4[0][0]

conv2d_5 (Conv2D) (None, None, None, 64) 36928 dropout_2[0][0]

activation_5 (Activation) (None, None, None, 64) 0 conv2d_5[0][0]

max_pooling2d_2 (MaxPooling2D) (None, None, None, 64) 0 activation_5[0][0]

conv2d_6 (Conv2D) (None, None, None, 128) 73856 max_pooling2d_2[0][0]

activation_6 (Activation) (None, None, None, 128) 0 conv2d_6[0][0]

dropout_3 (Dropout) (None, None, None, 128) 0 activation_6[0][0]

conv2d_7 (Conv2D) (None, None, None, 128) 147584 dropout_3[0][0]

activation_7 (Activation) (None, None, None, 128) 0 conv2d_7[0][0]

max_pooling2d_3 (MaxPooling2D) (None, None, None, 128) 0 activation_7[0][0]

conv2d_8 (Conv2D) (None, None, None, 256) 295168 max_pooling2d_3[0][0]

activation_8 (Activation) (None, None, None, 256) 0 conv2d_8[0][0]

dropout_4 (Dropout) (None, None, None, 256) 0 activation_8[0][0]

conv2d_9 (Conv2D) (None, None, None, 256) 590080 dropout_4[0][0]

activation_9 (Activation) (None, None, None, 256) 0 conv2d_9[0][0]

conv2d_transpose (Conv2DTranspose) (None, None, None, 128) 131200 activation_9[0][0]

conv2d_10 (Conv2D) (None, None, None, 128) 16512 activation_7[0][0]

conv2d_11 (Conv2D) (None, None, None, 128) 16512 conv2d_transpose[0][0]

add (Add) (None, None, None, 128) 0 conv2d_10[0][0]
conv2d_11[0][0]

activation_10 (Activation) (None, None, None, 128) 0 add[0][0]

conv2d_12 (Conv2D) (None, None, None, 1) 129 activation_10[0][0]

activation_11 (Activation) (None, None, None, 1) 0 conv2d_12[0][0]

multiply (Multiply) (None, None, None, 128) 0 conv2d_transpose[0][0]
activation_11[0][0]

concatenate (Concatenate) (None, None, None, 256) 0 conv2d_transpose[0][0]
multiply[0][0]

conv2d_13 (Conv2D) (None, None, None, 128) 295040 concatenate[0][0]

activation_12 (Activation) (None, None, None, 128) 0 conv2d_13[0][0]

dropout_5 (Dropout) (None, None, None, 128) 0 activation_12[0][0]

conv2d_14 (Conv2D) (None, None, None, 128) 147584 dropout_5[0][0]

activation_13 (Activation) (None, None, None, 128) 0 conv2d_14[0][0]

conv2d_transpose_1 (Conv2DTranspose) (None, None, None, 64) 32832 activation_13[0][0]

conv2d_15 (Conv2D) (None, None, None, 64) 4160 activation_5[0][0]

conv2d_16 (Conv2D) (None, None, None, 64) 4160 conv2d_transpose_1[0][0]

add_1 (Add) (None, None, None, 64) 0 conv2d_15[0][0]
conv2d_16[0][0]

activation_14 (Activation) (None, None, None, 64) 0 add_1[0][0]

conv2d_17 (Conv2D) (None, None, None, 1) 65 activation_14[0][0]

activation_15 (Activation) (None, None, None, 1) 0 conv2d_17[0][0]

multiply_1 (Multiply) (None, None, None, 64) 0 conv2d_transpose_1[0][0]
activation_15[0][0]

concatenate_1 (Concatenate) (None, None, None, 128) 0 conv2d_transpose_1[0][0]
multiply_1[0][0]

conv2d_18 (Conv2D) (None, None, None, 64) 73792 concatenate_1[0][0]

activation_16 (Activation) (None, None, None, 64) 0 conv2d_18[0][0]

dropout_6 (Dropout) (None, None, None, 64) 0 activation_16[0][0]

conv2d_19 (Conv2D) (None, None, None, 64) 36928 dropout_6[0][0]

activation_17 (Activation) (None, None, None, 64) 0 conv2d_19[0][0]

conv2d_transpose_2 (Conv2DTranspose) (None, None, None, 32) 8224 activation_17[0][0]

conv2d_20 (Conv2D) (None, None, None, 32) 1056 activation_3[0][0]

conv2d_21 (Conv2D) (None, None, None, 32) 1056 conv2d_transpose_2[0][0]

add_2 (Add) (None, None, None, 32) 0 conv2d_20[0][0]
conv2d_21[0][0]

activation_18 (Activation) (None, None, None, 32) 0 add_2[0][0]

conv2d_22 (Conv2D) (None, None, None, 1) 33 activation_18[0][0]

activation_19 (Activation) (None, None, None, 1) 0 conv2d_22[0][0]

multiply_2 (Multiply) (None, None, None, 32) 0 conv2d_transpose_2[0][0]
activation_19[0][0]

concatenate_2 (Concatenate) (None, None, None, 64) 0 conv2d_transpose_2[0][0]
multiply_2[0][0]

conv2d_23 (Conv2D) (None, None, None, 32) 18464 concatenate_2[0][0]

activation_20 (Activation) (None, None, None, 32) 0 conv2d_23[0][0]

dropout_7 (Dropout) (None, None, None, 32) 0 activation_20[0][0]

conv2d_24 (Conv2D) (None, None, None, 32) 9248 dropout_7[0][0]

activation_21 (Activation) (None, None, None, 32) 0 conv2d_24[0][0]

conv2d_transpose_3 (Conv2DTranspose) (None, None, None, 16) 2064 activation_21[0][0]

conv2d_25 (Conv2D) (None, None, None, 16) 272 activation_1[0][0]

conv2d_26 (Conv2D) (None, None, None, 16) 272 conv2d_transpose_3[0][0]

add_3 (Add) (None, None, None, 16) 0 conv2d_25[0][0]
conv2d_26[0][0]

activation_22 (Activation) (None, None, None, 16) 0 add_3[0][0]

conv2d_27 (Conv2D) (None, None, None, 1) 17 activation_22[0][0]

activation_23 (Activation) (None, None, None, 1) 0 conv2d_27[0][0]

multiply_3 (Multiply) (None, None, None, 16) 0 conv2d_transpose_3[0][0]
activation_23[0][0]

concatenate_3 (Concatenate) (None, None, None, 32) 0 conv2d_transpose_3[0][0]
multiply_3[0][0]

conv2d_28 (Conv2D) (None, None, None, 16) 4624 concatenate_3[0][0]

activation_24 (Activation) (None, None, None, 16) 0 conv2d_28[0][0]

dropout_8 (Dropout) (None, None, None, 16) 0 activation_24[0][0]

conv2d_29 (Conv2D) (None, None, None, 16) 2320 dropout_8[0][0]

activation_25 (Activation) (None, None, None, 16) 0 conv2d_29[0][0]

conv2d_30 (Conv2D) (None, None, None, 1) 17 activation_25[0][0]

Total params: 1,985,061 Trainable params: 1,985,061 Non-trainable params: 0

#####################

TRAIN THE MODEL

#####################

2022-08-31 23:04:48.207881: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2022-08-31 23:04:48.227519: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3400095000 Hz Epoch 1/360 2022-08-31 23:04:49.902698: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7 2022-08-31 23:04:50.907411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2022-08-31 23:04:51.280770: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at conv_ops.cc:1106 : Not found: No algorithm worked! Traceback (most recent call last): File "/data12T/ydaugust/code/EM_domain_adaptation-main/EM_Image_Segmentation-master/main.py", line 185, in trainer.train() File "/data12T/ydaugust/code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py", line 267, in train self.results = self.model.fit(self.train_generator, validation_data=self.val_generator, File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit tmp_logs = self.train_function(iterator) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 828, in call result = self._call(*args, *kwds) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call return self._stateless_fn(args, **kwds) File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2942, in call return graph_function._call_flat( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call outputs = execute.execute( File "/data12T/ydaugust/anaconda3/envs/EM_tools/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[node model/conv2d/Conv2D (defined at /code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py:267) ]] [[Greater_1/_38]] (1) Not found: No algorithm worked! [[node model/conv2d/Conv2D (defined at /code/EM_domain_adaptation-main/EM_Image_Segmentation-master/engine/trainer.py:267) ]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_3792]

Function call stack: train_function -> train_function

2022-08-31 23:04:51.401468: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]

Process finished with exit code 1

danifranco commented 2 years ago

The input images seem to have 3 channels while you define PATCH_SIZE=(256, 256, 1). Seem that you have modify original Lucchi++ images, as they are originally grayscale 1-channel, i.e. (768, 1024, 1).

Also I just commited a minor change to print mask shape when loading 2D the data so it is more clear.

YDyd11 commented 2 years ago

Thank you! I will try it now!

danifranco commented 2 years ago

Did you manage to run it?