bhralzz
commented
2 years ago any comments?
Open bhralzz opened 2 years ago
bhralzz
commented
2 years ago any comments?
muellerdo
commented
2 years ago Hey @bhralzz,
MIScnn utilizes Tensorflow for Neural Network actions as well as their multi-gpu API. Sadly I'm unfamiliar with this error, however Tensorflow has various issues on multi-gpu support with Keras models due to it is still a quite 'experimental' integration.
I would highly recommend to generate a reproducible example of this behaviour (like in a Jupyter Notebook) and share it as a Tensorflow issue on their GitHub project. BUT: Please check out their wiki beforehand on how to use the multi-gpu support correctly (missing driver, incompatible hardware etc etc).
Cheers, Dominik
Hi, Dear all contributors, I have tested multi gpu mode and came up with an error: Restarting kernel... just by switching the multi gpu flag to True its working some few batch progress the stopped suddenly, when the single gpu mode worked fine, without any error all the logs of running as below:
Thanks for your valuable hints
debugfile('/media/hpds/harda/MISE/KITS/MIScnn-master_WV_400_04_31/KITS_TEST.py', wdir='/media/hpds/harda/MISE/KITS/MIScnn-master_WV_400_04_31')
!continue 2021-08-01 06:51:45.454993: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 All samples: ['case_00001', 'case_00002', 'case_00003', 'case_00004', 'case_00005', 'case_00006', 'case_00007', 'case_00008', 'case_00009', 'case_00011', 'case_00012', 'case_00013', 'case_00014', 'case_00015', 'case_00016', 'case_00017', 'case_00018', 'case_00019', 'case_00021', 'case_00022', 'case_00023', 'case_00024', 'case_00025', 'case_00026', 'case_00027', 'case_00028', 'case_00029', 'case_00031', 'case_00032', 'case_00033', 'case_00034', 'case_00035', 'case_00036', 'case_00037', 'case_00038', 'case_00039', 'case_00041', 'case_00042', 'case_00043', 'case_00044', 'case_00045', 'case_00046', 'case_00047', 'case_00048', 'case_00049', 'case_00051', 'case_00052', 'case_00053', 'case_00054', 'case_00055', 'case_00056', 'case_00057', 'case_00058', 'case_00059', 'case_00061', 'case_00062', 'case_00063', 'case_00064', 'case_00065', 'case_00066', 'case_00067', 'case_00068', 'case_00069', 'case_00071', 'case_00072', 'case_00073', 'case_00074', 'case_00075', 'case_00076', 'case_00077', 'case_00078', 'case_00079', 'case_00081', 'case_00082', 'case_00083', 'case_00084', 'case_00085', 'case_00086', 'case_00087', 'case_00088', 'case_00089', 'case_00091', 'case_00092', 'case_00093', 'case_00094', 'case_00095', 'case_00096', 'case_00097', 'case_00098', 'case_00099', 'case_00101', 'case_00102', 'case_00103', 'case_00104', 'case_00105', 'case_00106', 'case_00107', 'case_00108', 'case_00109', 'case_00111', 'case_00112', 'case_00113', 'case_00114', 'case_00115', 'case_00116', 'case_00117', 'case_00118', 'case_00119', 'case_00121', 'case_00122', 'case_00123', 'case_00124', 'case_00125', 'case_00126', 'case_00127', 'case_00128', 'case_00129', 'case_00131', 'case_00132', 'case_00133', 'case_00134', 'case_00135', 'case_00136', 'case_00137', 'case_00138', 'case_00139', 'case_00141', 'case_00142', 'case_00143', 'case_00144', 'case_00145', 'case_00146', 'case_00147', 'case_00148', 'case_00149', 'case_00151', 'case_00152', 'case_00153', 'case_00154', 'case_00155', 'case_00156', 'case_00157', 'case_00158', 'case_00159', 'case_00161', 'case_00162', 'case_00163', 'case_00164', 'case_00165', 'case_00166', 'case_00167', 'case_00168', 'case_00169', 'case_00171', 'case_00172', 'case_00173', 'case_00174', 'case_00175', 'case_00176', 'case_00177', 'case_00178', 'case_00179', 'case_00181', 'case_00182', 'case_00183', 'case_00184', 'case_00185', 'case_00186', 'case_00187', 'case_00188', 'case_00189', 'case_00191', 'case_00192', 'case_00193', 'case_00194', 'case_00195', 'case_00196', 'case_00197', 'case_00198', 'case_00199', 'case_00201', 'case_00202', 'case_00203', 'case_00204', 'case_00205', 'case_00206', 'case_00207', 'case_00208', 'case_00209'] WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. 2021-08-01 06:51:48.991371: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-08-01 06:51:49.595050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:04:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.600184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 1 with properties: pciBusID: 0000:06:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.605247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 2 with properties: pciBusID: 0000:07:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.609882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 3 with properties: pciBusID: 0000:08:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.614662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 4 with properties: pciBusID: 0000:0c:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.619480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 5 with properties: pciBusID: 0000:0d:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.624138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 6 with properties: pciBusID: 0000:0e:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.628801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 7 with properties: pciBusID: 0000:0f:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:49.628832: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-08-01 06:51:49.632107: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-08-01 06:51:49.632158: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2021-08-01 06:51:49.633234: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2021-08-01 06:51:49.633475: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 2021-08-01 06:51:49.634412: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11 2021-08-01 06:51:49.635249: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11 2021-08-01 06:51:49.635386: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-08-01 06:51:49.708710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7 2021-08-01 06:51:49.710022: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-08-01 06:51:52.131807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:04:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.134313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 1 with properties: pciBusID: 0000:06:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.136454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 2 with properties: pciBusID: 0000:07:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.138645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 3 with properties: pciBusID: 0000:08:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.141143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 4 with properties: pciBusID: 0000:0c:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.143483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 5 with properties: pciBusID: 0000:0d:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.148341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 6 with properties: pciBusID: 0000:0e:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.155488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 7 with properties: pciBusID: 0000:0f:00.0 name: Quadro RTX 8000 computeCapability: 7.5 coreClock: 1.77GHz coreCount: 72 deviceMemorySize: 47.46GiB deviceMemoryBandwidth: 625.94GiB/s 2021-08-01 06:51:52.193012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7 2021-08-01 06:51:52.193112: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-08-01 06:51:55.478323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-08-01 06:51:55.478373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 1 2 3 4 5 6 7 2021-08-01 06:51:55.478382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N Y Y Y Y Y Y Y 2021-08-01 06:51:55.478387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 1: Y N Y Y Y Y Y Y 2021-08-01 06:51:55.478391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 2: Y Y N Y Y Y Y Y 2021-08-01 06:51:55.478396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 3: Y Y Y N Y Y Y Y 2021-08-01 06:51:55.478401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 4: Y Y Y Y N Y Y Y 2021-08-01 06:51:55.478405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 5: Y Y Y Y Y N Y Y 2021-08-01 06:51:55.478410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 6: Y Y Y Y Y Y N Y 2021-08-01 06:51:55.478414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 7: Y Y Y Y Y Y Y N 2021-08-01 06:51:55.520887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 47220 MB memory) -> physical GPU (device: 0, name: Quadro RTX 8000, pci bus id: 0000:04:00.0, compute capability: 7.5) 2021-08-01 06:51:55.524191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 47220 MB memory) -> physical GPU (device: 1, name: Quadro RTX 8000, pci bus id: 0000:06:00.0, compute capability: 7.5) 2021-08-01 06:51:55.527576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 47220 MB memory) -> physical GPU (device: 2, name: Quadro RTX 8000, pci bus id: 0000:07:00.0, compute capability: 7.5) 2021-08-01 06:51:55.530847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 47220 MB memory) -> physical GPU (device: 3, name: Quadro RTX 8000, pci bus id: 0000:08:00.0, compute capability: 7.5) 2021-08-01 06:51:55.534140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 47220 MB memory) -> physical GPU (device: 4, name: Quadro RTX 8000, pci bus id: 0000:0c:00.0, compute capability: 7.5) 2021-08-01 06:51:55.537678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 47220 MB memory) -> physical GPU (device: 5, name: Quadro RTX 8000, pci bus id: 0000:0d:00.0, compute capability: 7.5) 2021-08-01 06:51:55.540923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 47220 MB memory) -> physical GPU (device: 6, name: Quadro RTX 8000, pci bus id: 0000:0e:00.0, compute capability: 7.5) 2021-08-01 06:51:55.544237: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 47220 MB memory) -> physical GPU (device: 7, name: Quadro RTX 8000, pci bus id: 0000:0f:00.0, compute capability: 7.5) INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3', '/job:localhost/replica:0/task:0/device:GPU:4', '/job:localhost/replica:0/task:0/device:GPU:5', '/job:localhost/replica:0/task:0/device:GPU:6', '/job:localhost/replica:0/task:0/device:GPU:7') INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',). /root/anaconda3/envs/MSE1/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:374: UserWarning: The
lrargument is deprecated, uselearning_rateinstead. warnings.warn( Validation samples: ['case_00001', 'case_00002', 'case_00003', 'case_00004', 'case_00005', 'case_00006', 'case_00007', 'case_00008', 'case_00009', 'case_00011', 'case_00012', 'case_00013', 'case_00014', 'case_00015', 'case_00016', 'case_00017', 'case_00018', 'case_00019', 'case_00021', 'case_00022', 'case_00023', 'case_00024', 'case_00025', 'case_00026', 'case_00027', 'case_00028', 'case_00029', 'case_00031', 'case_00032', 'case_00033', 'case_00034', 'case_00035', 'case_00036', 'case_00037', 'case_00038', 'case_00039', 'case_00041', 'case_00042', 'case_00043', 'case_00044', 'case_00045', 'case_00046', 'case_00047', 'case_00048', 'case_00049', 'case_00051', 'case_00052', 'case_00053', 'case_00054', 'case_00055', 'case_00056', 'case_00057', 'case_00058', 'case_00059', 'case_00061', 'case_00062', 'case_00063', 'case_00064', 'case_00065', 'case_00066', 'case_00067', 'case_00068', 'case_00069', 'case_00071', 'case_00072', 'case_00073', 'case_00074', 'case_00075', 'case_00076', 'case_00077', 'case_00078', 'case_00079', 'case_00081', 'case_00082', 'case_00083', 'case_00084', 'case_00085', 'case_00086', 'case_00087', 'case_00088', 'case_00089', 'case_00091', 'case_00092', 'case_00093', 'case_00094', 'case_00095', 'case_00096', 'case_00097', 'case_00098', 'case_00099', 'case_00101', 'case_00102', 'case_00103', 'case_00104', 'case_00105', 'case_00106', 'case_00107', 'case_00108', 'case_00109', 'case_00111', 'case_00112', 'case_00113', 'case_00114', 'case_00115', 'case_00116', 'case_00117', 'case_00118', 'case_00119', 'case_00121', 'case_00122', 'case_00123', 'case_00124', 'case_00125', 'case_00126', 'case_00127', 'case_00128', 'case_00129', 'case_00131', 'case_00132', 'case_00133', 'case_00134', 'case_00135', 'case_00136', 'case_00137', 'case_00138', 'case_00139', 'case_00141', 'case_00142', 'case_00143', 'case_00144', 'case_00145', 'case_00146', 'case_00147', 'case_00148', 'case_00149', 'case_00151', 'case_00152', 'case_00153', 'case_00154', 'case_00155', 'case_00156', 'case_00157', 'case_00158', 'case_00159', 'case_00161', 'case_00162', 'case_00163', 'case_00164', 'case_00165', 'case_00166', 'case_00167', 'case_00168', 'case_00169', 'case_00171', 'case_00172', 'case_00173', 'case_00174', 'case_00175', 'case_00176', 'case_00177', 'case_00178', 'case_00179', 'case_00181', 'case_00182', 'case_00183', 'case_00184', 'case_00185', 'case_00186', 'case_00187', 'case_00188', 'case_00189', 'case_00191', 'case_00192', 'case_00193', 'case_00194', 'case_00195', 'case_00196', 'case_00197', 'case_00198', 'case_00199', 'case_00201', 'case_00202', 'case_00203', 'case_00204', 'case_00205', 'case_00206', 'case_00207', 'case_00208', 'case_00209'] 2021-08-01 07:04:00.514661: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:695] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_2" op: "FlatMapDataset" input: "TensorDataset/_1" attr { key: "Targuments" value { list { } } } attr { key: "f" value { func { name: "__inference_Dataset_flat_map_flat_map_fn_10297" } } } attr { key: "output_shapes" value { list { shape { dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } } shape { dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } } shape { dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } dim { size: -1 } } } } } attr { key: "output_types" value { list { type: DT_FLOAT type: DT_FLOAT type: DT_FLOAT } } } . Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a newtf.data.Options()object then settingoptions.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATAbefore applying the options object to the dataset viadataset.with_options(options). 2021-08-01 07:04:00.661989: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) 2021-08-01 07:04:00.683221: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199990000 Hz Epoch 1/500 INFO:tensorflow:batch_all_reduce: 82 all-reduces with algorithm = hierarchical_copy, num_packs = 1 INFO:tensorflow:batch_all_reduce: 82 all-reduces with algorithm = hierarchical_copy, num_packs = 1 2021-08-01 07:06:32.153633: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-08-01 07:06:33.447127: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:37.275470: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:38.062356: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-08-01 07:06:38.426856: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:40.052920: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:40.475483: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2021-08-01 07:06:41.434251: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:42.987919: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:44.766192: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101 2021-08-01 07:06:46.079553: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 81013/100 [..............................] - ETA: 2:16 - loss: 2.4387 - dice_soft: 0.1871 - dice_crossentropy: 2.4058
Restarting kernel...