mezoderma
commented
10 months ago Although with my own data (1.2 Mb vs 6 Mb test data) I got another one error:
(deepte) ilbor@ilbor-MS-7C91:~/deepTE$ python ~/soft/DeepTE/DeepTE.py -d working_dir -o output_dir -i consensi.fa.classified -sp M -m_dir ~/miniconda3/envs/deepte/models/Metazoans_model/
Using TensorFlow backend.
start time is 1705069063.0767448
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
2024-01-12 17:17:43.223107: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2024-01-12 17:17:43.246928: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3799960000 Hz
2024-01-12 17:17:43.247580: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a56f2514e0 executing computations on platform Host. Devices:
2024-01-12 17:17:43.247602: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2024-01-12 17:17:43.248847: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2024-01-12 17:17:43.312145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.312433: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a56f251330 executing computations on platform CUDA. Devices:
2024-01-12 17:17:43.312460: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA GeForce GT 710, Compute Capability 3.5
2024-01-12 17:17:43.312583: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.312806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:2b:00.0
2024-01-12 17:17:43.312956: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 17:17:43.313858: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2024-01-12 17:17:43.314896: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2024-01-12 17:17:43.315037: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2024-01-12 17:17:43.315941: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2024-01-12 17:17:43.316455: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2024-01-12 17:17:43.318378: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2024-01-12 17:17:43.318446: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.318694: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.318888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2024-01-12 17:17:43.318908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 17:17:43.319468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-12 17:17:43.319478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2024-01-12 17:17:43.319481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2024-01-12 17:17:43.319541: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.319758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.319966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1428 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GT 710, pci bus id: 0000:2b:00.0, compute capability: 3.5)
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2024-01-12 17:17:55.812661: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2024-01-12 17:17:55.901100: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2024-01-12 17:17:57.287230: W tensorflow/core/common_runtime/bfc_allocator.cc:237] Allocator (GPU_0_bfc) ran out of memory trying to allocate 299.89MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2024-01-12 17:19:03.452000: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 900B (rounded to 1024). Current allocation summary follows.
2024-01-12 17:19:03.452051: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 51, Chunks in use: 51. 12.8KiB allocated for chunks. 12.8KiB in use in bin. 264B client-requested in use in bin.
2024-01-12 17:19:03.452063: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 22, Chunks in use: 22. 12.8KiB allocated for chunks. 12.8KiB in use in bin. 10.8KiB client-requested in use in bin.
2024-01-12 17:19:03.452073: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 19, Chunks in use: 19. 22.5KiB allocated for chunks. 22.5KiB in use in bin. 21.0KiB client-requested in use in bin.
2024-01-12 17:19:03.452082: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452091: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452099: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452109: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452117: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452125: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452135: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 5, Chunks in use: 5. 880.0KiB allocated for chunks. 880.0KiB in use in bin. 878.9KiB client-requested in use in bin.
2024-01-12 17:19:03.452145: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 5, Chunks in use: 5. 1.93MiB allocated for chunks. 1.93MiB in use in bin. 1.93MiB client-requested in use in bin.
2024-01-12 17:19:03.452155: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452163: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452172: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452181: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452189: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452198: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452206: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452215: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452224: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 5, Chunks in use: 5. 1.10GiB allocated for chunks. 1.10GiB in use in bin. 1.10GiB client-requested in use in bin.
2024-01-12 17:19:03.452234: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 1, Chunks in use: 1. 301.76MiB allocated for chunks. 301.76MiB in use in bin. 224.78MiB client-requested in use in bin.
2024-01-12 17:19:03.452243: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 1.0KiB was 1.0KiB, Chunk State:
2024-01-12 17:19:03.452251: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1497890816
2024-01-12 17:19:03.452261: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40000 next 1 of size 1280
2024-01-12 17:19:03.452269: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40500 next 2 of size 256
2024-01-12 17:19:03.452276: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40600 next 3 of size 256
2024-01-12 17:19:03.452283: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40700 next 4 of size 256
2024-01-12 17:19:03.452290: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40800 next 5 of size 256
2024-01-12 17:19:03.452297: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40900 next 6 of size 256
2024-01-12 17:19:03.452305: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40a00 next 7 of size 512
2024-01-12 17:19:03.452313: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40c00 next 8 of size 256
2024-01-12 17:19:03.452320: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40d00 next 9 of size 256
2024-01-12 17:19:03.452327: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40e00 next 10 of size 1024
2024-01-12 17:19:03.452334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41200 next 11 of size 256
2024-01-12 17:19:03.452342: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41300 next 12 of size 256
2024-01-12 17:19:03.452349: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41400 next 13 of size 768
2024-01-12 17:19:03.452356: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41700 next 14 of size 256
2024-01-12 17:19:03.452363: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41800 next 15 of size 256
2024-01-12 17:19:03.452370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41900 next 16 of size 512
2024-01-12 17:19:03.452377: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41b00 next 17 of size 512
2024-01-12 17:19:03.452384: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41d00 next 18 of size 1280
2024-01-12 17:19:03.452391: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d42200 next 19 of size 180224
2024-01-12 17:19:03.452399: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d6e200 next 20 of size 405248
2024-01-12 17:19:03.452407: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1100 next 21 of size 1536
2024-01-12 17:19:03.452414: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1700 next 22 of size 256
2024-01-12 17:19:03.452421: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1800 next 23 of size 512
2024-01-12 17:19:03.452429: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1a00 next 24 of size 235699200
2024-01-12 17:19:03.452436: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99600 next 25 of size 1024
2024-01-12 17:19:03.452443: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99a00 next 26 of size 768
2024-01-12 17:19:03.452450: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99d00 next 35 of size 256
2024-01-12 17:19:03.452457: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99e00 next 36 of size 1536
2024-01-12 17:19:03.452464: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a400 next 32 of size 512
2024-01-12 17:19:03.452471: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a600 next 33 of size 235699200
2024-01-12 17:19:03.452478: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62200 next 31 of size 1024
2024-01-12 17:19:03.452485: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62600 next 27 of size 405248
2024-01-12 17:19:03.452492: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5500 next 28 of size 768
2024-01-12 17:19:03.452499: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5800 next 29 of size 256
2024-01-12 17:19:03.452506: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5900 next 34 of size 180224
2024-01-12 17:19:03.452513: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1900 next 30 of size 512
2024-01-12 17:19:03.452520: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1b00 next 37 of size 1280
2024-01-12 17:19:03.452527: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2000 next 38 of size 256
2024-01-12 17:19:03.452534: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2100 next 39 of size 256
2024-01-12 17:19:03.452541: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2200 next 40 of size 256
2024-01-12 17:19:03.452548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2300 next 41 of size 256
2024-01-12 17:19:03.452554: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2400 next 42 of size 256
2024-01-12 17:19:03.452561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2500 next 43 of size 256
2024-01-12 17:19:03.452568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2600 next 44 of size 1024
2024-01-12 17:19:03.452575: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2a00 next 45 of size 1024
2024-01-12 17:19:03.452582: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2e00 next 46 of size 512
2024-01-12 17:19:03.452589: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3000 next 47 of size 256
2024-01-12 17:19:03.452596: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3100 next 48 of size 256
2024-01-12 17:19:03.452603: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3200 next 49 of size 256
2024-01-12 17:19:03.452610: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3300 next 50 of size 256
2024-01-12 17:19:03.452617: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3400 next 51 of size 256
2024-01-12 17:19:03.452624: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3500 next 52 of size 180224
2024-01-12 17:19:03.452631: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f500 next 53 of size 1280
2024-01-12 17:19:03.452638: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01fa00 next 54 of size 235699200
2024-01-12 17:19:03.452645: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7600 next 55 of size 256
2024-01-12 17:19:03.452652: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7700 next 56 of size 256
2024-01-12 17:19:03.452658: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7800 next 57 of size 256
2024-01-12 17:19:03.452665: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7900 next 58 of size 1536
2024-01-12 17:19:03.452672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7f00 next 59 of size 235699200
2024-01-12 17:19:03.452679: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afb00 next 60 of size 256
2024-01-12 17:19:03.452686: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afc00 next 61 of size 1536
2024-01-12 17:19:03.452693: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0200 next 62 of size 1280
2024-01-12 17:19:03.452700: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0700 next 63 of size 512
2024-01-12 17:19:03.452707: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0900 next 64 of size 405248
2024-01-12 17:19:03.452714: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213800 next 65 of size 256
2024-01-12 17:19:03.452721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213900 next 66 of size 256
2024-01-12 17:19:03.452728: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213a00 next 67 of size 180224
2024-01-12 17:19:03.452735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923fa00 next 68 of size 256
2024-01-12 17:19:03.452742: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923fb00 next 69 of size 405248
2024-01-12 17:19:03.452749: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2a00 next 70 of size 256
2024-01-12 17:19:03.452756: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2b00 next 71 of size 512
2024-01-12 17:19:03.452762: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2d00 next 72 of size 512
2024-01-12 17:19:03.452768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2f00 next 73 of size 768
2024-01-12 17:19:03.452773: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3200 next 74 of size 768
2024-01-12 17:19:03.452779: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3500 next 75 of size 256
2024-01-12 17:19:03.452785: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3600 next 76 of size 256
2024-01-12 17:19:03.452791: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3700 next 77 of size 256
2024-01-12 17:19:03.452797: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3800 next 90 of size 256
2024-01-12 17:19:03.452803: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3900 next 93 of size 256
2024-01-12 17:19:03.452809: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3a00 next 84 of size 256
2024-01-12 17:19:03.452815: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3b00 next 94 of size 256
2024-01-12 17:19:03.452820: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3c00 next 95 of size 256
2024-01-12 17:19:03.452826: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3d00 next 79 of size 256
2024-01-12 17:19:03.452832: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3e00 next 81 of size 256
2024-01-12 17:19:03.452838: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3f00 next 92 of size 512
2024-01-12 17:19:03.452844: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4100 next 80 of size 256
2024-01-12 17:19:03.452849: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4200 next 85 of size 256
2024-01-12 17:19:03.452855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4300 next 86 of size 1024
2024-01-12 17:19:03.452861: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4700 next 88 of size 256
2024-01-12 17:19:03.452866: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4800 next 89 of size 256
2024-01-12 17:19:03.452872: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4900 next 87 of size 768
2024-01-12 17:19:03.452878: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4c00 next 83 of size 256
2024-01-12 17:19:03.452884: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4d00 next 91 of size 256
2024-01-12 17:19:03.452890: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4e00 next 82 of size 512
2024-01-12 17:19:03.452895: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a5000 next 78 of size 512
2024-01-12 17:19:03.452901: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a5200 next 96 of size 405248
2024-01-12 17:19:03.452907: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339308100 next 97 of size 180224
2024-01-12 17:19:03.452913: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339334100 next 98 of size 235699200
2024-01-12 17:19:03.452919: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fbd00 next 99 of size 256
2024-01-12 17:19:03.452924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fbe00 next 100 of size 768
2024-01-12 17:19:03.452930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc100 next 101 of size 1024
2024-01-12 17:19:03.452936: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc500 next 102 of size 1024
2024-01-12 17:19:03.452943: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc900 next 103 of size 1280
2024-01-12 17:19:03.452948: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fce00 next 104 of size 512
2024-01-12 17:19:03.452954: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd000 next 111 of size 256
2024-01-12 17:19:03.452960: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd100 next 112 of size 1024
2024-01-12 17:19:03.452966: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd500 next 110 of size 512
2024-01-12 17:19:03.452972: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd700 next 18446744073709551615 of size 316418304
2024-01-12 17:19:03.452979: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2024-01-12 17:19:03.452986: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 51 Chunks of size 256 totalling 12.8KiB
2024-01-12 17:19:03.452994: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 15 Chunks of size 512 totalling 7.5KiB
2024-01-12 17:19:03.453000: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 7 Chunks of size 768 totalling 5.2KiB
2024-01-12 17:19:03.453006: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 9 Chunks of size 1024 totalling 9.0KiB
2024-01-12 17:19:03.453012: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 1280 totalling 7.5KiB
2024-01-12 17:19:03.453018: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 1536 totalling 6.0KiB
2024-01-12 17:19:03.453024: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 180224 totalling 880.0KiB
2024-01-12 17:19:03.453031: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 405248 totalling 1.93MiB
2024-01-12 17:19:03.453037: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 235699200 totalling 1.10GiB
2024-01-12 17:19:03.453044: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 316418304 totalling 301.76MiB
2024-01-12 17:19:03.453050: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 1.39GiB
2024-01-12 17:19:03.453056: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 1497890816 memory_limit_: 1497890816 available bytes: 0 curr_region_allocation_bytes_: 2995781632
2024-01-12 17:19:03.453068: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 1497890816
InUse: 1497890816
MaxInUse: 1497890816
NumAllocs: 1314
MaxAllocSize: 316421120
2024-01-12 17:19:03.453081: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ***********************************************************************************************xxxxx
2024-01-12 17:19:03.453107: W tensorflow/core/framework/op_kernel.cc:1479] OP_REQUIRES failed at constant_op.cc:77 : Resource exhausted: OOM when allocating tensor of shape [225] and type float
2024-01-12 17:19:03.453134: E tensorflow/core/common_runtime/executor.cc:641] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [225] and type float
[[{{node training_1/Adam/zeros_5}}]]
Traceback (most recent call last):
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [225] and type float
[[{{node training_1/Adam/zeros_5}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ilbor/soft/DeepTE/DeepTE.py", line 274, in <module>
main()
File "/home/ilbor/soft/DeepTE/DeepTE.py", line 249, in main
pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 401, in classify_pipeline
predict_te(model_file_dic[model_name], model_name, x_classI_ipt_test_list, y_classI_ipt_test_nm_list,input_spe_type,y_classI_ipt_test_nm_list,prop_thr)
File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
model = load_model(model)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 325, in _deserialize_model
model.optimizer.set_weights(optimizer_weight_values)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 126, in set_weights
param_values = K.batch_get_value(params)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2420, in batch_get_value
return get_session().run(ops)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 206, in get_session
session.run(tf.variables_initializer(uninitialized_vars))
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [225] and type float
[[node training_1/Adam/zeros_5 (defined at /miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:702) ]]
Original stack trace for 'training_1/Adam/zeros_5':
File "/soft/DeepTE/DeepTE.py", line 274, in <module>
main()
File "/soft/DeepTE/DeepTE.py", line 249, in main
pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
File "/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 401, in classify_pipeline
predict_te(model_file_dic[model_name], model_name, x_classI_ipt_test_list, y_classI_ipt_test_nm_list,input_spe_type,y_classI_ipt_test_nm_list,prop_thr)
File "/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
model = load_model(model)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 317, in _deserialize_model
model._make_train_function()
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/training.py", line 509, in _make_train_function
loss=self.total_loss)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 487, in get_updates
ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 487, in <listcomp>
ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 702, in zeros
v = tf.zeros(shape=shape, dtype=tf_dtype, name=name)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1871, in zeros
output = _constant_if_small(zero, shape, dtype, name)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1829, in _constant_if_small
return constant(value, shape=shape, dtype=dtype, name=name)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 246, in constant
allow_broadcast=True)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 290, in _constant_impl
name=name).outputs[0]
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()
Dear Song,
Thank you for the DeepTE. I have the following error with test dataset. The CUDA driver is installed (below). Does it mean that I need more GPU?
Thank you in advance, Ilya