search

shreyashampali / HOnnotate

CVPR2020. HOnnotate: A method for 3D Annotation of Hand and Object Poses
https://www.tugraz.at/index.php?id=40231
GNU General Public License v3.0
166 stars 15 forks source link

Please tell me your GPU name. #34

Open MY-CODE-1981 opened 7 months ago

MY-CODE-1981 commented 7 months ago

Thank you for your very helpful package on github. I am having trouble with an error at the object pose initialization. I am having problems with the following commands $ python objectTrackingSingleFrame.py --seq 'test'

Let me share a little briefly what I have been able to ascertain. Below is my execution environment. Basically, the execution environment is made of dockers. The docker recognizes the nvidia driver, and libraries around cuda are also running.

ubuntu16.04 cuda9.0 cudnn7.6.5 python3.5 tensorflow1.12 tensorflow-gpu1.12

sample image is below 00001_image

On segmentation, I could run below command and get result like this. $ python inference_seg.py --seq 'test' 00001_segmentation

And then, on hand 2d kyepoints, I could run below command and get result like this. python inference_hand.py --seq 'test' 00001_handpose

And next, on object pose initialization, if I run this command, $ python objectTrackingSingleFrame.py --seq 'test' An error happens like this, "Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB."

What GPU did you use with 29 GB of memory?

MY-CODE-1981 commented 7 months ago

To be sure, I show the full text of the error.

(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test' /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3 from IPython.utils.signatures import signature /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)

Additionally, to use tf.print in python 2.7, users must make sure to import the following:

from __future__ import print_function

/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead want_out = 'out' in inspect.getargspec(func).args /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) /home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use argmin instead 2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335 pciBusID: 0000:03:00.0 totalMemory: 15.90GiB freeMemory: 15.78GiB 2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725 pciBusID: 0000:02:00.0 totalMemory: 23.70GiB freeMemory: 22.76GiB 2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N 2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N 2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1) 2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6) Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). maskPC for Image test/0/00002 is 0.132587 [Loading New frame ][0/00002] 2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB. Current allocation summary follows. 2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin. 2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin. 2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin. 2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin. 2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin. 2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State: 2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712 2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7d0600100 of size 60817152 2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912 2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7e6c00000 of size 4194560 2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560 2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400 2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200 2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256 2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560 2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256 2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256 2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256 2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256 2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256 2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256 2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256 2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256 2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600 2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256 2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256 2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600 2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600 2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256 2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256 2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256 2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256 2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256 2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeef100 of size 256 2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256 2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256 2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256 2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256 2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256 2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256 2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256 2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256 2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256 2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeefb00 of size 256 2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256 2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600 2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eafd0d00 of size 1228800 2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600 2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb1ddd00 of size 137984 2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112 2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb21d500 of size 6171392 2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472 2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256 2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256 2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440 2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256 2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920 2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256 2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256 2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256 2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280 2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472 2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256 2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256 2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256 2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256 2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256 2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256 2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440 2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256 2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920 2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256 2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256 2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256 2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256 2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256 2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256 2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256 2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256 2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256 2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256 2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256 2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256 2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256 2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256 2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256 2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256 2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256 2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256 2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256 2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256 2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256 2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256 2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256 2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256 2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256 2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256 2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256 2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200 2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192 2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152 2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600 2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544 2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600 2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560 2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600 2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600 2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600 2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200 2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608 2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB 2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB 2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB 2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB 2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB 2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB 2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB 2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB 2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB 2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB 2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB 2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB 2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB 2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB 2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB 2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB 2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB 2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB 2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 16035977626 InUse: 60618752 MaxInUse: 64813312 NumAllocs: 139 MaxAllocSize: 12582912

2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfcallocator.cc:271] *****____****xx__**** 2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700 2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480 2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700 2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480 Traceback (most recent call last): File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "objectTrackingSingleFrame.py", line 358, in objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 156, in objectTracker opti1.runOptimization(session, 1, {loadData:True}) File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed result = method(*args, **kw) File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization session.run(self.optOp, feed_dict=feedDict) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'map/while/sub_1', defined at: File "objectTrackingSingleFrame.py", line 358, in objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 92, in objectTracker icpLoss = icp.getLoss(finalMesh.vUnClipped) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn maximum_iterations=n) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop return_same_structure) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop body_result = body(packed_vars_for_body) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in body = lambda i, lv: (i + 1, orig_body(lv)) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute packed_fn_values = fn(packed_values) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1)) # N, File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper return func(x, y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub "Sub", x=x, y=y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Lxxiaodiao commented 6 months ago

To be sure, I show the full text of the error.

(HOnnotate) initial@d3050eb4fa4f:~/workspace/HOnnotate/optimization$ python objectTrackingSingleFrame.py --seq 'test' /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/IPython/lib/pretty.py:91: DeprecationWarning: IPython.utils.signatures backport for Python 2 is deprecated in IPython 6, which only supports Python 3 from IPython.utils.signatures import signature /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead return _inspect.getargspec(target) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:225: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20. Instructions for updating: Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:

    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)

Additionally, to use tf.print in python 2.7, users must make sure to import the following:

from __future__ import print_function

/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/chumpy/ch.py:1203: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead want_out = 'out' in inspect.getargspec(func).args /home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/numpy/matrixlib/defmatrix.py:68: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) /home/initial/workspace/HOnnotate/optimization/object/batch_object.py:65: RuntimeWarning: invalid value encountered in true_divide vn = mesh.vn / np.expand_dims(np.linalg.norm(mesh.vn, ord=2, axis=1), 1) # normalize to unit vec /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) /home/initial/workspace/HOnnotate/optimization/ghope/loss.py:106: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future. gaussKernel = np.expand_dims(gaussKernel, 3) WARNING:tensorflow:From /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112: arg_min (from tensorflow.python.ops.gen_math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use argmin instead 2024-01-14 10:50:00.623402: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2024-01-14 10:50:00.999041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: Quadro P5000 major: 6 minor: 1 memoryClockRate(GHz): 1.7335 pciBusID: 0000:03:00.0 totalMemory: 15.90GiB freeMemory: 15.78GiB 2024-01-14 10:50:01.314176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.725 pciBusID: 0000:02:00.0 totalMemory: 23.70GiB freeMemory: 22.76GiB 2024-01-14 10:50:01.314243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1 2024-01-14 10:50:28.016592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2024-01-14 10:50:28.016640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2024-01-14 10:50:28.016651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N 2024-01-14 10:50:28.016657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N 2024-01-14 10:50:28.016832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15293 MB memory) -> physical GPU (device: 0, name: Quadro P5000, pci bus id: 0000:03:00.0, compute capability: 6.1) 2024-01-14 10:50:28.033847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22064 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:02:00.0, compute capability: 8.6) Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). maskPC for Image test/0/00002 is 0.132587 [Loading New frame ][0/00002] 2024-01-14 10:50:47.882958: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 29.75GiB. Current allocation summary follows. 2024-01-14 10:50:47.883086: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 68, Chunks in use: 66. 17.0KiB allocated for chunks. 16.5KiB in use in bin. 788B client-requested in use in bin. 2024-01-14 10:50:47.883121: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883151: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883177: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883229: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 2, Chunks in use: 2. 18.5KiB allocated for chunks. 18.5KiB in use in bin. 18.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883281: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 2, Chunks in use: 2. 97.5KiB allocated for chunks. 97.5KiB in use in bin. 97.2KiB client-requested in use in bin. 2024-01-14 10:50:47.883309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 3, Chunks in use: 3. 301.8KiB allocated for chunks. 301.8KiB in use in bin. 301.4KiB client-requested in use in bin. 2024-01-14 10:50:47.883358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 0. 134.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883390: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 3, Chunks in use: 3. 1014.2KiB allocated for chunks. 1014.2KiB in use in bin. 900.0KiB client-requested in use in bin. 2024-01-14 10:50:47.883417: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 10, Chunks in use: 10. 8.79MiB allocated for chunks. 8.79MiB in use in bin. 8.79MiB client-requested in use in bin. 2024-01-14 10:50:47.883442: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 4, Chunks in use: 3. 5.17MiB allocated for chunks. 4.00MiB in use in bin. 3.84MiB client-requested in use in bin. 2024-01-14 10:50:47.883468: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 5.61MiB allocated for chunks. 5.61MiB in use in bin. 4.20MiB client-requested in use in bin. 2024-01-14 10:50:47.883495: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 5, Chunks in use: 3. 23.89MiB allocated for chunks. 14.00MiB in use in bin. 14.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883525: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 24.00MiB allocated for chunks. 24.00MiB in use in bin. 20.00MiB client-requested in use in bin. 2024-01-14 10:50:47.883550: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 58.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883600: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883625: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883649: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2024-01-14 10:50:47.883675: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 29.75GiB was 256.00MiB, Chunk State: 2024-01-14 10:50:47.883700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7d0000000 of size 6291712 2024-01-14 10:50:47.883722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7d0600100 of size 60817152 2024-01-14 10:50:47.883743: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e6000000 of size 12582912 2024-01-14 10:50:47.883763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7e6c00000 of size 4194560 2024-01-14 10:50:47.883784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7000100 of size 4194560 2024-01-14 10:50:47.883805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7e7400200 of size 12582400 2024-01-14 10:50:47.883825: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea800000 of size 307200 2024-01-14 10:50:47.883845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b000 of size 256 2024-01-14 10:50:47.883865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ea84b100 of size 4194560 2024-01-14 10:50:47.883885: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b200 of size 256 2024-01-14 10:50:47.883905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b300 of size 256 2024-01-14 10:50:47.883924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b400 of size 256 2024-01-14 10:50:47.883944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b500 of size 256 2024-01-14 10:50:47.883963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b600 of size 256 2024-01-14 10:50:47.883982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b700 of size 256 2024-01-14 10:50:47.884001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b800 of size 256 2024-01-14 10:50:47.884021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4b900 of size 256 2024-01-14 10:50:47.884040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eac4ba00 of size 921600 2024-01-14 10:50:47.884060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2ca00 of size 256 2024-01-14 10:50:47.884079: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cb00 of size 256 2024-01-14 10:50:47.884098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7ead2cc00 of size 921600 2024-01-14 10:50:47.884117: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eae0dc00 of size 921600 2024-01-14 10:50:47.884136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeec00 of size 256 2024-01-14 10:50:47.884155: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeed00 of size 256 2024-01-14 10:50:47.884174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeee00 of size 256 2024-01-14 10:50:47.884193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeeef00 of size 256 2024-01-14 10:50:47.884213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef000 of size 256 2024-01-14 10:50:47.884233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeef100 of size 256 2024-01-14 10:50:47.884251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef200 of size 256 2024-01-14 10:50:47.884270: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef300 of size 256 2024-01-14 10:50:47.884289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef400 of size 256 2024-01-14 10:50:47.884308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef500 of size 256 2024-01-14 10:50:47.884327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef600 of size 256 2024-01-14 10:50:47.884346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef700 of size 256 2024-01-14 10:50:47.884365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef800 of size 256 2024-01-14 10:50:47.884384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeef900 of size 256 2024-01-14 10:50:47.884403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefa00 of size 256 2024-01-14 10:50:47.884421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eaeefb00 of size 256 2024-01-14 10:50:47.884440: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefc00 of size 256 2024-01-14 10:50:47.884459: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eaeefd00 of size 921600 2024-01-14 10:50:47.884479: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eafd0d00 of size 1228800 2024-01-14 10:50:47.884498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb0fcd00 of size 921600 2024-01-14 10:50:47.884517: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb1ddd00 of size 137984 2024-01-14 10:50:47.884536: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc7eb1ff800 of size 122112 2024-01-14 10:50:47.884556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fc7eb21d500 of size 6171392 2024-01-14 10:50:47.884576: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a00000 of size 9472 2024-01-14 10:50:47.884596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02500 of size 256 2024-01-14 10:50:47.884615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02600 of size 256 2024-01-14 10:50:47.884635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a02700 of size 93440 2024-01-14 10:50:47.884654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19400 of size 256 2024-01-14 10:50:47.884674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a19500 of size 49920 2024-01-14 10:50:47.884693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25800 of size 256 2024-01-14 10:50:47.884712: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25900 of size 256 2024-01-14 10:50:47.884731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25a00 of size 256 2024-01-14 10:50:47.884751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a25b00 of size 1280 2024-01-14 10:50:47.884771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a26000 of size 9472 2024-01-14 10:50:47.884790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28500 of size 256 2024-01-14 10:50:47.884810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28600 of size 256 2024-01-14 10:50:47.884829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28700 of size 256 2024-01-14 10:50:47.884848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28800 of size 256 2024-01-14 10:50:47.884867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28900 of size 256 2024-01-14 10:50:47.884887: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28a00 of size 256 2024-01-14 10:50:47.884906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a28b00 of size 93440 2024-01-14 10:50:47.884925: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f800 of size 256 2024-01-14 10:50:47.884944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a3f900 of size 49920 2024-01-14 10:50:47.884963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bc00 of size 256 2024-01-14 10:50:47.884983: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bd00 of size 256 2024-01-14 10:50:47.885002: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4be00 of size 256 2024-01-14 10:50:47.885021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4bf00 of size 256 2024-01-14 10:50:47.885040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c000 of size 256 2024-01-14 10:50:47.885059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c100 of size 256 2024-01-14 10:50:47.885078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c200 of size 256 2024-01-14 10:50:47.885098: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c300 of size 256 2024-01-14 10:50:47.885118: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c400 of size 256 2024-01-14 10:50:47.885138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c500 of size 256 2024-01-14 10:50:47.885157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c600 of size 256 2024-01-14 10:50:47.885177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c700 of size 256 2024-01-14 10:50:47.885196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c800 of size 256 2024-01-14 10:50:47.885215: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4c900 of size 256 2024-01-14 10:50:47.885235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ca00 of size 256 2024-01-14 10:50:47.885253: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cb00 of size 256 2024-01-14 10:50:47.885272: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cc00 of size 256 2024-01-14 10:50:47.885292: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cd00 of size 256 2024-01-14 10:50:47.885311: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4ce00 of size 256 2024-01-14 10:50:47.885330: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4cf00 of size 256 2024-01-14 10:50:47.885349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d000 of size 256 2024-01-14 10:50:47.885368: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d100 of size 256 2024-01-14 10:50:47.885387: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d200 of size 256 2024-01-14 10:50:47.885406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d300 of size 256 2024-01-14 10:50:47.885424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d400 of size 256 2024-01-14 10:50:47.885444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d500 of size 256 2024-01-14 10:50:47.885464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d600 of size 256 2024-01-14 10:50:47.885482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a4d700 of size 307200 2024-01-14 10:50:47.885502: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2a98700 of size 424192 2024-01-14 10:50:47.885523: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f2e00000 of size 2097152 2024-01-14 10:50:47.885543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3000000 of size 921600 2024-01-14 10:50:47.885563: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f30e1000 of size 1260544 2024-01-14 10:50:47.885582: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3214c00 of size 921600 2024-01-14 10:50:47.885602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f32f5c00 of size 1090560 2024-01-14 10:50:47.885622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3400000 of size 921600 2024-01-14 10:50:47.885640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f34e1000 of size 921600 2024-01-14 10:50:47.885659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f35c2000 of size 921600 2024-01-14 10:50:47.885679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f36a3000 of size 1843200 2024-01-14 10:50:47.885700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fc8f3865000 of size 3780608 2024-01-14 10:50:47.885720: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size: 2024-01-14 10:50:47.885745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 66 Chunks of size 256 totalling 16.5KiB 2024-01-14 10:50:47.885769: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB 2024-01-14 10:50:47.885791: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 9472 totalling 18.5KiB 2024-01-14 10:50:47.885813: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 49920 totalling 97.5KiB 2024-01-14 10:50:47.885836: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 93440 totalling 182.5KiB 2024-01-14 10:50:47.885858: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 122112 totalling 119.2KiB 2024-01-14 10:50:47.885880: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 307200 totalling 600.0KiB 2024-01-14 10:50:47.885901: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 424192 totalling 414.2KiB 2024-01-14 10:50:47.885923: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 921600 totalling 8.79MiB 2024-01-14 10:50:47.885945: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1090560 totalling 1.04MiB 2024-01-14 10:50:47.885966: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1260544 totalling 1.20MiB 2024-01-14 10:50:47.885987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1843200 totalling 1.76MiB 2024-01-14 10:50:47.886008: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2097152 totalling 2.00MiB 2024-01-14 10:50:47.886029: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 3780608 totalling 3.61MiB 2024-01-14 10:50:47.886050: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 4194560 totalling 8.00MiB 2024-01-14 10:50:47.886071: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6291712 totalling 6.00MiB 2024-01-14 10:50:47.886094: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582400 totalling 12.00MiB 2024-01-14 10:50:47.886116: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12582912 totalling 12.00MiB 2024-01-14 10:50:47.886138: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 57.81MiB 2024-01-14 10:50:47.886165: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats: Limit: 16035977626 InUse: 60618752 MaxInUse: 64813312 NumAllocs: 139 MaxAllocSize: 12582912

2024-01-14 10:50:47.886214: W tensorflow/core/common_runtime/bfcallocator.cc:271] *****____****xx__**** 2024-01-14 10:50:47.901539: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc 2024-01-14 10:50:48.043748: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e5fff700 2024-01-14 10:50:48.303976: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e5fff700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.664465: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_egl.cpp:266] reinitialised framebuffer with size 320 x 480 2024-01-14 10:50:48.726655: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:66] selected egl device #1 to match cuda device #0 for thread 0x7fc7e57fe700 2024-01-14 10:50:48.749039: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/gl_common.h:84] successfully created new GL context on thread 0x7fc7e57fe700 (EGL = 1.5, GL = 4.6.0 NVIDIA 470.223.02, renderer = Quadro P5000/PCIe/SSE2) 2024-01-14 10:50:48.751036: I /home/initial/workspace/thirdparty_for_HOnnotate/dirt/csrc/rasterise_grad_egl.cpp:314] reinitialised framebuffer with size 320 x 480 Traceback (most recent call last): File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node map/while/sub_1}} = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "objectTrackingSingleFrame.py", line 358, in objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 156, in objectTracker opti1.runOptimization(session, 1, {loadData:True}) File "/home/initial/workspace/HOnnotate/optimization/ghope/utils.py", line 159, in timed result = method(*args, **kw) File "/home/initial/workspace/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization session.run(self.optOp, feed_dict=feedDict) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'map/while/sub_1', defined at: File "objectTrackingSingleFrame.py", line 358, in objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData) File "objectTrackingSingleFrame.py", line 92, in objectTracker icpLoss = icp.getLoss(finalMesh.vUnClipped) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 191, in getLoss loss = tf.map_fn(self.getLossPerFrame, (finalMesh, self.pcl2DAllViews, self.realObservs.seg, segColor), parallel_iterations=int(self.numViews), dtype=tf.float32) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 494, in map_fn maximum_iterations=n) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3291, in while_loop return_same_structure) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3004, in BuildLoop pred, body, original_loop_vars, loop_vars, shape_invariants) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2939, in _BuildLoop body_result = body(packed_vars_for_body) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3260, in body = lambda i, lv: (i + 1, orig_body(lv)) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/functional_ops.py", line 483, in compute packed_fn_values = fn(packed_values) File "/home/initial/workspace/HOnnotate/optimization/ghope/icp.py", line 112, in getLossPerFrame tf.arg_min(tf.reduce_sum(tf.abs(pclValid - verts), axis=2), dimension=1)) # N, File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 866, in binary_op_wrapper return func(x, y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 8318, in sub "Sub", x=x, y=y, name=name) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/initial/miniconda3/envs/HOnnotate/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[7617,262146,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node map/while/sub_1 (defined at /home/initial/workspace/HOnnotate/optimization/ghope/icp.py:112) = Sub[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/ExpandDims, map/while/TensorArrayReadV3)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Hello brother, I have the same problem as you. Have you solved it now?