search

BichenWuUCB / squeezeDet

A tensorflow implementation for SqueezeDet, a convolutional neural network for object detection.
BSD 2-Clause "Simplified" License
739 stars 306 forks source link

train squeezedet on KITTI #103

Open ismet29 opened 6 years ago

ismet29 commented 6 years ago

Hi, I encountered a problem when I trained the model on KITTI, I have the following errors: OS : win7

I launched the follopwing command to start the training : python ./src/train.py --dataset=KITTI --pretrained_model_path=./data/squeezenet_v1.0_SR_0.750.pkl --data_path=./data/KITTI --image_set=train --train_dir=./train12 --net=squeezeDet --summary_step=100 --checkpoint_step=500 --gpu 0

Thx for replying.

2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B46EC8500 of size 589824 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B46F58500 of size 131072 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B46F78500 of size 65536 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B46F88500 of size 589824 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B47018500 of size 196608 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B47048500 of size 147456 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B4706C500 of size 1327104 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B471B0500 of size 294912 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B471F8500 of size 147456 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B4721C500 of size 1558016 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B47398B00 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B47933100 of size 663552000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B6F203100 of size 663552000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000B96AD3100 of size 1327104000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000BE5C73100 of size 663552000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000C0D543100 of size 663552000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000C34E13100 of size 663552000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000C5C6E3100 of size 1327104000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CAB883100 of size 82944000 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Free at 0000000CB079D100 of size 250272768 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CBF64AD00 of size 17625600 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC0719F00 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC0CB4500 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC124EB00 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC17E9100 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC1D83700 of size 5875200 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC231DD00 of size 1990656 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC2503D00 of size 36864 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC250CD00 of size 36864 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC2515D00 of size 16384 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC2519D00 of size 16384 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC251DD00 of size 147456 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC2541D00 of size 32768 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC2549D00 of size 16384 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:665] Chunk at 0000000CC254DD00 of size 263613184 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:671] Summary of in-use Chunks by size: 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 183 Chunks of size 256 totalling 45.8KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 21 Chunks of size 512 totalling 10.5KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 12 Chunks of size 768 totalling 9.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 12 Chunks of size 1024 totalling 12.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 12 Chunks of size 1536 totalling 18.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 10 Chunks of size 4096 totalling 40.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 2 Chunks of size 6912 totalling 13.5KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 4 Chunks of size 8192 totalling 32.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 10 Chunks of size 16384 totalling 160.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 28672 totalling 28.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 4 Chunks of size 32768 totalling 128.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 12 Chunks of size 36864 totalling 432.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 40960 totalling 40.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 4 Chunks of size 49152 totalling 192.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 7 Chunks of size 65536 totalling 448.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 3 Chunks of size 73728 totalling 216.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 5 Chunks of size 98304 totalling 480.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 114688 totalling 112.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 3 Chunks of size 131072 totalling 384.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 13 Chunks of size 147456 totalling 1.83MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 6 Chunks of size 196608 totalling 1.13MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 293888 totalling 287.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 5 Chunks of size 294912 totalling 1.41MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 295936 totalling 289.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 8 Chunks of size 331776 totalling 2.53MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 5 Chunks of size 589824 totalling 2.81MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 995328 totalling 972.0KiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 1130496 totalling 1.08MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 5 Chunks of size 1327104 totalling 6.33MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 1558016 totalling 1.49MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 1843200 totalling 1.76MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 4 Chunks of size 1990656 totalling 7.59MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 12 Chunks of size 5875200 totalling 67.24MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 2 Chunks of size 17625600 totalling 33.62MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 23500800 totalling 22.41MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 82944000 totalling 79.10MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 165888000 totalling 158.20MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 263613184 totalling 251.40MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 331776000 totalling 316.41MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 1 Chunks of size 501350400 totalling 478.13MiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 5 Chunks of size 663552000 totalling 3.09GiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:674] 2 Chunks of size 1327104000 totalling 2.47GiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:678] Sum Total of in-use chunks: 6.97GiB 2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:680] Stats: Limit: 7730841600 InUse: 7480552704 MaxInUse: 7480552704 NumAllocs: 513 MaxAllocSize: 2654208000

2018-05-18 12:11:09.236327: W T:\src\github\tensorflow\tensorflow\core\common_ru ntime\bfc_allocator.cc:279] **** *****__***x 2018-05-18 12:11:09.236327: W T:\src\github\tensorflow\tensorflow\core\framework \op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:386 : Resource exhausted: OOM when allocating tensor with shape[20,135,240,128] and type float on /job:loc alhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1322, in _do_call return fn(args) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat ing tensor with shape[20,135,240,128] and type float on /job:localhost/replica:0 /task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on _gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze 1x1/relu, fire4/expand1x1/kernels/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recvdevice="/job:localhost/replica:0/task:0/device:CPU:0", send device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1 , tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./src/train.py", line 401, in tf.app.run() File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\platform\app.py", line 126, in run _sys.exit(main(argv)) File "./src/train.py", line 397, in main train() File "./src/train.py", line 340, in train op_list, feed_dict=feed_dict) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 900, in run run_metadata_ptr) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1316, in _do_run run_metadata) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\client\session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat ing tensor with shape[20,135,240,128] and type float on /job:localhost/replica:0 /task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on _gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze 1x1/relu, fire4/expand1x1/kernels/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recvdevice="/job:localhost/replica:0/task:0/device:CPU:0", send device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1 , tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'fire4/expand1x1/convolution', defined at: File "./src/train.py", line 401, in tf.app.run() File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\platform\app.py", line 126, in run _sys.exit(main(argv)) File "./src/train.py", line 397, in main train() File "./src/train.py", line 130, in train model = SqueezeDet(mc) File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 25, in init self._add_forward_graph() File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 55, in _add_forward_gr aph 'fire4', pool3, s1x1=32, e1x1=128, e3x3=128, freeze=False) File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 102, in _fire_layer padding='SAME', stddev=stddev, freeze=freeze) File "E:\ismet\squeezeDet\src\nn_skeleton.py", line 567, in _conv_layer name='convolution') File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\ops\gen_nn_ops.py", line 1042, in conv2d data_format=data_format, dilations=dilations, name=name) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\framework\ops.py", line 3392, in create_op op_def=op_def) File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt hon\framework\ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected- access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor wit h shape[20,135,240,128] and type float on /job:localhost/replica:0/task:0/device :GPU:0 by allocator GPU_0_bfc [[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on _gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze 1x1/relu, fire4/expand1x1/kernels/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recvdevice="/job:localhost/replica:0/task:0/device:CPU:0", send device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1 , tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] Hint: If you want to see a list of allocated tensors when OOM happens, add repor t_tensor_allocations_upon_oom to RunOptions for current allocation info.