it seems that the gpu memory could not be free between iterations or rounds in benchmark,

[howin98]

whether:

def test_alexnet_batch_size1(benchmark):
    benchmark.pedantic(run_alexnet_batch_size1, rounds=50)

or

def test_alexnet_batch_size1(benchmark):
    benchmark.pedantic(run_alexnet_batch_size1, iterations=50)

the output is: platform linux -- Python 3.7.10, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python3 benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /home/ci-user/runners/provision/_work/get-oneflow/get-oneflow/flow_vision plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0 collecting ... collected 5 items

flow_vision/benchmark/test_alexnet.py::test_alexnet_batch_size16 loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 W20220329 05:37:29.127727 139 cuda_allocator.cpp:282] OOM error is detected, process will exit. And it will start to reset CUDA device for releasing device memory. F20220329 05:37:30.156129 139 cuda_allocator.cpp:285] Error! : Out of memory when allocate size : 150994944. The total_memory_bytes allocated by this CudaAllocator is : 4907335680 Check failure stack trace: @ 0x7fdf4caaa2ea (unknown) @ 0x7fdf4caaa5d2 (unknown) @ 0x7fdf4caa9e57 (unknown) @ 0x7fdf4caac9c9 (unknown) @ 0x7fdf46ed1c3a oneflow::vm::CudaAllocator::Allocate() @ 0x7fdf46edfb2d oneflow::vm::ThreadSafeAllocator::Allocate() @ 0x7fdf44875120 oneflow::vm::EagerBlobObject::TryAllocateBlobBodyMemory() @ 0x7fdf4487cd5f oneflow::vm::LocalCallOpKernelUtil::AllocateOutputBlobsMemory() @ 0x7fdf4487d6bf oneflow::vm::LocalCallOpKernelUtil::Compute() @ 0x7fdf4487c68b oneflow::vm::LocalCallOpKernelInstructionType::ComputeInFuseMode() @ 0x7fdf46ed7ce6 oneflow::vm::FuseInstructionType<>::Compute() @ 0x7fdf46ed680a oneflow::vm::CudaStreamType::Compute() @ 0x7fdf46eebd94 oneflow::vm::VirtualMachineEngine::DispatchInstruction() @ 0x7fdf46eec94f oneflow::vm::VirtualMachineEngine::DispatchAndPrescheduleInstructions() @ 0x7fdf46ef1d18 oneflow::vm::VirtualMachineEngine::Schedule() @ 0x7fdf46ee2a10 oneflow::VirtualMachine::ScheduleLoop() @ 0x7fdf4f4bd82f (unknown) @ 0x7fdf8a2bc6db start_thread @ 0x7fdf89fe561f clone Fatal Python error: Aborted

Thread 0x00007fdf8a6ed740 (most recent call first): File "/opt/conda/lib/python3.7/site-packages/oneflow/framework/tensor.py", line 985 in _numpy File "/home/ci-user/runners/provision/_work/get-oneflow/get-oneflow/flow_vision/benchmark/test_alexnet.py", line 22 in run_alexnet_batch_size16 File "/opt/conda/lib/python3.7/site-packages/pytest_benchmark/fixture.py", line 97 in runner File "/opt/conda/lib/python3.7/site-packages/pytest_benchmark/fixture.py", line 222 in _raw_pedantic File "/opt/conda/lib/python3.7/site-packages/pytest_benchmark/fixture.py", line 140 in pedantic File "/home/ci-user/runners/provision/_work/get-oneflow/get-oneflow/flow_vision/benchmark/test_alexnet.py", line 27 in test_alexnet_batch_size16 File "/opt/conda/lib/python3.7/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call File "/opt/conda/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/opt/conda/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/opt/conda/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/opt/conda/lib/python3.7/site-packages/_pytest/python.py", line 1761 in runtest File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call File "/opt/conda/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/opt/conda/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/opt/conda/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 259 in File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 338 in from_call File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 259 in call_runtest_hook File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 219 in call_and_report File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 130 in runtestprotocol File "/opt/conda/lib/python3.7/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol File "/opt/conda/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/opt/conda/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/opt/conda/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 347 in pytest_runtestloop File "/opt/conda/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/opt/conda/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/opt/conda/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 322 in _main File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 268 in wrap_session File "/opt/conda/lib/python3.7/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main File "/opt/conda/lib/python3.7/site-packages/pluggy/_callers.py", line 39 in _multicall File "/opt/conda/lib/python3.7/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/opt/conda/lib/python3.7/site-packages/pluggy/_hooks.py", line 265 in call File "/opt/conda/lib/python3.7/site-packages/_pytest/config/init.py", line 165 in main File "/opt/conda/lib/python3.7/site-packages/_pytest/config/init.py", line 187 in console_main File "/opt/conda/lib/python3.7/site-packages/pytest/main.py", line 5 in File "/opt/conda/lib/python3.7/runpy.py", line 85 in _run_code File "/opt/conda/lib/python3.7/runpy.py", line 193 in _run_module_as_main Error: Error: The process '/usr/bin/docker' failed with exit code 134

[howin98]

tks a lot for any suggestions