search

NVlabs / NVBit

198 stars 18 forks source link

Using Nvbit with docker #93

Closed mahmoodn closed 1 year ago

mahmoodn commented 1 year ago

Is it possible to use Nvbit with MLPerf docker image? I tried to run

(mlperf) mahmood@mlperf-inference-mnaderan-x86_64:/work$ LD_PRELOAD=/work/nvbit_release/tools/opcode_hist/opcode_hist.so ___RNNT command___ > output.txt
Segmentation fault (core dumped)

Has this been tested before?

ovilla commented 1 year ago

We use nvbit with docker images regularly. I am not sure what could be the issue. Can you please try running with VERBOSE=3 in front of LD_PRELOAD?

mahmoodn commented 1 year ago

OK. Apparently, when Nvbit is used, more memory is occupied on the device due to the overhead. Therefore, the batch_size of 128 which is fine when running the application alone, is not workable with Nvbit. When I reduced the batch_size, the problem was fixed.