search

NVlabs / NVBit

199 stars 18 forks source link

How to use nvbit to instrcutment pytroch applications #72

Closed luckyq closed 1 year ago

luckyq commented 2 years ago

Now, I have a deep learning task that used PyTorch. The command is as follows: LD_PRELOAD=mem_trace/mem_trace.so python examples/mnist/main.py

It will not report an error. But also no traces are generated.

Does it need some preprocessing for running this?

Thanks in advance.

ovilla commented 2 years ago

You could try CUDA_INJECTION64_PATH instead of LD_PRELOAD, sometime machine learning frameworks override LD_PRELOAD before the application is actually launched

Diksha-Moolchandani commented 1 year ago

Was this resolved? I am getting the following error with the instruction count tool: Cuda error in function 'cudaDeviceSynchronize()' file 'instr_count.cu' in line 226 : an illegal memory access was encountered. @luckyq @ovilla if it is resolved, please let me know.

ovilla commented 1 year ago

As far as we know there is no issue in instrumenting pytorch applications. The error on the illegal memory access is very likely something unrelated and worth debugging (after proper repro test case).

Diksha-Moolchandani commented 1 year ago

Thanks it is resolved

elijahberscheid commented 9 months ago

@luckyq how did you resolve the issue? I am having the same problem with my pytorch program.