Vaitrace w/ DPU-PYNQ

[tafk7]

Is it possible to run vaitrace with DPU-PYNQ to generate a CSV file for Vitis AI Profiler?

I moved my functioning DPU-PYNQ code from a notebook to a python file and ran this from the command line on my ZCU111:

python -m vaitrace_py ./my_test.py <path-to-xmodel>/model.xmodel

And got this result:

WARNING:root:No such file or directory: '/sys/kernel/debug/tracing/trace_clock'
WARNING:root:CPU function tracing feature is disabled
INFO:root:VART will run xmodel in [NORMAL] mode
INFO:root:vaitrace compile python code: ./my_test.py
INFO:root:vaitrace exec poython code: ./my_test.py
Exception in thread Thread-1 (pyRunCtx):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/bin/xlnx/vaitrace/vaitracePyRunner.py", line 52, in pyRunCtx
    exec(code, globs, None)
  File "./my_test.py", line 2, in <module>
    from pynq_dpu import DpuOverlay
ModuleNotFoundError: No module named 'pynq_dpu'
INFO:root:vaitrace timeout, waiting for Python thread terminated
APM Stop Collecting
INFO:root:Generating VTF
ERROR:root:[]
ERROR:root:xmodel info raw info error
INFO:root:[vart_trace.csv]: Events number: 0

[nerbloc]

I am having the exact same problem. Have you found any solutions?

[skalade]

Hi there, pynq_dpu is installed in the python pynq-venv virtual environment, seems like you might be running vaitrace outside of it and that's why it can't find the pynq_dpu package. Try running it from the venv:

sudo su
source /etc/profile.d/pynq_venv.sh
vaitrace ...

Alternatively, if you run your scripts from the jupyterlab terminal it's automatically launched from pynq-venv so you shouldn't have to worry about sourcing that script.

[nerbloc]

Hi @skalade , I have tried running from venv and from the Jupyterlab terminal and it does not work. I get the same error as the first time.

[skalade]

Alright, taking a look at the vaitrace code it appears that the python shebang prevents it from running in the virtual environment. There's two files you need to edit to get this to work. First one:

/usr/bin/vaitrace.py

replace #!/usr/bin/python3 -> #!/usr/bin/env python.

Second one I had to to edit was:

/usr/bin/xlnx/vaitrace/vaitracePyRunner.py

add these lines just before exec(code, globs, None) (around line 52)

import asyncio
asyncio.set_event_loop(asyncio.new_event_loop())

The reason for starting another event loop here is that vaitrace appears to destroy the asyncio event loop pynq relies on. If you don't start a new event loop you should see an error like this:

RuntimeError: There is no current event loop in thread 'Thread-2 (pyRunCtx)'.

Not sure if there's some defensive programming we can do in xrt_device or this should be addressed in vaitrace, but hopefully this workaround suffices for now.

[nerbloc]

Thank you very much. I was able to run vaitrace from the jupyterlab terminal by editing the two files. But these warnings are not resolved.

WARNING:root:No such file or directory: '/sys/kernel/debug/tracing/trace_clock'
WARNING:root:CPU function tracing feature is disabled

Therefore, it is unable to trace the CPU. Also, vaitrace is timeout. Is this normal behavior?

INFO:root:vaitrace timeout, waiting for Python thread terminated

[skalade]

vaitrace isn't a tool we maintain or I'm too familiar with, so I can't really comment on any of these warnings. I assume there's a config you can pass it to enable CPU tracing. I'd refer to the Vitis AI docs https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/vaitrace-Usage or ask on the AMD support forums / Vitis AI github.

I see the same warnings running our mnist example as a python script to trace, however it does produce the basic info and csv's you'd expect.

root@pynq:/home/xilinx/jupyter_notebooks/pynq-dpu# python3 -m vaitrace_py test.py 
WARNING:root:No such file or directory: '/sys/kernel/debug/tracing/trace_clock'
WARNING:root:CPU function tracing feature is disabled
INFO:root:VART will run xmodel in [NORMAL] mode
INFO:root:vaitrace compile python code: test.py
INFO:root:vaitrace exec poython code: test.py
Total number of test images: 10000
  Dimension of each picture: 28x28
Classifying 10000 digit pictures ...
Overall accuracy: 0.9881
  Execution time: 3.3275s
      Throughput: 3005.2699FPS
INFO:root:vaitrace timeout, waiting for Python thread terminated
APM Stop Collecting
INFO:root:Generating VTF
INFO:root:Processing xmodel information
INFO:root:[Summary] DPU cores involved: 1
INFO:root:[Summary] Total inferences: 10000
INFO:root:[Summary] Overall inferences/second: 3005.82
INFO:root:[vart_trace.csv]: Events number: 20000

[nerbloc]

Ok, I understand. I will ask the question in other forums as well. Thank you for your assistance.