search

Xilinx / Vitis-AI

Vitis AI is Xilinx’s development stack for AI inference on Xilinx hardware platforms, including both edge devices and Alveo cards.
https://www.xilinx.com/ai
Apache License 2.0
1.44k stars 624 forks source link

Vitis-AI profiler support for non ZOCL runtime #1202

Closed MaistoV closed 1 year ago

MaistoV commented 1 year ago

Does Vitis-AI profiler support:

Have there been any changes between v2.0 and v2.5?

Thank you for the help.

jileilamd commented 1 year ago

hi @MaistoV : 1.Does Vitis-AI profiler support: we support simple linux runtime without ZOCL(this same to Vivado flow) 2.Have there been any changes between v2.0 and v2.5? There is no changes from user perspective

MaistoV commented 1 year ago

I built the kernel and rootfs with Petalinux 2021.2 and enabled Kernel tracing as in https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/16-profiler_introduction/README.md . As a matter of fact, I can use kprobes and uprobes directly.

I am currently using design synthesized from Vivado flow from TRD from Vitis-AI 2.0 ( https://github.com/Xilinx/Vitis-AI/tree/v2.0/dsa/DPU-TRD/prj/Vivado ). I succesfully ran multiple CNNs on different designs with several DPU configurations and one or more DPU cores.

When I run vaitrace I get the following warning and no computing units are found:

I checked sysfs and none of the following devices, used by the tracer python scripts, can be found:

Edocit commented 1 year ago

I reported exactly the same issue in my post here but I used 2022.2 petalinux and vivado. For the Vitis-AI the version was 2.5. However, for VAI 3 it is even worse.

https://support.xilinx.com/s/question/0D74U000007Lvu4SAC/detail?language=en_US&fromEmail=1&s1oid=00D2E000000nHq7&s1nid=0DB2E000000XdtH&s1uid=0052E00000N3TnZ&s1ext=0&emkind=chatterCommentNotification&emtm=1679880861901

I also provided additional information. Please help us, I'm stuck waiting for a solution since I need to use the tracer.

jileilamd commented 1 year ago

@Edocit hello @Edocit I check your issue in our board and user vart version3.0: image The version: image

jileilamd commented 1 year ago

I built the kernel and rootfs with Petalinux 2021.2 and enabled Kernel tracing as in https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/16-profiler_introduction/README.md . As a matter of fact, I can use kprobes and uprobes directly.

I am currently using design synthesized from Vivado flow from TRD from Vitis-AI 2.0 ( https://github.com/Xilinx/Vitis-AI/tree/v2.0/dsa/DPU-TRD/prj/Vivado ). I succesfully ran multiple CNNs on different designs with several DPU configurations and one or more DPU cores.

When I run vaitrace I get the following warning and no computing units are found:

  • WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'

I checked sysfs and none of the following devices, used by the tracer python scripts, can be found:

  • '/sys/devices/platform/amba/amba:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/axi/axi:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/amba_pl@0/amba_pl@0:zyxclmm_drm/ip_layout'

hi @MaistoV
Maybe this warning has no effect on vaitrace, Whether the vaitrace works correctly?

Edocit commented 1 year ago

Can you detail the procedure you used to test with VAI 3.0. Did you used Vivado 2022.2 ? Did you create the project from BSP or from --template zynqMP ? Have you made something particular in the Vivado design ? I still have issues with petalinux 2022.2 (at boot it can't bring up ethernet and then when running the patch for the PMIC it says the PMIC is busy and can't talk to it via I2C). Thank you, Edoardo

MaistoV commented 1 year ago

I built the kernel and rootfs with Petalinux 2021.2 and enabled Kernel tracing as in https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/16-profiler_introduction/README.md . As a matter of fact, I can use kprobes and uprobes directly. I am currently using design synthesized from Vivado flow from TRD from Vitis-AI 2.0 ( https://github.com/Xilinx/Vitis-AI/tree/v2.0/dsa/DPU-TRD/prj/Vivado ). I succesfully ran multiple CNNs on different designs with several DPU configurations and one or more DPU cores. When I run vaitrace I get the following warning and no computing units are found:

  • WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'

I checked sysfs and none of the following devices, used by the tracer python scripts, can be found:

  • '/sys/devices/platform/amba/amba:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/axi/axi:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/amba_pl@0/amba_pl@0:zyxclmm_drm/ip_layout'

hi @MaistoV Maybe this warning has no effect on vaitrace, Whether the vaitrace works correctly?

It has effect, as a matter of fact, the sources of vaitrace are available here and this warning is generated from hwInfo.py

Edocit commented 1 year ago

Hi, what was the process you did to build and successfully test the platform ? Petalinux 2022.2 and Vivado 2022.2 and the master branch of Vitis-AI on github.

I create a project using the --template zynqMP option, then using petalinux-config --get-hw-description I imported the Vivado platform (the XSA) and changed in the DTG settings the platform to be zcu104-revc. Then added the DPU kernel module after petalinux-config -c kernel and added the needed stuff in order to use vaitrace according to https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Getting-Started-with-the-Vitis-AI-Profiler Then copied the 2 recipies and commented the line to disable vitis flow in vart.bb Added vitis-ai and vitis-ai-dev in the menu prompted by petalinux-config -c rootfs and enabled the packagegroup-selfhosted

After petalinux-build and package when the board boots I faced 2 issues: the first one is that I can no more succesfully run the dpu_optimize_script to adjust PMIC settings and then dnf (that I added in the rootfs before build) does not work with output message 'var/lock exist' message. I ran ip a command and I saw network was no more able to setup at boot and when I execute if up eth0 the interface becomes active but can't associate a DNS. It seems something missing in the rootfs, and I think it is something related to tmp filesystem directories.

Do you have some hint about it ? the Vivado platform is a simple design only including a dual core DPU. However, I can provide pictures about it if needed or if this can be useful for you to help me ?

Thank you, Edoardo

Il giorno lun 10 apr 2023 alle 08:22 jileilxinlix @.***> ha scritto:

@Edocit https://github.com/Edocit hello @Edocit https://github.com/Edocit I check your issue in our board and user vart version3.0: [image: image] https://user-images.githubusercontent.com/104976662/230839850-9255a069-bb9d-41c2-a801-78f10d0b5f15.png The version: [image: image] https://user-images.githubusercontent.com/104976662/230839965-dfffffd6-3b6e-424e-9d40-94fb8b927852.png

— Reply to this email directly, view it on GitHub https://github.com/Xilinx/Vitis-AI/issues/1202#issuecomment-1501448587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGZWTITCOYHUJOK7VM47ULXAORJDANCNFSM6AAAAAAWBHO3AQ . You are receiving this because you were mentioned.Message ID: @.***>

jileilamd commented 1 year ago

I built the kernel and rootfs with Petalinux 2021.2 and enabled Kernel tracing as in https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/16-profiler_introduction/README.md . As a matter of fact, I can use kprobes and uprobes directly. I am currently using design synthesized from Vivado flow from TRD from Vitis-AI 2.0 ( https://github.com/Xilinx/Vitis-AI/tree/v2.0/dsa/DPU-TRD/prj/Vivado ). I succesfully ran multiple CNNs on different designs with several DPU configurations and one or more DPU cores. When I run vaitrace I get the following warning and no computing units are found:

  • WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'

I checked sysfs and none of the following devices, used by the tracer python scripts, can be found:

  • '/sys/devices/platform/amba/amba:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/axi/axi:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/amba_pl@0/amba_pl@0:zyxclmm_drm/ip_layout'

hi @MaistoV Maybe this warning has no effect on vaitrace, Whether the vaitrace works correctly?

It has effect, as a matter of fact, the sources of vaitrace are available here and this warning is generated from hwInfo.py

@MaistoV :

  1. After we double check the code, this warning not effect vaitrace, because we not use hare ware info data in vaitrace version2.0.This hwinfo data just collect. But we still thank your for your commit, for vivado flow we should not use this method collect hwinfo data. In addition, do you run vaitrace success except this warnning?

BR Jilei

jileilamd commented 1 year ago

Hi, what was the process you did to build and successfully test the platform ? Petalinux 2022.2 and Vivado 2022.2 and the master branch of Vitis-AI on github. I create a project using the --template zynqMP option, then using petalinux-config --get-hw-description I imported the Vivado platform (the XSA) and changed in the DTG settings the platform to be zcu104-revc. Then added the DPU kernel module after petalinux-config -c kernel and added the needed stuff in order to use vaitrace according to https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Getting-Started-with-the-Vitis-AI-Profiler Then copied the 2 recipies and commented the line to disable vitis flow in vart.bb Added vitis-ai and vitis-ai-dev in the menu prompted by petalinux-config -c rootfs and enabled the packagegroup-selfhosted After petalinux-build and package when the board boots I faced 2 issues: the first one is that I can no more succesfully run the dpu_optimize_script to adjust PMIC settings and then dnf (that I added in the rootfs before build) does not work with output message 'var/lock exist' message. I ran ip a command and I saw network was no more able to setup at boot and when I execute if up eth0 the interface becomes active but can't associate a DNS. It seems something missing in the rootfs, and I think it is something related to tmp filesystem directories. Do you have some hint about it ? the Vivado platform is a simple design only including a dual core DPU. However, I can provide pictures about it if needed or if this can be useful for you to help me ? Thank you, Edoardo Il giorno lun 10 apr 2023 alle 08:22 jileilxinlix @.> ha scritto: … @Edocit https://github.com/Edocit hello @Edocit https://github.com/Edocit I check your issue in our board and user vart version3.0: [image: image] https://user-images.githubusercontent.com/104976662/230839850-9255a069-bb9d-41c2-a801-78f10d0b5f15.png The version: [image: image] https://user-images.githubusercontent.com/104976662/230839965-dfffffd6-3b6e-424e-9d40-94fb8b927852.png — Reply to this email directly, view it on GitHub <#1202 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGZWTITCOYHUJOK7VM47ULXAORJDANCNFSM6AAAAAAWBHO3AQ . You are receiving this because you were mentioned.Message ID: @.>

hi @Edocit : Our image build and vitis-ai install flow:https://github.com/Xilinx/Vitis-AI/blob/master/dpu/ref_design_docs/README_DPUCZ_Vitis.md ,we use vitis flow. If you want to use vivado flow, you should follow:https://github.com/Xilinx/Vitis-AI/blob/master/dpu/ref_design_docs/README_DPUCZ_Vivado.md

I will recurrence your issue in my environment follow your step, but i need more time.

MaistoV commented 1 year ago

I built the kernel and rootfs with Petalinux 2021.2 and enabled Kernel tracing as in https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/16-profiler_introduction/README.md . As a matter of fact, I can use kprobes and uprobes directly. I am currently using design synthesized from Vivado flow from TRD from Vitis-AI 2.0 ( https://github.com/Xilinx/Vitis-AI/tree/v2.0/dsa/DPU-TRD/prj/Vivado ). I succesfully ran multiple CNNs on different designs with several DPU configurations and one or more DPU cores. When I run vaitrace I get the following warning and no computing units are found:

  • WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'

I checked sysfs and none of the following devices, used by the tracer python scripts, can be found:

  • '/sys/devices/platform/amba/amba:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/axi/axi:zyxclmm_drm/ip_layout',
  • '/sys/devices/platform/amba_pl@0/amba_pl@0:zyxclmm_drm/ip_layout'

hi @MaistoV Maybe this warning has no effect on vaitrace, Whether the vaitrace works correctly?

It has effect, as a matter of fact, the sources of vaitrace are available here and this warning is generated from hwInfo.py

@MaistoV :

  1. After we double check the code, this warning not effect vaitrace, because we not use hare ware info data in vaitrace version2.0.This hwinfo data just collect. But we still thank your for your commit, for vivado flow we should not use this method collect hwinfo data. In addition, do you run vaitrace success except this warnning?

BR Jilei

No, I get this error and no output files are produced:

Traceback (most recent call last): File "/usr/bin/vaitrace", line 186, in main(parseCmdLine()) File "/usr/bin/vaitrace", line 181, in main vaitraceWriter.write(options) File "/usr/bin/xlnx/vaitrace/vaitraceWriter.py", line 83, in write writer.vtf.convert.xat_to_vtf(rawData) File "/usr/bin/xlnx/vaitrace/writer/vtf/convert.py", line 331, in xat_to_vtf convert_dpu(xat.get('vart')) File "/usr/bin/xlnx/vaitrace/writer/vtf/convert.py", line 140, in convert_dpu dpu_profile_summary = dpu_parser.get_dpu_profile_summary() File "/usr/bin/xlnx/vaitrace/writer/parser/vart_parser.py", line 351, in get_dpu_profile_summary perf = workload batch / avg_t * 1000 # to GOP/ms ZeroDivisionError: float division by zero

The cause, as described above, is that no computing units are found, so the variable avg_t holds a zero value.

bvleeuwentue commented 1 year ago

Did anyone find a solution to this issue? I get a similar Error when using the Vivado flow for 2022.2 and VitisAI 3.0.

Traceback (most recent call last):
  File "/usr/bin/vaitrace", line 195, in <module>
    main(*parseCmdLine())
  File "/usr/bin/vaitrace", line 190, in main
    vaitraceWriter.write(options)
  File "/usr/bin/xlnx/vaitrace/vaitraceWriter.py", line 94, in write
    writer.vtf.convert.xat_to_vtf(rawData, options)
  File "/usr/bin/xlnx/vaitrace/writer/vtf/convert.py", line 327, in xat_to_vtf
    convert_dpu(xat.get('vart'), xat.get('hwInfo'), options)
  File "/usr/bin/xlnx/vaitrace/writer/vtf/convert.py", line 158, in convert_dpu
    s, e = toVTF(event)
  File "/usr/bin/xlnx/vaitrace/writer/vtf/convert.py", line 95, in toVTF
    efficiency = round(workload / time_dur, 2)
ZeroDivisionError: float division by zero

I followed the steps described here but I did not get it to work.

The DPU version: image

Thanks in advance

jileilamd commented 1 year ago

@bvleeuwentue This error will be encounter in vivado flow, we will fix it in version-3.01.

Shreyas-NR commented 8 months ago

Hi @qianglin-xlnx , @jileilamd @bvleeuwentue Is this issue fixed? When can I expect the version 3.01 to be released? What steps I need to do to get this upgrade? Could you please help me with the above?

-Shreyas

jileilamd commented 8 months ago

@Shreyas-NR you could use lasted code to fix this error, the fix has been included in lasted version

Shreyas-NR commented 8 months ago

Hi @jileilamd , @qianglin-xlnx , @Edocit , @bvleeuwentue Thanks for the quick response, I am using the latest version. Could you please direct me to the latest code so I can make the necessary changes?

Shreyas-NR commented 8 months ago

Hi @jileilamd , I'm doing Vivado flow, petalinux 2022.2 Vitis AI 3.0

image

image

Could you please help me which code to change and its location?

Looking forward to hearing from you.

jileilamd commented 8 months ago

hi @Shreyas-NR : you could use https://github.com/Xilinx/Vitis-AI/blob/master/src/vai_runtime/vart/trace/src/time.cpp this file replace your same file, and build again. please have a try , if any problem please feel free to let me know

Shreyas-NR commented 8 months ago

Hi @jileilamd , Thank you so much for your replies. It is helping me a lot.

I updated the Vitis AI on my board with the latest Vitis AI 3.5. For some reason vaitrace didn't generate the .run_summary file

root@xilinx-kv260-starterkit-20222:~/app/samples/python# python3 -m vaitrace_py resnet50_mt_custom.py 2
INFO:root:VART will run xmodel in [NORMAL] mode
Analyzing symbol tables...
61 / 61
81 / 81
6 / 6
45 / 45
3 / 3
INFO:root:vaitrace compile python code: resnet50_mt_custom.py
INFO:root:vaitrace exec poython code: resnet50_mt_custom.py
Thread: 281473665454416 , Top[0] 280 0.144820 "grey fox, gray fox, Urocyon cinereoargenteus,",
Thread: 281473657000272 , Top[0] 945 0.779940 "bell pepper,",
Thread: 281473665454416 , Top[1] 279 0.112786 "Arctic fox, white fox, Alopex lagopus,",
Thread: 281473657000272 , Top[1] 941 0.286924 "acorn squash,",
Thread: 281473665454416 , Top[2] 269 0.064263 "timber wolf, grey wolf, gray wolf, Canis lupus,",
Thread: 281473657000272 , Top[2] 943 0.223457 "cucumber, cuke,",
Thread: 281473665454416 , Top[3] 278 0.056712 "kit fox, Vulpes macrotis,",
Thread: 281473657000272 , Top[3] 952 0.119608 "fig,",
Thread: 281473665454416 , Top[4] 272 0.053276 "coyote, prairie wolf, brush wolf, Canis latrans,",
Thread: 281473657000272 , Top[4] 939 0.087507 "zucchini, courgette,",
FPS=8.89, total frames = 1.00 , time=0.112459 seconds
APM Stop Collecting
WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'
INFO:root:Generating VTF
INFO:root:Processing xmodel information
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_AvgPool2d_avgpool__8077_fix|8202529256]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_QuantStub_quant_stub__input_1_upload_0|0]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Conv2d_conv1__input_2|238637056]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv1__input_5|25890816]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv2__input_8|231411712]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv3__input_11|103563264]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Add_skip_add__input_13|105168896]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Conv2d_conv1__input_15|102961152]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Conv2d_conv2__input_18|231411712]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Add_skip_add__input_22|105168896]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Conv2d_conv1__input_24|102961152]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Conv2d_conv2__input_27|231411712]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Add_skip_add__input_31|105168896]
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_0|0]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv1__input_33|205922304]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv2__input_36|231311360]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv3__input_39|103161856]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Add_skip_add__input_41|206725120]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Conv2d_conv1__input_43|102860800]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Conv2d_conv2__input_46|231311360]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Add_skip_add__input_50|103964672]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Conv2d_conv1__input_52|102860800]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Conv2d_conv2__input_55|231311360]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Add_skip_add__input_59|103964672]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Conv2d_conv1__input_61|102860800]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Conv2d_conv2__input_64|231311360]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Add_skip_add__input_68|103964672]
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_1|0]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv1__input_70|205721600]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv2__input_73|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv3__input_76|102961152]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Add_skip_add__input_78|206123008]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Conv2d_conv1__input_80|102810624]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Conv2d_conv2__input_83|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Add_skip_add__input_87|103362560]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Conv2d_conv1__input_89|102810624]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Conv2d_conv2__input_92|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Add_skip_add__input_96|103362560]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Conv2d_conv1__input_98|102810624]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Conv2d_conv2__input_101|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Add_skip_add__input_105|103362560]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Conv2d_conv1__input_107|102810624]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Conv2d_conv2__input_110|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Add_skip_add__input_114|103362560]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Conv2d_conv1__input_116|102810624]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Conv2d_conv2__input_119|231261184]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Add_skip_add__input_123|103362560]
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_2|0]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv1__input_125|205621248]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv2__input_128|231236096]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv3__input_131|102860800]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Add_skip_add__input_133|205821952]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Conv2d_conv1__input_135|102785536]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Conv2d_conv2__input_138|231236096]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Add_skip_add__input_142|103061504]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Conv2d_conv1__input_144|102785536]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Conv2d_conv2__input_147|231236096]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Add_skip_add__input_151|103061504]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_AvgPool2d_avgpool__8077_fix|200704]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)|4097000]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)_inserted_fix_1_reshaped|0]
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)_inserted_fix_1_reshaped_download_0|0]
INFO:root:[vart_trace.csv]: Events number: 4
root@xilinx-kv260-starterkit-20222:~/resnet50_pt# dnf info vart
OE Remote Repo: sswreleases rel-v2022 generic rpm all                                                                   1.2 kB/s | 257  B     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm any                                                                   2.3 kB/s | 257  B     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm noarch                                                                945 kB/s | 153 kB     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm aarch64                                                               2.9 kB/s | 257  B     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm armv8a                                                                2.6 kB/s | 257  B     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm armv8a_crc                                                            1.0 kB/s | 257  B     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm cortexa72_cortexa53                                                   2.2 MB/s | 5.4 MB     00:02
OE Remote Repo: sswreleases rel-v2022 generic rpm zynqmp                                                                113 kB/s |  31 kB     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm zynqmp_ev                                                             1.5 MB/s | 697 kB     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm zynqmp_generic                                                        1.3 MB/s | 459 kB     00:00
OE Remote Repo: sswreleases rel-v2022 generic rpm xilinx_k26_kv                                                         1.8 MB/s | 431 kB     00:00
Installed Packages
Name         : vart
Version      : 3.5
Release      : r0.0
Architecture : cortexa72_cortexa53
Size         : 2.8 M
Source       : vart-3.5-r0.0.src.rpm
Repository   : @System
From repo    : oe-repo
Summary      : Vitis AI RunTime
License      : Apache-2.0
Description  : Runner is an application level runtime interface for DPU IPs based on XRT.
             : It use XIR subgraph as input, and run it on different targets. There are
             : also emulators implemented with same interface to make debuging eaiser.

root@xilinx-kv260-starterkit-20222:~/app/samples/python# vaitrace -v
vaitrace ver:
3.5.0_b7953a2a9_2024-01-10-05:17:16
Project:
vart
Internal Git Ver:
[671d17e, -]
root@xilinx-kv260-starterkit-20222:~/app/samples/python#

root@xilinx-kv260-starterkit-20222:~/app/samples/python# ls -l
total 188
-rw-r--r-- 1 root root    724 Jan  9 22:06 profile_summary.csv
-rw-r--r-- 1 1122 1122  12110 Jan  9 01:04 resnet50_mt_custom.py
-rw-r--r-- 1 root root    585 Jan  9 22:06 vart_trace.csv
-rw-r--r-- 1 root root 142273 Jan  9 22:06 vitis_ai_profile.csv
-rw-r--r-- 1 1122 1122  25674 Jan  8 21:39 words.txt
root@xilinx-kv260-starterkit-20222:~/app/samples/python#

Can you help me with this issue?

jileilamd commented 8 months ago

hi @Shreyas-NR : you could add -d your command and this will output vart.trace. please send this file to me , I'll debug it.

Shreyas-NR commented 8 months ago

Hi @jileilamd, Thanks for getting back

Here is the log of the execution. 

root@xilinx-kv260-starterkit-20222:~/app/samples/python# python3 -m vaitrace_py -d  resnet50_mt_custom.py 2

DEBUG:root:Command line args:
DEBUG:root:Namespace(cmd=['resnet50_mt_custom.py', '2'], config='', debug=True, traceSaveTo=None, timeout=None, showversion=False, bypass=False, python=True, va=False, xat=False, txt=False, json=False)
DEBUG:root:Use boot as trace clock
INFO:root:VART will run xmodel in [NORMAL] mode
DEBUG:root:Control Options:
DEBUG:root:{'cmd': ['resnet50_mt_custom.py', '2'], 'xat': {'compress': True, 'filename': '/home/root/app/samples/python/resnet50_mt_cus.xat'}, 'config': None, 'timeout': 60, 'debug': True, 'traceClock': 'boot', 'x86_tsc_khz': 0, 'platform': {'idcode': '0x4724093', 'fmlyid': '0x23', 'sub_fmlyid': '0x4', 'devid': '0x24', 'model': 'xlnx,zocl', 'os': 'Linux', 'release': ['5', '15'], 'machine': 'aarch64'}, 'runmode': 'normal'}
DEBUG:root:Preparing [hwInfo]...
DEBUG:root:Compatible [hwInfo]: True
DEBUG:root:Preparing [function]...
DEBUG:root:Compatible [function]: True
Analyzing symbol tables...
61 / 61
81 / 81
6 / 6
45 / 45
3 / 3
DEBUG:root:Preparing [sched]...
DEBUG:root:Preparing [xapm]...
DEBUG:root:Compatible [xapm]: True
DEBUG:root:Preparing [nmu]...
DEBUG:root:Compatible [nmu]: True
DEBUG:root:Preparing [cmd]...
DEBUG:root:Compatible [cmd]: True
DEBUG:root:Preparing [vart]...
DEBUG:root:Compatible [vart]: True
DEBUG:root:Preparing [pyfunc]...
DEBUG:root:Compatible [pyfunc]: True
DEBUG:root:Preparing [power]...
DEBUG:root:boot
DEBUG:root:### uprobe p:_fun_3296584_entry /usr/lib/libvitis_ai_library-facedetectrecog.so:0xe2d0
DEBUG:root:### uprobe p:_fun_7663178_entry /usr/lib/libvitis_ai_library-facedetectrecog.so:0xe7c0
DEBUG:root:### uprobe p:_fun_5011292_entry /usr/lib/libvitis_ai_library-mnistclassification.so:0x3450
DEBUG:root:### uprobe p:_fun_471863_entry /usr/lib/libvitis_ai_library-textmountain.so:0x3a50
DEBUG:root:### uprobe p:_fun_6052259_entry /usr/lib/libvitis_ai_library-textmountain.so:0x3ff0
DEBUG:root:### uprobe p:_fun_2015311_entry /usr/lib/libvitis_ai_library-fusion_cnn.so:0xd7f0
DEBUG:root:### uprobe p:_fun_806461_entry /usr/lib/libvitis_ai_library-facelandmark.so:0x3460
DEBUG:root:### uprobe p:_fun_3563505_entry /usr/lib/libvitis_ai_library-facelandmark.so:0x2e30
DEBUG:root:### uprobe p:_fun_6711098_entry /usr/lib/libvitis_ai_library-pointpillars.so:0x9b950
DEBUG:root:### uprobe p:_fun_6989732_entry /usr/lib/libvitis_ai_library-pointpillars.so:0x98fb0
DEBUG:root:### uprobe p:_fun_6989735_entry /usr/lib/libvitis_ai_library-pointpillars.so:0x9bb60
DEBUG:root:### uprobe p:_fun_6989734_entry /usr/lib/libvitis_ai_library-pointpillars.so:0x98d30
DEBUG:root:### uprobe p:_fun_1778374_entry /usr/lib/libvitis_ai_library-openpose.so:0x2bc0
DEBUG:root:### uprobe p:_fun_7506392_entry /usr/lib/libvitis_ai_library-openpose.so:0x3220
DEBUG:root:### uprobe p:_fun_8144855_entry /usr/lib/libvitis_ai_library-clocs.so:0x1a230
DEBUG:root:### uprobe p:_fun_9328147_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x8770
DEBUG:root:### uprobe p:_fun_2531021_entry /usr/lib/libvitis_ai_library-facequality5pt.so:0x2c10
DEBUG:root:### uprobe p:_fun_7046079_entry /usr/lib/libvitis_ai_library-facequality5pt.so:0x34e0
DEBUG:root:### uprobe p:_fun_2906651_entry /usr/lib/libvitis_ai_library-dpu_task.so:0x1a7c0
DEBUG:root:### uprobe p:_fun_2105605_entry /usr/lib/libvitis_ai_library-medicaldetection.so:0x2820
DEBUG:root:### uprobe p:_fun_7223479_entry /usr/lib/libvitis_ai_library-medicaldetection.so:0x2d40
DEBUG:root:### uprobe p:_fun_4198415_entry /usr/lib/libvitis_ai_library-platerecog.so:0x4d60
DEBUG:root:### uprobe p:_fun_6892153_entry /usr/lib/libvitis_ai_library-platerecog.so:0x46b0
DEBUG:root:### uprobe p:_fun_652414_entry /usr/lib/libvitis_ai_library-bcc.so:0x34f0
DEBUG:root:### uprobe p:_fun_1382232_entry /usr/lib/libvitis_ai_library-bcc.so:0x3980
DEBUG:root:### uprobe p:_fun_2600446_entry /usr/lib/libvitis_ai_library-ocr.so:0x5340
DEBUG:root:### uprobe p:_fun_5979864_entry /usr/lib/libvitis_ai_library-ocr.so:0x5800
DEBUG:root:### uprobe p:_fun_59902_entry /usr/lib/libvitis_ai_library-pmg.so:0x3950
DEBUG:root:### uprobe p:_fun_9868632_entry /usr/lib/libvitis_ai_library-pmg.so:0x33f0
DEBUG:root:### uprobe p:_fun_4845822_entry /usr/lib/libvitis_ai_library-ssd.so:0x3150
DEBUG:root:### uprobe p:_fun_227544_entry /usr/lib/libvitis_ai_library-ssd.so:0x3ab0
DEBUG:root:### uprobe p:_fun_227547_entry /usr/lib/libvitis_ai_library-ssd.so:0x34b0
DEBUG:root:### uprobe p:_fun_260502_entry /usr/lib/libvitis_ai_library-3Dsegmentation.so:0x6a30
DEBUG:root:### uprobe p:_fun_6662456_entry /usr/lib/libvitis_ai_library-3Dsegmentation.so:0x6580
DEBUG:root:### uprobe p:_fun_3363993_entry /usr/lib/libvitis_ai_library-arflow.so:0x6430
DEBUG:root:### uprobe p:_fun_6582195_entry /usr/lib/libvitis_ai_library-arflow.so:0x3e50
DEBUG:root:### uprobe p:_fun_3787441_entry /usr/lib/libvitis_ai_library-platenum.so:0x61c0
DEBUG:root:### uprobe p:_fun_2166835_entry /usr/lib/libvitis_ai_library-platenum.so:0x5360
DEBUG:root:### uprobe p:_fun_3305263_entry /usr/lib/libvitis_ai_library-bcc.so:0x2e90
DEBUG:root:### uprobe p:_fun_2858850_entry /usr/lib/libvitis_ai_library-clocs.so:0x13200
DEBUG:root:### uprobe p:_fun_2514660_entry /usr/lib/libvitis_ai_library-clocs.so:0x13260
DEBUG:root:### uprobe p:_fun_2514663_entry /usr/lib/libvitis_ai_library-clocs.so:0x10630
DEBUG:root:### uprobe p:_fun_2514662_entry /usr/lib/libvitis_ai_library-clocs.so:0x139e0
DEBUG:root:### uprobe p:_fun_7960046_entry /usr/lib/libvitis_ai_library-c2d2_lite.so:0x3380
DEBUG:root:### uprobe p:_fun_3124760_entry /usr/lib/libvitis_ai_library-c2d2_lite.so:0x2f60
DEBUG:root:### uprobe p:_fun_7076559_entry /usr/lib/libvitis_ai_library-pointpillars.so:0xb9350
DEBUG:root:### uprobe p:_fun_7543705_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x8130
DEBUG:root:### uprobe p:_fun_2702816_entry /usr/lib/libvitis_ai_library-ultrafast.so:0x2da0
DEBUG:root:### uprobe p:_fun_4985266_entry /usr/lib/libvitis_ai_library-ultrafast.so:0x3460
DEBUG:root:### uprobe p:_fun_473145_entry /usr/lib/libvitis_ai_library-clocs.so:0xd2d0
DEBUG:root:### uprobe p:_fun_8917417_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x9340
DEBUG:root:### uprobe p:_fun_681901_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x84e0
DEBUG:root:### uprobe p:_fun_681902_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x85a0
DEBUG:root:### uprobe p:_fun_681903_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x8540
DEBUG:root:### uprobe p:_fun_9004837_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x6ae0
DEBUG:root:### uprobe p:_fun_6514919_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x7a10
DEBUG:root:### uprobe p:_fun_6514916_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x72b0
DEBUG:root:### uprobe p:_fun_9107182_entry /usr/lib/libvitis_ai_library-hourglass.so:0x3290
DEBUG:root:### uprobe p:_fun_5710104_entry /usr/lib/libvitis_ai_library-hourglass.so:0x2c60
DEBUG:root:### uprobe p:_fun_4997196_entry /usr/lib/libvitis_ai_library-lanedetect.so:0x2ec0
DEBUG:root:### uprobe p:_fun_265190_entry /usr/lib/libvitis_ai_library-lanedetect.so:0x36f0
DEBUG:root:### uprobe p:_fun_3874151_entry /usr/lib/libvitis_ai_library-efficientdet_d2.so:0x4070
DEBUG:root:### uprobe p:_fun_2220497_entry /usr/lib/libvitis_ai_library-efficientdet_d2.so:0x4710
DEBUG:root:### uprobe p:_fun_3712175_entry /usr/lib/libvitis_ai_library-ocr.so:0x4c30
DEBUG:root:### uprobe p:_fun_8275777_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x93a0
DEBUG:root:### uprobe p:_fun_244707_entry /usr/lib/libvitis_ai_library-pointpillars_nuscenes.so:0x63b0
DEBUG:root:### uprobe p:_fun_7041471_entry /usr/lib/libvitis_ai_library-facerecog.so:0x6e00
DEBUG:root:### uprobe p:_fun_8638897_entry /usr/lib/libvitis_ai_library-facerecog.so:0xa560
DEBUG:root:### uprobe p:_fun_7534546_entry /usr/lib/libvitis_ai_library-clocs.so:0x1a3e0
DEBUG:root:### uprobe p:_fun_1504257_entry /usr/lib/libvitis_ai_library-mnistclassification.so:0x3320
DEBUG:root:### uprobe p:_fun_6711922_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x7970
DEBUG:root:### uprobe p:_fun_9616692_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x9aa0
DEBUG:root:### uprobe p:_fun_2792817_entry /usr/lib/libvitis_ai_library-fusion_cnn.so:0xacb0
DEBUG:root:### uprobe p:_fun_579400_entry /usr/lib/libvitis_ai_library-mnistclassification.so:0x3a20
DEBUG:root:### uprobe p:_fun_1577634_entry /usr/lib/libvitis_ai_library-mnistclassification.so:0x3500
DEBUG:root:### uprobe p:_fun_8100825_entry /usr/lib/libvitis_ai_library-RGBDsegmentation.so:0x3730
DEBUG:root:### uprobe p:_fun_1441238_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x7910
DEBUG:root:### uprobe p:_fun_7523441_entry /usr/lib/libvitis_ai_library-medicalsegcell.so:0x3d20
DEBUG:root:### uprobe p:_fun_5964187_entry /usr/lib/libvitis_ai_library-medicalsegcell.so:0x37b0
DEBUG:root:### uprobe p:_fun_5086420_entry /usr/lib/libvitis_ai_library-ofa_yolo.so:0x2ee0
DEBUG:root:### uprobe p:_fun_5483318_entry /usr/lib/libvitis_ai_library-ofa_yolo.so:0x2c90
DEBUG:root:### uprobe p:_fun_7777668_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x67b0
DEBUG:root:### uprobe p:_fun_8798638_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x6820
DEBUG:root:### uprobe p:_fun_2358504_entry /usr/lib/libvitis_ai_library-clocs.so:0x16ec0
DEBUG:root:### uprobe p:_fun_7161634_entry /usr/lib/libvitis_ai_library-bcc.so:0x2d20
DEBUG:root:### uprobe p:_fun_5902212_entry /usr/lib/libvitis_ai_library-bcc.so:0x3340
DEBUG:root:### uprobe p:_fun_3487987_entry /usr/lib/libvitis_ai_library-dpu_task.so:0xa880
DEBUG:root:### uprobe p:_fun_6592447_entry /usr/lib/libvitis_ai_library-refinedet.so:0x3500
DEBUG:root:### uprobe p:_fun_4041393_entry /usr/lib/libvitis_ai_library-refinedet.so:0x3a10
DEBUG:root:### uprobe p:_fun_4041394_entry /usr/lib/libvitis_ai_library-refinedet.so:0x30b0
DEBUG:root:### uprobe p:_fun_9522122_entry /usr/lib/libvitis_ai_library-pointpillars.so:0xb3750
DEBUG:root:### uprobe p:_fun_4612804_entry /usr/lib/libvitis_ai_library-pointpillars.so:0xafe30
DEBUG:root:### uprobe p:_fun_2110197_entry /usr/lib/libvitis_ai_library-clocs.so:0xea00
DEBUG:root:### uprobe p:_fun_8674270_entry /usr/lib/libvitis_ai_library-efficientdet_d2.so:0x37a0
DEBUG:root:### uprobe p:_fun_1988770_entry /usr/lib/libvitis_ai_library-pmg.so:0x31a0
DEBUG:root:### uprobe p:_fun_2545028_entry /usr/lib/libvitis_ai_library-pmg.so:0x3240
DEBUG:root:### uprobe p:_fun_816328_entry /usr/lib/libvitis_ai_library-vehicleclassification.so:0x3440
DEBUG:root:### uprobe p:_fun_5430382_entry /usr/lib/libvitis_ai_library-vehicleclassification.so:0x2c10
DEBUG:root:### uprobe p:_fun_6412075_entry /usr/lib/libvitis_ai_library-facefeature.so:0x3970
DEBUG:root:### uprobe p:_fun_7616077_entry /usr/lib/libvitis_ai_library-facefeature.so:0x4610
DEBUG:root:### uprobe p:_fun_89790_entry /usr/lib/libvitis_ai_library-clocs.so:0x1d2a0
DEBUG:root:### uprobe p:_fun_9068728_entry /usr/lib/libvitis_ai_library-clocs.so:0x1d240
DEBUG:root:### uprobe p:_fun_8003218_entry /usr/lib/libvitis_ai_library-yolov2.so:0x39d0
DEBUG:root:### uprobe p:_fun_8392412_entry /usr/lib/libvitis_ai_library-yolov2.so:0x43a0
DEBUG:root:### uprobe p:_fun_918816_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x3220
DEBUG:root:### uprobe p:_fun_5141434_entry /usr/lib/libvitis_ai_library-centerpoint.so:0x43d0
DEBUG:root:### uprobe p:_fun_469517_entry /usr/lib/libvitis_ai_library-medicalsegmentation.so:0x2e60
DEBUG:root:### uprobe p:_fun_2036623_entry /usr/lib/libvitis_ai_library-medicalsegmentation.so:0x34f0
DEBUG:root:### uprobe p:_fun_668756_entry /usr/lib/libvitis_ai_library-yolovx.so:0x3af0
DEBUG:root:### uprobe p:_fun_4176686_entry /usr/lib/libvitis_ai_library-yolovx.so:0x31e0
DEBUG:root:### uprobe p:_fun_7391309_entry /usr/lib/libvitis_ai_library-pointpainting.so:0x6c60
DEBUG:root:### uprobe p:_fun_2216858_entry /usr/lib/libvitis_ai_library-rcan.so:0x35d0
DEBUG:root:### uprobe p:_fun_8418516_entry /usr/lib/libvitis_ai_library-rcan.so:0x2b40
DEBUG:root:### uprobe p:_fun_8418519_entry /usr/lib/libvitis_ai_library-rcan.so:0x3180
DEBUG:root:### uprobe p:_fun_7932442_entry /usr/lib/libvitis_ai_library-reid.so:0x2fb0
DEBUG:root:### uprobe p:_fun_4558548_entry /usr/lib/libvitis_ai_library-reid.so:0x3400
DEBUG:root:### uprobe p:_fun_4558551_entry /usr/lib/libvitis_ai_library-reid.so:0x2a00
DEBUG:root:### uprobe p:_fun_3009690_entry /usr/lib/libvitis_ai_library-solo.so:0x4750
DEBUG:root:### uprobe p:_fun_4959956_entry /usr/lib/libvitis_ai_library-solo.so:0x3f50
DEBUG:root:### uprobe p:_fun_1646018_entry /usr/lib/libvitis_ai_library-multitaskv3.so:0x3110
DEBUG:root:### uprobe p:_fun_4775628_entry /usr/lib/libvitis_ai_library-multitaskv3.so:0x3900
DEBUG:root:### uprobe p:_fun_4420887_entry /usr/lib/libvitis_ai_library-carplaterecog.so:0x5420
DEBUG:root:### uprobe p:_fun_673235_entry /usr/lib/libvitis_ai_library-carplaterecog.so:0x4a30
DEBUG:root:### uprobe p:_fun_605818_entry /usr/lib/libvitis_ai_library-platedetect.so:0x34c0
DEBUG:root:### uprobe p:_fun_6298948_entry /usr/lib/libvitis_ai_library-platedetect.so:0x2e40
DEBUG:root:### uprobe p:_fun_5595006_entry /usr/lib/libvitis_ai_library-covid19segmentation.so:0x3a10
DEBUG:root:### uprobe p:_fun_7539129_entry /usr/lib/libvitis_ai_library-covid19segmentation.so:0x3a80
DEBUG:root:### uprobe p:_fun_6720296_entry /usr/lib/libvitis_ai_library-covid19segmentation.so:0x3c40
DEBUG:root:### uprobe p:_fun_7236537_entry /usr/lib/libvitis_ai_library-covid19segmentation.so:0x3cb0
DEBUG:root:### uprobe p:_fun_2206823_entry /usr/lib/libvitis_ai_library-polypsegmentation.so:0x3a20
DEBUG:root:### uprobe p:_fun_7872809_entry /usr/lib/libvitis_ai_library-polypsegmentation.so:0x38c0
DEBUG:root:### uprobe p:_fun_6904480_entry /usr/lib/libvitis_ai_library-facedetect.so:0x4050
DEBUG:root:### uprobe p:_fun_2918546_entry /usr/lib/libvitis_ai_library-facedetect.so:0x3b40
DEBUG:root:### uprobe p:_fun_2918545_entry /usr/lib/libvitis_ai_library-facedetect.so:0x33e0
DEBUG:root:### uprobe p:_fun_8871128_entry /usr/lib/libvitis_ai_library-fusion_cnn.so:0xee80
DEBUG:root:### uprobe p:_fun_2666906_entry /usr/lib/libvitis_ai_library-fusion_cnn.so:0xf4b0
DEBUG:root:### uprobe p:_fun_9195190_entry /usr/lib/libvitis_ai_library-posedetect.so:0x32f0
DEBUG:root:### uprobe p:_fun_4754616_entry /usr/lib/libvitis_ai_library-posedetect.so:0x2a90
DEBUG:root:### uprobe p:_fun_6710857_entry /usr/lib/libvitis_ai_library-medicalsegcell.so:0x2bd0
DEBUG:root:### uprobe p:_fun_5073091_entry /usr/lib/libvitis_ai_library-medicalsegcell.so:0x43b0
DEBUG:root:### uprobe p:_fun_5769698_entry /usr/lib/libvitis_ai_library-tfssd.so:0x35e0
DEBUG:root:### uprobe p:_fun_229476_entry /usr/lib/libvitis_ai_library-tfssd.so:0x3c70
DEBUG:root:### uprobe p:_fun_2477074_entry /usr/lib/libvitis_ai_library-retinaface.so:0x3740
DEBUG:root:### uprobe p:_fun_5994108_entry /usr/lib/libvitis_ai_library-retinaface.so:0x3dd0
DEBUG:root:### uprobe p:_fun_5994111_entry /usr/lib/libvitis_ai_library-retinaface.so:0x33e0
DEBUG:root:### uprobe p:_fun_9411509_entry /usr/lib/libvart-runner.so:0x1e950
DEBUG:root:### uprobe p:_fun_3991223_entry /usr/lib/libvart-runner.so:0x1e6d0
DEBUG:root:### uprobe p:_fun_4183140_entry /usr/lib/libvart-runner.so:0x131f0
DEBUG:root:### uprobe p:_fun_6789988_entry /usr/lib/libvart-runner.so:0x1ede0
DEBUG:root:### uprobe p:_fun_4549999_entry /usr/lib/libvart-runner.so:0x10dd0
DEBUG:root:### uprobe p:_fun_2814658_entry /usr/lib/libvart-runner.so:0x11140
DEBUG:root:### disable tracing /sys/kernel/debug/tracing/instances/function
DEBUG:root:### ftrace start
DEBUG:root:### enable tracing /sys/kernel/debug/tracing/instances/function
INFO:root:vaitrace compile python code: resnet50_mt_custom.py
INFO:root:vaitrace exec poython code: resnet50_mt_custom.py
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0110 01:30:42.075338  2665 trace.cpp:132] [vaitrace]initialize...
I0110 01:30:42.075484  2665 trace.cpp:53] [vaitrace][options]buf_size_mb:2
I0110 01:30:42.075523  2665 trace.cpp:53] [vaitrace][options]enable:1
I0110 01:30:42.075557  2665 trace.cpp:53] [vaitrace][options]logger_file_path:/tmp/tmpfst99ccv/vaitrace_2665
I0110 01:30:42.075592  2665 trace.cpp:53] [vaitrace][options]pid:2665
I0110 01:30:42.075626  2665 trace.cpp:53] [vaitrace][options]trace_log_dir:/tmp/tmpfst99ccv/
Thread: 281473634541904 , Top[0] 945 0.779940 "bell pepper,",
Thread: 281473642996048 , Top[0] 280 0.144820 "grey fox, gray fox, Urocyon cinereoargenteus,",
Thread: 281473634541904 , Top[1] 941 0.286924 "acorn squash,",
Thread: 281473642996048 , Top[1] 279 0.112786 "Arctic fox, white fox, Alopex lagopus,",
Thread: 281473634541904 , Top[2] 943 0.223457 "cucumber, cuke,",
Thread: 281473642996048 , Top[2] 269 0.064263 "timber wolf, grey wolf, gray wolf, Canis lupus,",
Thread: 281473634541904 , Top[3] 952 0.119608 "fig,",
Thread: 281473642996048 , Top[3] 278 0.056712 "kit fox, Vulpes macrotis,",
Thread: 281473634541904 , Top[4] 939 0.087507 "zucchini, courgette,",
Thread: 281473642996048 , Top[4] 272 0.053276 "coyote, prairie wolf, brush wolf, Canis latrans,",
FPS=8.82, total frames = 1.00 , time=0.113349 seconds
DEBUG:root:### ftrace stop
DEBUG:root:### disable tracing /sys/kernel/debug/tracing/instances/function
APM Stop Collecting
I0110 01:30:45.801537  2068 trace.cpp:170] [vaitrace]Disabling...
I0110 01:30:45.801630  2068 trace.cpp:181] [vaitrace]Dumping...
I0110 01:30:45.801982  2068 trace.cpp:210] [vaitrace]Dumping to:/tmp/tmpfst99ccv/vaitrace_2665
I0110 01:30:45.802300  2068 trace.cpp:220] [vaitrace]Stoping...
DEBUG:root:Trace file was saved to [/home/root/app/samples/python/function.trace]
DEBUG:root:Processing [hwInfo]...
DEBUG:root:Processing [function]...
DEBUG:root:Processing [xapm]...
DEBUG:root:Processing [cmd]...
DEBUG:root:Processing [vart]...
DEBUG:root:Processing [pyfunc]...
DEBUG:root:Getting Data [hwInfo]...
WARNING:root:Cannot open 'zyxclmm_drm/ip_layout'
DEBUG:root:[time sync]: hwInfo: 0.000000
DEBUG:root:Getting Data [function]...
DEBUG:root:[time sync]: function: 0.000000
DEBUG:root:Getting Data [xapm]...
DEBUG:root:[time sync]: xapm: 0.000000
DEBUG:root:Getting Data [cmd]...
DEBUG:root:[time sync]: cmd: 0.000000
DEBUG:root:Getting Data [vart]...
DEBUG:root:[time sync]: vart: 0.000000
DEBUG:root:Getting Data [pyfunc]...
DEBUG:root:[time sync]: pyfunc: 0.000000
INFO:root:.xat file Saved to [/home/root/app/samples/python/vtf_debug.xat]
INFO:root:Generating VTF
DEBUG:root:Open xmodel information file: /tmp/tmpfst99ccv/vaitrace_subgraph_info_2068_0
DEBUG:root:Subgraphs to process: 62
DEBUG:root:Open xmodel information file: /tmp/tmpfst99ccv/vaitrace_subgraph_info_2068_1
DEBUG:root:Subgraphs to process: 62
INFO:root:Processing xmodel information
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_AvgPool2d_avgpool__8077_fix|8202529256]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_QuantStub_quant_stub__input_1_upload_0|0]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Conv2d_conv1__input_2|238637056]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv1__input_5|25890816]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv2__input_8|231411712]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Conv2d_conv3__input_11|103563264]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_0__Add_skip_add__input_13|105168896]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Conv2d_conv1__input_15|102961152]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Conv2d_conv2__input_18|231411712]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_1__Add_skip_add__input_22|105168896]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Conv2d_conv1__input_24|102961152]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Conv2d_conv2__input_27|231411712]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer1__Bottleneck_2__Add_skip_add__input_31|105168896]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_0|0]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv1__input_33|205922304]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv2__input_36|231311360]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Conv2d_conv3__input_39|103161856]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_0__Add_skip_add__input_41|206725120]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Conv2d_conv1__input_43|102860800]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Conv2d_conv2__input_46|231311360]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_1__Add_skip_add__input_50|103964672]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Conv2d_conv1__input_52|102860800]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Conv2d_conv2__input_55|231311360]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_2__Add_skip_add__input_59|103964672]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Conv2d_conv1__input_61|102860800]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Conv2d_conv2__input_64|231311360]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer2__Bottleneck_3__Add_skip_add__input_68|103964672]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_1|0]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv1__input_70|205721600]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv2__input_73|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Conv2d_conv3__input_76|102961152]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_0__Add_skip_add__input_78|206123008]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Conv2d_conv1__input_80|102810624]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Conv2d_conv2__input_83|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_1__Add_skip_add__input_87|103362560]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Conv2d_conv1__input_89|102810624]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Conv2d_conv2__input_92|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_2__Add_skip_add__input_96|103362560]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Conv2d_conv1__input_98|102810624]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Conv2d_conv2__input_101|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_3__Add_skip_add__input_105|103362560]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Conv2d_conv1__input_107|102810624]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Conv2d_conv2__input_110|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_4__Add_skip_add__input_114|103362560]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Conv2d_conv1__input_116|102810624]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Conv2d_conv2__input_119|231261184]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer3__Bottleneck_5__Add_skip_add__input_123|103362560]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_fake_downsample_2|0]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv1__input_125|205621248]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv2__input_128|231236096]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Conv2d_conv3__input_131|102860800]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_0__Add_skip_add__input_133|205821952]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Conv2d_conv1__input_135|102785536]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Conv2d_conv2__input_138|231236096]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_1__Add_skip_add__input_142|103061504]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Conv2d_conv1__input_144|102785536]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Conv2d_conv2__input_147|231236096]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Sequential_layer4__Bottleneck_2__Add_skip_add__input_151|103061504]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_AvgPool2d_avgpool__8077_fix|200704]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)|4097000]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)_inserted_fix_1_reshaped|0]
DEBUG:root:Compatible [DPUCZDX8G_ISA1]: True
DEBUG:root:Processing DPU mc [DPUCZDX8G_ISA1]...
INFO:root:SubGraph index duplication: [subgraph_ResNet__ResNet_Linear_fc__8088(TransferMatMulToConv2d)_inserted_fix_1_reshaped_download_0|0]
INFO:root:[vart_trace.csv]: Events number: 4
root@xilinx-kv260-starterkit-20222:~/app/samples/python#

root@xilinx-kv260-starterkit-20222:~/app/samples/python# xdputil query
{
    "DPU IP Spec":{
        "DPU Core Count":2,
        "IP version":"v4.1.0",
        "enable softmax":"False"
    },
    "VAI Version":{
        "libvart-runner.so":"Xilinx vart-runner Version: 3.5.0-b7953a2a9f60e23efdfced5c186328dd1449665c  2024-01-10-05:17:16 ",
        "libvitis_ai_library-dpu_task.so":"Advanced Micro Devices vitis_ai_library dpu_task Version: 3.5.0-b7953a2a9f60e23efdfced5c186328dd1449665c  2023-06-29 03:20:28 [UTC] ",
        "libxir.so":"Xilinx xir Version: xir-b7953a2a9f60e23efdfced5c186328dd1449665c 2024-01-10-05:16:38",
        "target_factory":"target-factory.3.5.0 b7953a2a9f60e23efdfced5c186328dd1449665c"
    },
    "kernels":[
        {
            "DPU Arch":"DPUCZDX8G_ISA1_B1600",
            "DPU Frequency (MHz)":325,
            "XRT Frequency (MHz)":100,
            "cu_idx":0,
            "fingerprint":"0x101000056010404",
            "is_vivado_flow":true,
            "name":"DPU Core 0"
        },
        {
            "DPU Arch":"DPUCZDX8G_ISA1_B1600",
            "DPU Frequency (MHz)":325,
            "XRT Frequency (MHz)":100,
            "cu_idx":1,
            "fingerprint":"0x101000056010404",
            "is_vivado_flow":true,
            "name":"DPU Core 1"
        }
    ]
}
root@xilinx-kv260-starterkit-20222:~/app/samples/python#

Vivado flow, Petlainux 2022.2

To update the Vitis AI to 3.5 I copied the latest petalinux recipes from this link and build the petalinux and booted the image.

vart.zip

Shreyas-NR commented 8 months ago

Hi @jileilamd, Were you able to figure out anything from the trace logs? Please let me know if you want me to try out anything that can be helpful in your debug. I have attached the python file and the xmodel that I'm using. Looking forward to hearing from you.

Thanks, Shreyas python.zip

Shreyas-NR commented 7 months ago

Hi @jileilamd, @qianglin-xlnx Following up, were you able to figure out anything related to the problem? I have opened a thread on the community forum too. I have shared my progress in debugging there. Please have a look at it. https://support.xilinx.com/s/question/0D74U000008v2wSSAQ/detail?language=en_US&fromEma%5B%E2%80%A6%5D1ext=0&emkind=chatterCommentNotification&emtm=1705273335430

I'm eagerly awaiting your response.

Best regards, Shreyas N R

Shreyas-NR commented 7 months ago

Hello @jileilamd Do you have any updates? It makes sense for me to wait if you respond with at least yes or no. You are free to tag any other engineers who can resolve this issue.

I have opened another issue on this repo, you can find the link here I read a similar case on the xiilinx forum that mentions, vaitrace doesn't support vivado flow. I tried doing Vitis flow for KV260, but I'm stuck with that also, I couldn't get the DPU working using Vitis flow.

Looking forward to hearing from you. Best regards, Shreyas N R