search

nvdla / sw

NVDLA SW
Other
489 stars 193 forks source link

Debug info in nvdla compiler loadable #153

Open filipthoen2 opened 5 years ago

filipthoen2 commented 5 years ago

Hello,

To ease debugging, we're looking at correlating the graph node name in the original network prototext file with the tasks dispatched to the NVDLA.

I reviewed the Loadable file format description (https://github.com/nvdla/sw/blob/master/umd/core/common/loadable.fbs) and tried some small examples. It looks like the nvdla_compiler only includes "TensorDesc" for the input & output nodes of the graph when I dump this table in the Runtime::load() run-time call.

Is there any other way to include this type of "debug" info in the loadable, making correlating things easier?

Thanks,

Filip

prasshantg commented 5 years ago

@filipthoen2 thank you for the suggestion!

if I understand correctly, you want some correlation between tensors from Caffe model to Tensors in loadable?

filipthoen2 commented 5 years ago

yes, as this will help debugging CNN graphs running on the NVDLA. The nvdla_compiler will manipulate the graph, and e.g. add extra nodes for "bias" operations, add extra SDP operations to CONV to use the SDP WDMA & implement certain operations in the software emulator running in the UMD. This all makes it hard to debug if something goes wrong (e.g. compiler bug, scheduler bug, …). Currently, all operations are given an index, and only for the input & output nodes of the graph a TensorDesc is kept in addition.

Having this type of additional correlation info -- think of it as "debug info" like enabled for source code debugging of embedded C++ code through the DWARF format -- will help debug graphs mapped to NVDLA.

prasshantg commented 5 years ago

@filipthoen2 NVDLA compiler is open source now. Is it possible for you to share sample change for this feature request?

filipthoen2 commented 5 years ago

I'll try to find the time for this, but can't promise anything.