MNIST working with Docker image but not on FPGA

[fisherxue]

Issue: Running same network on MNIST works with the Docker image but not on the FPGA

What I'm doing:

Running /nvdla_runtime --loadable sw/regression/flatbufs/kmd/NN/N_L0_0_fbuf --image sw/regression/images/digits/eight.pgm

Running docker with: docker run -it -v /home:/home nvdla/vp:1.3

# ./nvdla_runtime --loadable sw/regression/flatbufs/kmd/NN/NN_L0_0_fbuf --image 0.983398 3.57628e-0# 
sw/regression/images/digits/seven.pgm --rawdump && cat output.dimg 
creating new runtime context...
Emulator starting
ppgminfo 1 28 28
pgm2dimg 1 28 28 1 32 896 896
submitting tasks...
Work Found!
Work Done
[  547.737359] Enter:dla_read_network_config
[  547.738031] Exit:dla_read_network_config status=0
[  547.738332] Enter: dla_initiate_processors
[  547.738830] Enter: dla_submit_operation
[  547.739101] Prepare Convolution operation index 0 ROI 0 dep_count 1
[  547.739463] Enter: dla_prepare_operation
[  547.739875] processor:Convolution group:0, rdma_group:0 available
[  547.742838] Enter: dla_read_config
[  547.749235] Exit: dla_read_config
[  547.749658] Exit: dla_prepare_operation status=0
[  547.749967] Enter: dla_program_operation
[  547.750241] Program Convolution operation index 0 ROI 0 Group[0]
[  547.759203] no desc get due to index==-1
[  547.761576] no desc get due to index==-1
[  547.761915] no desc get due to index==-1
[  547.762155] no desc get due to index==-1
[  547.762394] no desc get due to index==-1
[  547.762630] Enter: dla_op_programmed
[  547.762862] Update dependency operation index 3 ROI 0 DEP_COUNT=3
[  547.763185] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[  547.763505] enable SDP in dla_update_dependency as depdency are resolved
[  547.763897] Enter: dla_enable_operation
[  547.764134] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  547.764646] Exit: dla_enable_operation status=0
[  547.765027] Exit: dla_op_programmed
[  547.765312] Exit: dla_program_operation status=0
[  547.765670] Exit: dla_submit_operation
[  547.765977] Enter: dla_dequeue_operation
[  547.766290] Dequeue op from Convolution processor, index=3 ROI=0
[  547.766731] Enter: dla_submit_operation
[  547.767069] Prepare Convolution operation index 3 ROI 0 dep_count 2
[  547.767642] Enter: dla_prepare_operation
[  547.768312] processor:Convolution group:1, rdma_group:0 available
[  547.768732] Enter: dla_read_config
[  547.777724] Exit: dla_read_config
[  547.778090] Exit: dla_prepare_operation status=0
[  547.778617] Enter: dla_program_operation
[  547.778946] Program Convolution operation index 3 ROI 0 Group[1]
[  547.784636] no desc get due to index==-1
[  547.785408] no desc get due to index==-1
[  547.785666] no desc get due to index==-1
[  547.785916] no desc get due to index==-1
[  547.786163] no desc get due to index==-1
[  547.786415] Enter: dla_op_programmed
[  547.786649] Update dependency operation index 6 ROI 0 DEP_COUNT=3
[  547.786979] Update dependency operation index 4 ROI 0 DEP_COUNT=2
[  547.787324] Exit: dla_op_programmed
[  547.787552] Exit: dla_program_operation status=0
[  547.789823] Exit: dla_submit_operation
[  547.790127] Exit: dla_dequeue_operation
[  547.790376] Enter: dla_submit_operation
[  547.790617] Prepare SDP operation index 1 ROI 0 dep_count 0
[  547.790938] Enter: dla_prepare_operation
[  547.791334] processor:SDP group:1, rdma_group:1 available
[  547.791630] Enter: dla_read_config
[  547.799392] Exit: dla_read_config
[  547.799764] Exit: dla_prepare_operation status=0
[  547.802435] Enter: dla_program_operation
[  547.802962] Program SDP operation index 1 ROI 0 Group[1]
[  547.806264] no desc get due to index==-1
[  547.806675] no desc get due to index==-1
[  547.807331] no desc get due to index==-1
[  547.807680] no desc get due to index==-1
[  547.809967] Enter: dla_op_programmed
[  547.810316] Update dependency operation index 4 ROI 0 DEP_COUNT=1
[  547.810681] enable SDP in dla_update_dependency as depdency are resolved
[  547.811068] Enter: dla_enable_operation
[  547.811330] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  547.811827] Exit: dla_enable_operation status=0
[  547.814813] Exit: dla_op_programmed
[  547.815153] Exit: dla_program_operation status=0
[  547.815429] Enter: dla_enable_operation
[  547.815704] Enable SDP operation index 1 ROI 0
[  547.818279] Enter: dla_op_enabled
[  547.818594] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[  547.818971] enable Convolution in dla_update_dependency as depdency are resolved
[  547.819412] Enter: dla_enable_operation
[  547.819709] Enable Convolution operation index 0 ROI 0
[  547.825952] Enter: dla_op_enabled
[  547.826339] Exit: dla_op_enabled
[  547.826576] Exit: dla_enable_operation status=0
[  547.826851] Exit: dla_op_enabled
[  547.827078] Exit: dla_enable_operation status=0
[  547.827350] Exit: dla_submit_operation
[  547.827582] Enter: dla_dequeue_operation
[  547.827830] Dequeue op from SDP processor, index=4 ROI=0
[  547.828572] Enter: dla_submit_operation
[  547.828940] Prepare SDP operation index 4 ROI 0 dep_count 0
[  547.829321] Enter: dla_prepare_operation
[  547.839277] processor:SDP group:0, rdma_group:0 available
[  547.841662] Enter: dla_read_config
[  547.903325] Exit: dla_read_config
[  547.904247] Exit: dla_prepare_operation status=0
[  547.905175] Enter: dla_program_operation
[  547.905701] Program SDP operation index 4 ROI 0 Group[0]
[  547.924772] no desc get due to index==-1
[  547.926239] no desc get due to index==-1
[  586.026440] no desc get due to index==-1
[  586.026819] no desc get due to index==-1
[  586.027532] Enter: dla_op_programmed
[  586.027782] Update dependency operation index 7 ROI 0 DEP_COUNT=2
[  586.030359] Exit: dla_op_programmed
[  586.030694] Exit: dla_program_operation status=0
[  586.031008] Enter: dla_enable_operation
[  586.031288] Enable SDP operation index 4 ROI 0
[  586.034112] Enter: dla_op_enabled
[  586.034746] Update dependency operation index 3 ROI 0 DEP_COUNT=2
[  586.035106] Exit: dla_op_enabled
[  586.035338] Exit: dla_enable_operation status=0
[  586.035620] Exit: dla_submit_operation
[  586.038241] Exit: dla_dequeue_operation
[  586.038590] Enter: dla_submit_operation
[  586.038860] Prepare PDP operation index 2 ROI 0 dep_count 1
[  586.039182] Enter: dla_prepare_operation
[  586.043012] processor:PDP group:0, rdma_group:0 available
[  586.043749] Enter: dla_read_config
[  586.049580] Exit: dla_read_config
[  586.050276] Exit: dla_prepare_operation status=0
[  586.050953] Enter: dla_program_operation
[  586.051231] Program PDP operation index 2 ROI 0 Group[0]
[  586.051568] group id 0 rdma id 0
[  586.056883] no desc get due to index==-1
[  586.057640] no desc get due to index==-1
[  586.057988] no desc get due to index==-1
[  586.058592] no desc get due to index==-1
[  586.058996] no desc get due to index==-1
[  586.059360] Enter: dla_op_programmed
[  586.059723] Update dependency operation index 5 ROI 0 DEP_COUNT=2
[  586.062683] Exit: dla_op_programmed
[  586.063004] Exit: dla_program_operation status=0
[  586.063305] Exit: dla_submit_operation
[  586.063556] Enter: dla_dequeue_operation
[  586.063822] Dequeue op from PDP processor, index=5 ROI=0
[  586.066778] Enter: dla_submit_operation
[  586.067123] Prepare PDP operation index 5 ROI 0 dep_count 1
[  586.067469] Enter: dla_prepare_operation
[  586.070058] processor:PDP group:1, rdma_group:1 available
[  586.070611] Enter: dla_read_config
[  586.074202] Exit: dla_read_config
[  586.076528] Exit: dla_prepare_operation status=0
[  586.076840] Enter: dla_program_operation
[  586.077127] Program PDP operation index 5 ROI 0 Group[1]
[  586.077444] group id 1 rdma id 1
[  586.079016] no desc get due to index==-1
[  586.079390] no desc get due to index==-1
[  586.079660] no desc get due to index==-1
[  586.081544] no desc get due to index==-1
[  586.081763] no desc get due to index==-1
[  586.081963] no desc get due to index==-1
[  586.082278] Enter: dla_op_programmed
[  586.082594] Exit: dla_op_programmed
[  586.082855] Exit: dla_program_operation status=0
[  586.083160] Exit: dla_submit_operation
[  586.083435] Exit: dla_dequeue_operation
[  586.083708] Exit: dla_initiate_processors status=0
[  586.086916] Enter:dla_handle_events, processor:BDMA
[  586.087308] Exit:dla_handle_events, ret:0
[  586.087587] Enter:dla_handle_events, processor:Convolution
[  586.088178] Handle cdma weight done event, processor Convolution group 0
[  586.088744] Handle cdma data done event, processor Convolution group 0
[  586.089392] Handle op complete event, processor Convolution group 0
[  586.089887] Enter:dla_op_completion processor Convolution group0
[  586.090325] Completed Convolution operation index 0 ROI 0
[  586.090676] 1 HWLs done, totally 11 layers
[  586.090959] Enter: dla_dequeue_operation
[  586.091231] Dequeue op from Convolution processor, index=6 ROI=0
[  586.091628] Enter: dla_submit_operation
[  586.092109] Prepare Convolution operation index 6 ROI 0 dep_count 2
[  586.092632] Enter: dla_prepare_operation
[  586.093023] processor:Convolution group:0, rdma_group:0 available
[  586.093381] Enter: dla_read_config
[  586.108237] Exit: dla_read_config
[  586.108751] Exit: dla_prepare_operation status=0
[  586.109172] Enter: dla_program_operation
[  586.109570] Program Convolution operation index 6 ROI 0 Group[0]
[  586.112052] no desc get due to index==-1
[  586.112607] no desc get due to index==-1
[  586.112881] no desc get due to index==-1
[  586.113145] no desc get due to index==-1
[  586.113416] no desc get due to index==-1
[  586.113682] Enter: dla_op_programmed
[  586.113946] Update dependency operation index 9 ROI 0 DEP_COUNT=3
[  586.114310] Update dependency operation index 7 ROI 0 DEP_COUNT=1
[  586.114667] enable SDP in dla_update_dependency as depdency are resolved
[  586.115052] Enter: dla_enable_operation
[  586.115318] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  586.116723] Exit: dla_enable_operation status=0
[  586.117108] Exit: dla_op_programmed
[  586.117431] Exit: dla_program_operation status=0
[  586.117728] Exit: dla_submit_operation
[  586.118005] Exit: dla_dequeue_operation
[  586.118244] Exit:dla_op_completion processor Convolution group0 status=0
[  586.118635] Exit:dla_handle_events, ret:0
[  586.118893] Enter:dla_handle_events, processor:SDP
[  586.119245] Handle op complete event, processor SDP group 1
[  586.119603] Enter:dla_op_completion processor SDP group1
[  586.119974] Completed SDP operation index 1 ROI 0
[  586.120280] Update dependency operation index 2 ROI 0 DEP_COUNT=1
[  586.120658] enable PDP in dla_update_dependency as depdency are resolved
[  586.121146] Enter: dla_enable_operation
[  586.121467] Enable PDP operation index 2 ROI 0
[  586.121904] group id 0 rdma id 0
[  586.123377] rdma needed 1
[  586.124521] Enter: dla_op_enabled
[  586.125126] Exit: dla_op_enabled
[  586.125432] Exit: dla_enable_operation status=0
[  586.125711] 2 HWLs done, totally 11 layers
[  586.126030] Enter: dla_free_op_desc op desc index 0 ROI 0
[  586.135366] Exit: dla_free_op_desc
[  586.135955] Enter: dla_dequeue_operation
[  586.136501] Dequeue op from SDP processor, index=7 ROI=0
[  586.137211] Enter: dla_submit_operation
[  586.137653] Prepare SDP operation index 7 ROI 0 dep_count 0
[  586.138162] Enter: dla_prepare_operation
[  586.140572] processor:SDP group:1, rdma_group:1 available
[  586.141563] Enter: dla_read_config
[  586.582808] Exit: dla_read_config
[  586.583145] Exit: dla_prepare_operation status=0
[  586.583481] Enter: dla_program_operation
[  586.583781] Program SDP operation index 7 ROI 0 Group[1]
[  586.585279] no desc get due to index==-1
[  586.585709] no desc get due to index==-1
[  586.586339] no desc get due to index==-1
[  586.586812] no desc get due to index==-1
[  586.587357] no desc get due to index==-1
[  586.587681] Enter: dla_op_programmed
[  586.587951] Exit: dla_op_programmed
[  586.588571] Exit: dla_program_operation status=0
[  586.589260] Enter: dla_enable_operation
[  586.589528] Enable SDP operation index 7 ROI 0
[  586.589987] Enter: dla_op_enabled
[  586.590234] Update dependency operation index 6 ROI 0 DEP_COUNT=2
[  586.590579] Exit: dla_op_enabled
[  586.590838] Exit: dla_enable_operation status=0
[  586.591186] Exit: dla_submit_operation
[  586.591509] Exit: dla_dequeue_operation
[  586.591810] Enter: dla_free_op_desc op desc index 1 ROI 0
[  586.600960] Exit: dla_free_op_desc
[  586.602032] Exit:dla_op_completion processor SDP group1 status=0
[  586.602562] Exit:dla_handle_events, ret:0
[  586.602822] Enter:dla_handle_events, processor:PDP
[  586.603110] Exit:dla_handle_events, ret:0
[  586.603349] Enter:dla_handle_events, processor:CDP
[  586.604549] Exit:dla_handle_events, ret:0
[  586.605059] Enter:dla_handle_events, processor:RUBIK
[  586.605653] Exit:dla_handle_events, ret:0
[  586.614545] Enter:dla_handle_events, processor:BDMA
[  586.615066] Exit:dla_handle_events, ret:0
[  586.615987] Enter:dla_handle_events, processor:Convolution
[  586.616564] Exit:dla_handle_events, ret:0
[  586.617060] Enter:dla_handle_events, processor:SDP
[  586.617642] Exit:dla_handle_events, ret:0
[  586.618259] Enter:dla_handle_events, processor:PDP
[  586.618744] Handle op complete event, processor PDP group 0
[  586.619070] Enter:dla_op_completion processor PDP group0
[  586.619476] Completed PDP operation index 2 ROI 0
[  586.620020] Update dependency operation index 3 ROI 0 DEP_COUNT=1
[  586.620935] enable Convolution in dla_update_dependency as depdency are resolved
[  586.621770] Enter: dla_enable_operation
[  586.622673] Enable Convolution operation index 3 ROI 0
[  586.624103] Enter: dla_op_enabled
[  586.624749] Exit: dla_op_enabled
[  586.625490] Exit: dla_enable_operation status=0
[  586.627084] 3 HWLs done, totally 11 layers
[  586.627673] Enter: dla_dequeue_operation
[  586.628412] exit PDP as there's no further operation
[  586.628941] Exit: dla_dequeue_operation
[  586.629744] Enter: dla_free_op_desc op desc index 2 ROI 0
[  586.640648] Exit: dla_free_op_desc
[  586.641184] Exit:dla_op_completion processor PDP group0 status=0
[  586.641995] Exit:dla_handle_events, ret:0
[  586.642367] Enter:dla_handle_events, processor:CDP
[  586.642772] Exit:dla_handle_events, ret:0
[  586.643134] Enter:dla_handle_events, processor:RUBIK
[  586.643470] Exit:dla_handle_events, ret:0
[  586.663481] Enter:dla_handle_events, processor:BDMA
[  586.664007] Exit:dla_handle_events, ret:0
[  586.664297] Enter:dla_handle_events, processor:Convolution
[  586.664639] Exit:dla_handle_events, ret:0
[  586.664958] Enter:dla_handle_events, processor:SDP
[  586.665283] Exit:dla_handle_events, ret:0
[  586.665522] Enter:dla_handle_events, processor:PDP
[  586.665826] Exit:dla_handle_events, ret:0
[  586.666114] Enter:dla_handle_events, processor:CDP
[  586.666388] Exit:dla_handle_events, ret:0
[  586.666572] Enter:dla_handle_events, processor:RUBIK
[  586.666770] Exit:dla_handle_events, ret:0
[  586.770375] Enter:dla_handle_events, processor:BDMA
[  586.770823] Exit:dla_handle_events, ret:0
[  586.771064] Enter:dla_handle_events, processor:Convolution
[  586.771365] Handle cdma data done event, processor Convolution group 1
[  586.771725] Exit:dla_handle_events, ret:0
[  586.772106] Enter:dla_handle_events, processor:SDP
[  586.772397] Exit:dla_handle_events, ret:0
[  586.772651] Enter:dla_handle_events, processor:PDP
[  586.772934] Exit:dla_handle_events, ret:0
[  586.773186] Enter:dla_handle_events, processor:CDP
[  586.773471] Exit:dla_handle_events, ret:0
[  586.773728] Enter:dla_handle_events, processor:RUBIK
[  586.774009] Exit:dla_handle_events, ret:0
[  597.236653] Enter:dla_handle_events, processor:BDMA
[  597.237457] Exit:dla_handle_events, ret:0
[  597.237918] Enter:dla_handle_events, processor:Convolution
[  597.238735] Handle cdma weight done event, processor Convolution group 1
[  597.239157] Handle op complete event, processor Convolution group 1
[  597.239830] Enter:dla_op_completion processor Convolution group1
[  597.240257] Completed Convolution operation index 3 ROI 0
[  597.240584] 4 HWLs done, totally 11 layers
[  597.240847] Enter: dla_dequeue_operation
[  597.241098] Dequeue op from Convolution processor, index=9 ROI=0
[  597.241427] Enter: dla_submit_operation
[  597.241673] Prepare Convolution operation index 9 ROI 0 dep_count 2
[  597.242033] Enter: dla_prepare_operation
[  597.242371] processor:Convolution group:1, rdma_group:0 available
[  597.242725] Enter: dla_read_config
[  597.245918] Exit: dla_read_config
[  597.246219] Exit: dla_prepare_operation status=0
[  597.246536] Enter: dla_program_operation
[  597.246797] Program Convolution operation index 9 ROI 0 Group[1]
[  597.252543] no desc get due to index==-1
[  597.253284] no desc get due to index==-1
[  597.254494] no desc get due to index==-1
[  597.255267] no desc get due to index==-1
[  597.256111] no desc get due to index==-1
[  597.257040] no desc get due to index==-1
[  597.257381] Enter: dla_op_programmed
[  597.257682] Update dependency operation index 10 ROI 0 DEP_COUNT=2
[  597.258137] Exit: dla_op_programmed
[  597.258393] Exit: dla_program_operation status=0
[  597.258730] Exit: dla_submit_operation
[  597.259614] Exit: dla_dequeue_operation
[  597.260028] Exit:dla_op_completion processor Convolution group1 status=0
[  597.260522] Exit:dla_handle_events, ret:0
[  597.260802] Enter:dla_handle_events, processor:SDP
[  597.261200] Handle op complete event, processor SDP group 0
[  597.261616] Enter:dla_op_completion processor SDP group0
[  597.261994] Completed SDP operation index 4 ROI 0
[  597.262322] Update dependency operation index 5 ROI 0 DEP_COUNT=1
[  597.262751] enable PDP in dla_update_dependency as depdency are resolved
[  597.263206] Enter: dla_enable_operation
[  597.263512] Enable PDP operation index 5 ROI 0
[  597.264046] group id 1 rdma id 1
[  597.264663] rdma needed 1
[  597.265068] Enter: dla_op_enabled
[  597.265373] Exit: dla_op_enabled
[  597.265619] Exit: dla_enable_operation status=0
[  597.265891] 5 HWLs done, totally 11 layers
[  597.266223] Enter: dla_free_op_desc op desc index 3 ROI 0
[  597.272499] Exit: dla_free_op_desc
[  597.273221] Enter: dla_dequeue_operation
[  597.273615] Dequeue op from SDP processor, index=8 ROI=0
[  597.274046] Enter: dla_submit_operation
[  597.274338] Prepare SDP operation index 8 ROI 0 dep_count 1
[  597.274739] Enter: dla_prepare_operation
[  597.276621] processor:SDP group:0, rdma_group:0 available
[  597.276972] Enter: dla_read_config
[  597.363197] Exit: dla_read_config
[  597.363950] Exit: dla_prepare_operation status=0
[  597.364477] Enter: dla_program_operation
[  597.365081] Program SDP operation index 8 ROI 0 Group[0]
[  597.400705] no desc get due to index==-1
[  597.400984] no desc get due to index==-1
[  597.401213] no desc get due to index==-1
[  597.401418] no desc get due to index==-1
[  597.401644] no desc get due to index==-1
[  597.401869] Enter: dla_op_programmed
[  597.402082] Update dependency operation index 10 ROI 0 DEP_COUNT=1
[  597.402403] enable SDP in dla_update_dependency as depdency are resolved
[  597.402751] Enter: dla_enable_operation
[  597.402987] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  597.403441] Exit: dla_enable_operation status=0
[  597.403695] Exit: dla_op_programmed
[  597.404175] Exit: dla_program_operation status=0
[  597.404435] Exit: dla_submit_operation
[  597.404698] Exit: dla_dequeue_operation
[  597.404963] Enter: dla_free_op_desc op desc index 4 ROI 0
[  597.406209] Exit: dla_free_op_desc
[  597.406460] Exit:dla_op_completion processor SDP group0 status=0
[  597.406912] Exit:dla_handle_events, ret:0
[  597.407170] Enter:dla_handle_events, processor:PDP
[  597.407467] Exit:dla_handle_events, ret:0
[  597.407721] Enter:dla_handle_events, processor:CDP
[  597.408058] Exit:dla_handle_events, ret:0
[  597.408319] Enter:dla_handle_events, processor:RUBIK
[  597.408678] Exit:dla_handle_events, ret:0
[  597.417471] Enter:dla_handle_events, processor:BDMA
[  597.418115] Exit:dla_handle_events, ret:0
[  597.418522] Enter:dla_handle_events, processor:Convolution
[  597.418967] Exit:dla_handle_events, ret:0
[  597.419256] Enter:dla_handle_events, processor:SDP
[  597.419579] Exit:dla_handle_events, ret:0
[  597.419909] Enter:dla_handle_events, processor:PDP
[  597.420230] Handle op complete event, processor PDP group 1
[  597.420607] Enter:dla_op_completion processor PDP group1
[  597.420962] Completed PDP operation index 5 ROI 0
[  597.421277] Update dependency operation index 6 ROI 0 DEP_COUNT=1
[  597.421647] enable Convolution in dla_update_dependency as depdency are resolved
[  597.422129] Enter: dla_enable_operation
[  597.422438] Enable Convolution operation index 6 ROI 0
[  597.425253] Enter: dla_op_enabled
[  597.426224] Exit: dla_op_enabled
[  597.426647] Exit: dla_enable_operation status=0
[  597.427226] 6 HWLs done, totally 11 layers
[  597.427722] Enter: dla_dequeue_operation
[  597.428066] exit PDP as there's no further operation
[  597.428457] Exit: dla_dequeue_operation
[  597.428822] Enter: dla_free_op_desc op desc index 5 ROI 0
[  597.440142] Exit: dla_free_op_desc
[  597.440418] Exit:dla_op_completion processor PDP group1 status=0
[  597.440869] Exit:dla_handle_events, ret:0
[  597.441218] Enter:dla_handle_events, processor:CDP
[  597.441534] Exit:dla_handle_events, ret:0
[  597.441853] Enter:dla_handle_events, processor:RUBIK
[  597.442162] Exit:dla_handle_events, ret:0
[  597.489124] Enter:dla_handle_events, processor:BDMA
[  597.491533] Exit:dla_handle_events, ret:0
[  597.491985] Enter:dla_handle_events, processor:Convolution
[  597.492591] Handle cdma data done event, processor Convolution group 0
[  597.493254] Exit:dla_handle_events, ret:0
[  597.493759] Enter:dla_handle_events, processor:SDP
[  597.494341] Exit:dla_handle_events, ret:0
[  597.494838] Enter:dla_handle_events, processor:PDP
[  597.495408] Exit:dla_handle_events, ret:0
[  597.495832] Enter:dla_handle_events, processor:CDP
[  597.496340] Exit:dla_handle_events, ret:0
[  597.496758] Enter:dla_handle_events, processor:RUBIK
[  597.497263] Exit:dla_handle_events, ret:0
[  604.610673] Enter:dla_handle_events, processor:BDMA
[  604.611136] Exit:dla_handle_events, ret:0
[  604.611442] Enter:dla_handle_events, processor:Convolution
[  604.611801] Handle cdma weight done event, processor Convolution group 0
[  604.612290] Handle op complete event, processor Convolution group 0
[  604.612613] Enter:dla_op_completion processor Convolution group0
[  604.612929] Completed Convolution operation index 6 ROI 0
[  604.613222] 7 HWLs done, totally 11 layers
[  604.613459] Enter: dla_dequeue_operation
[  604.613687] exit Convolution as there's no further operation
[  604.613984] Exit: dla_dequeue_operation
[  604.614296] Exit:dla_op_completion processor Convolution group0 status=0
[  604.614912] Exit:dla_handle_events, ret:0
[  604.615223] Enter:dla_handle_events, processor:SDP
[  604.615599] Handle op complete event, processor SDP group 1
[  604.616098] Enter:dla_op_completion processor SDP group1
[  604.616629] Completed SDP operation index 7 ROI 0
[  604.617093] Update dependency operation index 8 ROI 0 DEP_COUNT=1
[  604.617528] enable SDP in dla_update_dependency as depdency are resolved
[  604.618027] Enter: dla_enable_operation
[  604.618342] Enable SDP operation index 8 ROI 0
[  604.619758] Enter: dla_op_enabled
[  604.620719] Exit: dla_op_enabled
[  604.621475] Exit: dla_enable_operation status=0
[  604.622289] 8 HWLs done, totally 11 layers
[  604.622852] Enter: dla_free_op_desc op desc index 6 ROI 0
[  604.629311] Exit: dla_free_op_desc
[  604.629604] Enter: dla_dequeue_operation
[  604.629894] Dequeue op from SDP processor, index=10 ROI=0
[  604.630238] Enter: dla_submit_operation
[  604.630531] Prepare SDP operation index 10 ROI 0 dep_count 0
[  604.630915] Enter: dla_prepare_operation
[  604.633142] processor:SDP group:1, rdma_group:1 available
[  604.635590] Enter: dla_read_config
[  604.652555] Exit: dla_read_config
[  604.653174] Exit: dla_prepare_operation status=0
[  604.653511] Enter: dla_program_operation
[  604.653784] Program SDP operation index 10 ROI 0 Group[1]
[  604.655910] no desc get due to index==-1
[  604.656168] no desc get due to index==-1
[  604.656433] no desc get due to index==-1
[  604.656694] no desc get due to index==-1
[  604.656951] no desc get due to index==-1
[  604.657209] no desc get due to index==-1
[  604.657476] Enter: dla_op_programmed
[  604.657734] Exit: dla_op_programmed
[  604.657974] Exit: dla_program_operation status=0
[  604.658272] Enter: dla_enable_operation
[  604.658567] Enable SDP operation index 10 ROI 0
[  604.659081] Enter: dla_op_enabled
[  604.659366] Update dependency operation index 9 ROI 0 DEP_COUNT=2
[  604.659818] Exit: dla_op_enabled
[  604.660076] Exit: dla_enable_operation status=0
[  604.660380] Exit: dla_submit_operation
[  604.660700] Exit: dla_dequeue_operation
[  604.661023] Enter: dla_free_op_desc op desc index 7 ROI 0
[  604.662251] Exit: dla_free_op_desc
[  604.662515] Exit:dla_op_completion processor SDP group1 status=0
[  604.662911] Exit:dla_handle_events, ret:0
[  604.663226] Enter:dla_handle_events, processor:PDP
[  604.663594] Exit:dla_handle_events, ret:0
[  604.663954] Enter:dla_handle_events, processor:CDP
[  604.664309] Exit:dla_handle_events, ret:0
[  604.664618] Enter:dla_handle_events, processor:RUBIK
[  604.665006] Exit:dla_handle_events, ret:0
[  604.710628] Enter:dla_handle_events, processor:BDMA
[  604.711068] Exit:dla_handle_events, ret:0
[  604.711310] Enter:dla_handle_events, processor:Convolution
[  604.711638] Exit:dla_handle_events, ret:0
[  604.711929] Enter:dla_handle_events, processor:SDP
[  604.712234] Handle op complete event, processor SDP group 0
[  604.712552] Enter:dla_op_completion processor SDP group0
[  604.712864] Completed SDP operation index 8 ROI 0
[  604.713155] Update dependency operation index 9 ROI 0 DEP_COUNT=1
[  604.713558] enable Convolution in dla_update_dependency as depdency are resolved
[  604.714049] Enter: dla_enable_operation
[  604.714320] Enable Convolution operation index 9 ROI 0
[  604.715059] Enter: dla_op_enabled
[  604.715334] Exit: dla_op_enabled
[  604.715555] Exit: dla_enable_operation status=0
[  604.715886] 9 HWLs done, totally 11 layers
[  604.716222] Enter: dla_dequeue_operation
[  604.716510] exit SDP as there's no further operation
[  604.716849] Exit: dla_dequeue_operation
[  604.717122] Enter: dla_free_op_desc op desc index 8 ROI 0
[  604.725838] Exit: dla_free_op_desc
[  604.726407] Exit:dla_op_completion processor SDP group0 status=0
[  604.727170] Exit:dla_handle_events, ret:0
[  604.727545] Enter:dla_handle_events, processor:PDP
[  604.728127] Exit:dla_handle_events, ret:0
[  604.728437] Enter:dla_handle_events, processor:CDP
[  604.728807] Exit:dla_handle_events, ret:0
[  604.729043] Enter:dla_handle_events, processor:RUBIK
[  604.729338] Exit:dla_handle_events, ret:0
[  604.815960] Enter:dla_handle_events, processor:BDMA
[  604.816498] Exit:dla_handle_events, ret:0
[  604.816801] Enter:dla_handle_events, processor:Convolution
[  604.817199] Handle cdma data done event, processor Convolution group 1
[  604.817663] Exit:dla_handle_events, ret:0
[  604.818005] Enter:dla_handle_events, processor:SDP
[  604.818430] Exit:dla_handle_events, ret:0
[  604.818769] Enter:dla_handle_events, processor:PDP
[  604.819362] Exit:dla_handle_events, ret:0
[  604.819718] Enter:dla_handle_events, processor:CDP
[  604.820157] Exit:dla_handle_events, ret:0
[  604.820505] Enter:dla_handle_events, processor:RUBIK
[  604.820922] Exit:dla_handle_events, ret:0
[  604.888037] Enter:dla_handle_events, processor:BDMA
[  604.889383] Exit:dla_handle_events, ret:0
[  604.890492] Enter:dla_handle_events, processor:Convolution
[  604.891109] Handle cdma weight done event, processor Convolution group 1
[  604.891756] Exit:dla_handle_events, ret:0
[  604.892102] Enter:dla_handle_events, processor:SDP
[  604.892496] Exit:dla_handle_events, ret:0
[  604.892843] Enter:dla_handle_events, processor:PDP
[  604.893229] Exit:dla_handle_events, ret:0
[  604.893580] Enter:dla_handle_events, processor:CDP
[  604.893977] Exit:dla_handle_events, ret:0
[  604.894318] Enter:dla_handle_events, processor:RUBIK
[  604.894729] Exit:dla_handle_events, ret:0
[  604.902109] Enter:dla_handle_events, processor:BDMA
[  604.902590] Exit:dla_handle_events, ret:0
[  604.902857] Enter:dla_handle_events, processor:Convolution
[  604.903181] Handle op complete event, processor Convolution group 1
[  604.903510] Enter:dla_op_completion processor Convolution group1
[  604.903921] Completed Convolution operation index 9 ROI 0
[  604.904580] 10 HWLs done, totally 11 layers
[  604.905229] Enter: dla_dequeue_operation
[  604.905921] exit Convolution as there's no further operation
[  604.906490] Exit: dla_dequeue_operation
[  604.906785] Exit:dla_op_completion processor Convolution group1 status=0
[  604.907172] Exit:dla_handle_events, ret:0
[  604.907440] Enter:dla_handle_events, processor:SDP
[  604.907750] Handle op complete event, processor SDP group 1
[  604.908117] Enter:dla_op_completion processor SDP group1
[  604.908477] Completed SDP operation index 10 ROI 0
[  604.908762] 11 HWLs done, totally 11 layers
[  604.909071] Enter: dla_free_op_desc op desc index 9 ROI 0
[  604.909667] Exit: dla_free_op_desc
[  604.909917] Enter: dla_free_op_desc op desc index 10 ROI 0
[  604.910532] Exit: dla_free_op_desc
[  604.910800] Exit:dla_op_completion processor SDP group1 status=0
[  604.911113] Exit:dla_handle_events, ret:0
[  604.911377] Enter:dla_handle_events, processor:PDP
[  604.911614] Exit:dla_handle_events, ret:0
[  604.911891] Enter:dla_handle_events, processor:CDP
[  604.912179] Exit:dla_handle_events, ret:0
[  604.912467] Enter:dla_handle_events, processor:RUBIK
[  604.912776] Exit:dla_handle_events, ret:0
[  604.922540] reset engine done
Work Found!
Work Done
Test pass
2.6226e-06 0.0133286 2.27094e-05 0.00288963 4.82798e-05 7.83801e-05 2.38419e-07 0.983398 3.57628e-07 0.000276089

It works:

./nvdla_runtime --loadable sw/regression/flatbufs/kmd/NN/NN_L0_0_fbuf --image 9644 0.000207067 0.048645 # 
sw/regression/images/digits/one.pgm --rawdump && cat output.dimg 
creating new runtime context...
Emulator starting
ppgminfo 1 28 28
pgm2dimg 1 28 28 1 32 896 896
submitting tasks...
Work Found!
Work Done
[  848.355580] Enter:dla_read_network_config
[  848.362052] Exit:dla_read_network_config status=0
[  848.362534] Enter: dla_initiate_processors
[  848.363124] Enter: dla_submit_operation
[  848.363448] Prepare Convolution operation index 0 ROI 0 dep_count 1
[  848.364141] Enter: dla_prepare_operation
[  848.368045] processor:Convolution group:0, rdma_group:0 available
[  848.371031] Enter: dla_read_config
[  848.379254] Exit: dla_read_config
[  848.379618] Exit: dla_prepare_operation status=0
[  848.380034] Enter: dla_program_operation
[  848.380400] Program Convolution operation index 0 ROI 0 Group[0]
[  848.387653] no desc get due to index==-1
[  848.395011] no desc get due to index==-1
[  848.395614] no desc get due to index==-1
[  848.396187] no desc get due to index==-1
[  848.396501] no desc get due to index==-1
[  848.396826] Enter: dla_op_programmed
[  848.397072] Update dependency operation index 3 ROI 0 DEP_COUNT=3
[  848.397401] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[  848.397756] enable SDP in dla_update_dependency as depdency are resolved
[  848.400465] Enter: dla_enable_operation
[  848.402884] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  848.403447] Exit: dla_enable_operation status=0
[  848.403733] Exit: dla_op_programmed
[  848.404045] Exit: dla_program_operation status=0
[  848.405540] Exit: dla_submit_operation
[  848.405798] Enter: dla_dequeue_operation
[  848.406043] Dequeue op from Convolution processor, index=3 ROI=0
[  848.406637] Enter: dla_submit_operation
[  848.406949] Prepare Convolution operation index 3 ROI 0 dep_count 2
[  848.407365] Enter: dla_prepare_operation
[  848.408124] processor:Convolution group:1, rdma_group:0 available
[  848.415617] Enter: dla_read_config
[  848.431172] Exit: dla_read_config
[  848.431493] Exit: dla_prepare_operation status=0
[  848.431659] Enter: dla_program_operation
[  848.431981] Program Convolution operation index 3 ROI 0 Group[1]
[  848.438018] no desc get due to index==-1
[  848.439043] no desc get due to index==-1
[  848.439306] no desc get due to index==-1
[  848.439561] no desc get due to index==-1
[  848.439797] no desc get due to index==-1
[  848.445216] Enter: dla_op_programmed
[  848.445854] Update dependency operation index 6 ROI 0 DEP_COUNT=3
[  848.446298] Update dependency operation index 4 ROI 0 DEP_COUNT=2
[  848.446684] Exit: dla_op_programmed
[  848.446951] Exit: dla_program_operation status=0
[  848.447270] Exit: dla_submit_operation
[  848.447536] Exit: dla_dequeue_operation
[  848.447825] Enter: dla_submit_operation
[  848.450148] Prepare SDP operation index 1 ROI 0 dep_count 0
[  848.450537] Enter: dla_prepare_operation
[  848.451052] processor:SDP group:1, rdma_group:1 available
[  848.451383] Enter: dla_read_config
[  848.466083] Exit: dla_read_config
[  848.466501] Exit: dla_prepare_operation status=0
[  848.466964] Enter: dla_program_operation
[  848.467331] Program SDP operation index 1 ROI 0 Group[1]
[  848.470769] no desc get due to index==-1
[  848.471070] no desc get due to index==-1
[  848.471582] no desc get due to index==-1
[  848.473092] no desc get due to index==-1
[  848.473484] Enter: dla_op_programmed
[  848.473721] Update dependency operation index 4 ROI 0 DEP_COUNT=1
[  848.474055] enable SDP in dla_update_dependency as depdency are resolved
[  848.474476] Enter: dla_enable_operation
[  848.474816] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  848.475581] Exit: dla_enable_operation status=0
[  848.476063] Exit: dla_op_programmed
[  848.476292] Exit: dla_program_operation status=0
[  848.476756] Enter: dla_enable_operation
[  848.477212] Enable SDP operation index 1 ROI 0
[  848.488280] Enter: dla_op_enabled
[  848.488626] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[  848.489067] enable Convolution in dla_update_dependency as depdency are resolved
[  848.489486] Enter: dla_enable_operation
[  848.489826] Enable Convolution operation index 0 ROI 0
[  848.491268] Enter: dla_op_enabled
[  848.491609] Exit: dla_op_enabled
[  848.491980] Exit: dla_enable_operation status=0
[  848.492402] Exit: dla_op_enabled
[  848.492711] Exit: dla_enable_operation status=0
[  848.493040] Exit: dla_submit_operation
[  848.493336] Enter: dla_dequeue_operation
[  848.493650] Dequeue op from SDP processor, index=4 ROI=0
[  848.494149] Enter: dla_submit_operation
[  848.494609] Prepare SDP operation index 4 ROI 0 dep_count 0
[  848.495188] Enter: dla_prepare_operation
[  848.508915] processor:SDP group:0, rdma_group:0 available
[  848.509375] Enter: dla_read_config
[  848.645306] Exit: dla_read_config
[  848.645683] Exit: dla_prepare_operation status=0
[  848.646032] Enter: dla_program_operation
[  898.126132] Program SDP operation index 4 ROI 0 Group[0]
[  898.137800] no desc get due to index==-1
[  898.138466] no desc get due to index==-1
[  898.140102] no desc get due to index==-1
[  898.144637] no desc get due to index==-1
[  898.149494] Enter: dla_op_programmed
[  898.153768] Update dependency operation index 7 ROI 0 DEP_COUNT=2
[  898.154360] Exit: dla_op_programmed
[  898.154853] Exit: dla_program_operation status=0
[  898.155715] Enter: dla_enable_operation
[  898.159082] Enable SDP operation index 4 ROI 0
[  898.159801] Enter: dla_op_enabled
[  898.160128] Update dependency operation index 3 ROI 0 DEP_COUNT=2
[  898.162230] Exit: dla_op_enabled
[  898.162568] Exit: dla_enable_operation status=0
[  898.163079] Exit: dla_submit_operation
[  898.163427] Exit: dla_dequeue_operation
[  898.163816] Enter: dla_submit_operation
[  898.166159] Prepare PDP operation index 2 ROI 0 dep_count 1
[  898.167287] Enter: dla_prepare_operation
[  898.176916] processor:PDP group:0, rdma_group:0 available
[  898.177360] Enter: dla_read_config
[  898.183167] Exit: dla_read_config
[  898.185144] Exit: dla_prepare_operation status=0
[  898.185509] Enter: dla_program_operation
[  898.185863] Program PDP operation index 2 ROI 0 Group[0]
[  898.186290] group id 0 rdma id 0
[  898.188007] no desc get due to index==-1
[  898.190511] no desc get due to index==-1
[  898.190806] no desc get due to index==-1
[  898.191069] no desc get due to index==-1
[  898.191364] no desc get due to index==-1
[  898.191663] Enter: dla_op_programmed
[  898.193859] Update dependency operation index 5 ROI 0 DEP_COUNT=2
[  898.194335] Exit: dla_op_programmed
[  898.194658] Exit: dla_program_operation status=0
[  898.194990] Exit: dla_submit_operation
[  898.195267] Enter: dla_dequeue_operation
[  898.195589] Dequeue op from PDP processor, index=5 ROI=0
[  898.197313] Enter: dla_submit_operation
[  898.197619] Prepare PDP operation index 5 ROI 0 dep_count 1
[  898.198093] Enter: dla_prepare_operation
[  898.201922] processor:PDP group:1, rdma_group:1 available
[  898.202724] Enter: dla_read_config
[  898.206679] Exit: dla_read_config
[  898.208402] Exit: dla_prepare_operation status=0
[  898.208733] Enter: dla_program_operation
[  898.208997] Program PDP operation index 5 ROI 0 Group[1]
[  898.209377] group id 1 rdma id 1
[  898.210948] no desc get due to index==-1
[  898.211277] no desc get due to index==-1
[  898.211639] no desc get due to index==-1
[  898.212088] no desc get due to index==-1
[  898.212492] no desc get due to index==-1
[  898.214899] no desc get due to index==-1
[  898.215159] Enter: dla_op_programmed
[  898.215424] Exit: dla_op_programmed
[  898.215681] Exit: dla_program_operation status=0
[  898.217317] Exit: dla_submit_operation
[  898.217881] Exit: dla_dequeue_operation
[  898.218156] Exit: dla_initiate_processors status=0
[  898.218482] Enter:dla_handle_events, processor:BDMA
[  898.218784] Exit:dla_handle_events, ret:0
[  898.219058] Enter:dla_handle_events, processor:Convolution
[  898.219388] Handle cdma weight done event, processor Convolution group 0
[  898.219813] Handle cdma data done event, processor Convolution group 0
[  898.220290] Handle op complete event, processor Convolution group 0
[  898.220700] Enter:dla_op_completion processor Convolution group0
[  898.222250] Completed Convolution operation index 0 ROI 0
[  898.222677] 1 HWLs done, totally 11 layers
[  898.222994] Enter: dla_dequeue_operation
[  898.223298] Dequeue op from Convolution processor, index=6 ROI=0
[  898.223741] Enter: dla_submit_operation
[  898.224036] Prepare Convolution operation index 6 ROI 0 dep_count 2
[  898.224473] Enter: dla_prepare_operation
[  898.224995] processor:Convolution group:0, rdma_group:0 available
[  898.225594] Enter: dla_read_config
[  898.230681] Exit: dla_read_config
[  898.231023] Exit: dla_prepare_operation status=0
[  898.232445] Enter: dla_program_operation
[  898.233113] Program Convolution operation index 6 ROI 0 Group[0]
[  898.236090] no desc get due to index==-1
[  898.236942] no desc get due to index==-1
[  898.237290] no desc get due to index==-1
[  898.237639] no desc get due to index==-1
[  898.237954] no desc get due to index==-1
[  898.238270] Enter: dla_op_programmed
[  898.238566] Update dependency operation index 9 ROI 0 DEP_COUNT=3
[  898.238986] Update dependency operation index 7 ROI 0 DEP_COUNT=1
[  898.239427] enable SDP in dla_update_dependency as depdency are resolved
[  898.239922] Enter: dla_enable_operation
[  898.240236] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  898.240814] Exit: dla_enable_operation status=0
[  898.241132] Exit: dla_op_programmed
[  898.241415] Exit: dla_program_operation status=0
[  898.241774] Exit: dla_submit_operation
[  898.242071] Exit: dla_dequeue_operation
[  898.242376] Exit:dla_op_completion processor Convolution group0 status=0
[  898.242852] Exit:dla_handle_events, ret:0
[  898.243149] Enter:dla_handle_events, processor:SDP
[  898.243490] Handle op complete event, processor SDP group 1
[  898.243897] Enter:dla_op_completion processor SDP group1
[  898.244256] Completed SDP operation index 1 ROI 0
[  898.244648] Update dependency operation index 2 ROI 0 DEP_COUNT=1
[  898.245643] enable PDP in dla_update_dependency as depdency are resolved
[  898.246129] Enter: dla_enable_operation
[  898.246448] Enable PDP operation index 2 ROI 0
[  898.246782] group id 0 rdma id 0
[  898.247204] rdma needed 1
[  898.247568] Enter: dla_op_enabled
[  898.247876] Exit: dla_op_enabled
[  898.248154] Exit: dla_enable_operation status=0
[  898.248484] 2 HWLs done, totally 11 layers
[  898.248901] Enter: dla_free_op_desc op desc index 0 ROI 0
[  898.257495] Exit: dla_free_op_desc
[  898.258162] Enter: dla_dequeue_operation
[  898.258537] Dequeue op from SDP processor, index=7 ROI=0
[  898.258929] Enter: dla_submit_operation
[  898.259283] Prepare SDP operation index 7 ROI 0 dep_count 0
[  898.259755] Enter: dla_prepare_operation
[  898.262676] processor:SDP group:1, rdma_group:1 available
[  898.263090] Enter: dla_read_config
[  898.344503] Exit: dla_read_config
[  898.344804] Exit: dla_prepare_operation status=0
[  898.345117] Enter: dla_program_operation
[  898.345394] Program SDP operation index 7 ROI 0 Group[1]
[  898.382188] no desc get due to index==-1
[  898.393498] no desc get due to index==-1
[  898.410318] no desc get due to index==-1
[  898.411026] no desc get due to index==-1
[  898.411840] no desc get due to index==-1
[  898.413540] Enter: dla_op_programmed
[  898.413809] Exit: dla_op_programmed
[  898.417117] Exit: dla_program_operation status=0
[  898.418537] Enter: dla_enable_operation
[  898.418826] Enable SDP operation index 7 ROI 0
[  898.422069] Enter: dla_op_enabled
[  898.422335] Update dependency operation index 6 ROI 0 DEP_COUNT=2
[  898.422687] Exit: dla_op_enabled
[  898.422997] Exit: dla_enable_operation status=0
[  898.423341] Exit: dla_submit_operation
[  898.423601] Exit: dla_dequeue_operation
[  898.424561] Enter: dla_free_op_desc op desc index 1 ROI 0
[  898.735631] Exit: dla_free_op_desc
[  898.735960] Exit:dla_op_completion processor SDP group1 status=0
[  898.736329] Exit:dla_handle_events, ret:0
[  898.736585] Enter:dla_handle_events, processor:PDP
[  898.736921] Exit:dla_handle_events, ret:0
[  898.737199] Enter:dla_handle_events, processor:CDP
[  898.737529] Exit:dla_handle_events, ret:0
[  898.737794] Enter:dla_handle_events, processor:RUBIK
[  898.738089] Exit:dla_handle_events, ret:0
[  898.745218] Enter:dla_handle_events, processor:BDMA
[  898.746059] Exit:dla_handle_events, ret:0
[  898.746351] Enter:dla_handle_events, processor:Convolution
[  898.746699] Exit:dla_handle_events, ret:0
[  898.746991] Enter:dla_handle_events, processor:SDP
[  898.747356] Exit:dla_handle_events, ret:0
[  898.747640] Enter:dla_handle_events, processor:PDP
[  898.747999] Handle op complete event, processor PDP group 0
[  898.748391] Enter:dla_op_completion processor PDP group0
[  898.748780] Completed PDP operation index 2 ROI 0
[  898.749133] Update dependency operation index 3 ROI 0 DEP_COUNT=1
[  898.749556] enable Convolution in dla_update_dependency as depdency are resolved
[  898.750075] Enter: dla_enable_operation
[  898.750368] Enable Convolution operation index 3 ROI 0
[  898.751896] Enter: dla_op_enabled
[  898.752225] Exit: dla_op_enabled
[  898.752469] Exit: dla_enable_operation status=0
[  898.752812] 3 HWLs done, totally 11 layers
[  898.753157] Enter: dla_dequeue_operation
[  898.753483] exit PDP as there's no further operation
[  898.753863] Exit: dla_dequeue_operation
[  898.754261] Enter: dla_free_op_desc op desc index 2 ROI 0
[  898.764316] Exit: dla_free_op_desc
[  898.764616] Exit:dla_op_completion processor PDP group0 status=0
[  898.764946] Exit:dla_handle_events, ret:0
[  898.765190] Enter:dla_handle_events, processor:CDP
[  898.766898] Exit:dla_handle_events, ret:0
[  898.767260] Enter:dla_handle_events, processor:RUBIK
[  898.767574] Exit:dla_handle_events, ret:0
[  898.814206] Enter:dla_handle_events, processor:BDMA
[  898.814648] Exit:dla_handle_events, ret:0
[  898.814942] Enter:dla_handle_events, processor:Convolution
[  898.815250] Exit:dla_handle_events, ret:0
[  898.815508] Enter:dla_handle_events, processor:SDP
[  898.815818] Exit:dla_handle_events, ret:0
[  898.816128] Enter:dla_handle_events, processor:PDP
[  898.816427] Exit:dla_handle_events, ret:0
[  898.816701] Enter:dla_handle_events, processor:CDP
[  898.816998] Exit:dla_handle_events, ret:0
[  898.817249] Enter:dla_handle_events, processor:RUBIK
[  898.817534] Exit:dla_handle_events, ret:0
[  898.924412] Enter:dla_handle_events, processor:BDMA
[  898.924824] Exit:dla_handle_events, ret:0
[  898.925081] Enter:dla_handle_events, processor:Convolution
[  898.925388] Handle cdma data done event, processor Convolution group 1
[  898.925794] Exit:dla_handle_events, ret:0
[  898.926045] Enter:dla_handle_events, processor:SDP
[  898.926327] Exit:dla_handle_events, ret:0
[  898.926573] Enter:dla_handle_events, processor:PDP
[  898.926952] Exit:dla_handle_events, ret:0
[  898.927202] Enter:dla_handle_events, processor:CDP
[  898.927483] Exit:dla_handle_events, ret:0
[  898.927729] Enter:dla_handle_events, processor:RUBIK
[  898.928362] Exit:dla_handle_events, ret:0
[  912.508730] Enter:dla_handle_events, processor:BDMA
[  912.509136] Exit:dla_handle_events, ret:0
[  912.509392] Enter:dla_handle_events, processor:Convolution
[  912.509715] Handle cdma weight done event, processor Convolution group 1
[  912.510118] Handle op complete event, processor Convolution group 1
[  912.510589] Enter:dla_op_completion processor Convolution group1
[  912.510986] Completed Convolution operation index 3 ROI 0
[  912.511426] 4 HWLs done, totally 11 layers
[  912.511728] Enter: dla_dequeue_operation
[  912.512557] Dequeue op from Convolution processor, index=9 ROI=0
[  912.513631] Enter: dla_submit_operation
[  912.514628] Prepare Convolution operation index 9 ROI 0 dep_count 2
[  912.515828] Enter: dla_prepare_operation
[  912.516227] processor:Convolution group:1, rdma_group:0 available
[  912.516587] Enter: dla_read_config
[  912.519746] Exit: dla_read_config
[  912.520035] Exit: dla_prepare_operation status=0
[  912.520317] Enter: dla_program_operation
[  912.520549] Program Convolution operation index 9 ROI 0 Group[1]
[  912.536292] no desc get due to index==-1
[  912.536794] no desc get due to index==-1
[  912.538385] no desc get due to index==-1
[  912.538736] no desc get due to index==-1
[  912.539098] no desc get due to index==-1
[  912.539448] no desc get due to index==-1
[  912.539824] Enter: dla_op_programmed
[  912.540229] Update dependency operation index 10 ROI 0 DEP_COUNT=2
[  912.540684] Exit: dla_op_programmed
[  912.541520] Exit: dla_program_operation status=0
[  912.542707] Exit: dla_submit_operation
[  912.543220] Exit: dla_dequeue_operation
[  912.544061] Exit:dla_op_completion processor Convolution group1 status=0
[  912.544620] Exit:dla_handle_events, ret:0
[  912.544997] Enter:dla_handle_events, processor:SDP
[  912.545500] Handle op complete event, processor SDP group 0
[  912.545874] Enter:dla_op_completion processor SDP group0
[  912.546215] Completed SDP operation index 4 ROI 0
[  912.546574] Update dependency operation index 5 ROI 0 DEP_COUNT=1
[  912.546973] enable PDP in dla_update_dependency as depdency are resolved
[  912.547404] Enter: dla_enable_operation
[  912.547654] Enable PDP operation index 5 ROI 0
[  912.548143] group id 1 rdma id 1
[  912.548634] rdma needed 1
[  912.549083] Enter: dla_op_enabled
[  912.549429] Exit: dla_op_enabled
[  912.549880] Exit: dla_enable_operation status=0
[  912.550290] 5 HWLs done, totally 11 layers
[  912.550970] Enter: dla_free_op_desc op desc index 3 ROI 0
[  912.558473] Exit: dla_free_op_desc
[  912.558759] Enter: dla_dequeue_operation
[  912.559063] Dequeue op from SDP processor, index=8 ROI=0
[  912.559543] Enter: dla_submit_operation
[  912.559805] Prepare SDP operation index 8 ROI 0 dep_count 1
[  912.560322] Enter: dla_prepare_operation
[  912.567720] processor:SDP group:0, rdma_group:0 available
[  912.569240] Enter: dla_read_config
[  912.700006] Exit: dla_read_config
[  912.700325] Exit: dla_prepare_operation status=0
[  912.700622] Enter: dla_program_operation
[  912.700902] Program SDP operation index 8 ROI 0 Group[0]
[  912.702470] no desc get due to index==-1
[  912.702771] no desc get due to index==-1
[  912.703170] no desc get due to index==-1
[  912.703503] no desc get due to index==-1
[  912.704891] no desc get due to index==-1
[  912.705447] Enter: dla_op_programmed
[  912.705697] Update dependency operation index 10 ROI 0 DEP_COUNT=1
[  912.706173] enable SDP in dla_update_dependency as depdency are resolved
[  912.706596] Enter: dla_enable_operation
[  912.706950] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  912.707527] Exit: dla_enable_operation status=0
[  912.707981] Exit: dla_op_programmed
[  912.708262] Exit: dla_program_operation status=0
[  912.708534] Exit: dla_submit_operation
[  912.708818] Exit: dla_dequeue_operation
[  912.709112] Enter: dla_free_op_desc op desc index 4 ROI 0
[  912.715133] Exit: dla_free_op_desc
[  912.715717] Exit:dla_op_completion processor SDP group0 status=0
[  912.716576] Exit:dla_handle_events, ret:0
[  912.717155] Enter:dla_handle_events, processor:PDP
[  912.717575] Exit:dla_handle_events, ret:0
[  912.717960] Enter:dla_handle_events, processor:CDP
[  912.718437] Exit:dla_handle_events, ret:0
[  912.718855] Enter:dla_handle_events, processor:RUBIK
[  912.719338] Exit:dla_handle_events, ret:0
[  912.756386] Enter:dla_handle_events, processor:BDMA
[  912.756783] Exit:dla_handle_events, ret:0
[  912.757038] Enter:dla_handle_events, processor:Convolution
[  912.757351] Exit:dla_handle_events, ret:0
[  912.757599] Enter:dla_handle_events, processor:SDP
[  912.757884] Exit:dla_handle_events, ret:0
[  912.758131] Enter:dla_handle_events, processor:PDP
[  912.758419] Handle op complete event, processor PDP group 1
[  912.758739] Enter:dla_op_completion processor PDP group1
[  912.759048] Completed PDP operation index 5 ROI 0
[  912.759336] Update dependency operation index 6 ROI 0 DEP_COUNT=1
[  912.759674] enable Convolution in dla_update_dependency as depdency are resolved
[  912.760805] Enter: dla_enable_operation
[  912.761452] Enable Convolution operation index 6 ROI 0
[  912.763092] Enter: dla_op_enabled
[  912.763917] Exit: dla_op_enabled
[  912.764686] Exit: dla_enable_operation status=0
[  912.765530] 6 HWLs done, totally 11 layers
[  912.766467] Enter: dla_dequeue_operation
[  912.767220] exit PDP as there's no further operation
[  912.768261] Exit: dla_dequeue_operation
[  912.768899] Enter: dla_free_op_desc op desc index 5 ROI 0
[  912.790498] Exit: dla_free_op_desc
[  912.790799] Exit:dla_op_completion processor PDP group1 status=0
[  912.791148] Exit:dla_handle_events, ret:0
[  912.791393] Enter:dla_handle_events, processor:CDP
[  912.791686] Exit:dla_handle_events, ret:0
[  912.791982] Enter:dla_handle_events, processor:RUBIK
[  912.792296] Exit:dla_handle_events, ret:0
[  912.843412] Enter:dla_handle_events, processor:BDMA
[  912.851671] Exit:dla_handle_events, ret:0
[  912.859300] Enter:dla_handle_events, processor:Convolution
[  912.859641] Handle cdma data done event, processor Convolution group 0
[  912.860023] Exit:dla_handle_events, ret:0
[  912.860282] Enter:dla_handle_events, processor:SDP
[  912.860579] Exit:dla_handle_events, ret:0
[  912.860836] Enter:dla_handle_events, processor:PDP
[  912.861126] Exit:dla_handle_events, ret:0
[  912.861386] Enter:dla_handle_events, processor:CDP
[  912.861697] Exit:dla_handle_events, ret:0
[  912.870324] Enter:dla_handle_events, processor:RUBIK
[  912.874039] Exit:dla_handle_events, ret:0
[  921.518348] random: crng init done
[  921.521749] Enter:dla_handle_events, processor:BDMA
[  921.522296] Exit:dla_handle_events, ret:0
[  921.522589] Enter:dla_handle_events, processor:Convolution
[  921.522934] Handle cdma weight done event, processor Convolution group 0
[  921.523312] Handle op complete event, processor Convolution group 0
[  921.523638] Enter:dla_op_completion processor Convolution group0
[  921.524194] Completed Convolution operation index 6 ROI 0
[  921.524688] 7 HWLs done, totally 11 layers
[  921.524957] Enter: dla_dequeue_operation
[  921.525207] exit Convolution as there's no further operation
[  921.525548] Exit: dla_dequeue_operation
[  921.525943] Exit:dla_op_completion processor Convolution group0 status=0
[  921.526357] Exit:dla_handle_events, ret:0
[  921.526668] Enter:dla_handle_events, processor:SDP
[  921.527006] Handle op complete event, processor SDP group 1
[  921.527335] Enter:dla_op_completion processor SDP group1
[  921.527708] Completed SDP operation index 7 ROI 0
[  921.527996] Update dependency operation index 8 ROI 0 DEP_COUNT=1
[  921.528442] enable SDP in dla_update_dependency as depdency are resolved
[  921.528820] Enter: dla_enable_operation
[  921.529058] Enable SDP operation index 8 ROI 0
[  921.529509] Enter: dla_op_enabled
[  921.529736] Exit: dla_op_enabled
[  921.529941] Exit: dla_enable_operation status=0
[  921.530201] 8 HWLs done, totally 11 layers
[  921.530440] Enter: dla_free_op_desc op desc index 6 ROI 0
[  921.539232] Exit: dla_free_op_desc
[  921.540077] Enter: dla_dequeue_operation
[  921.540394] Dequeue op from SDP processor, index=10 ROI=0
[  921.540758] Enter: dla_submit_operation
[  921.541030] Prepare SDP operation index 10 ROI 0 dep_count 0
[  921.541390] Enter: dla_prepare_operation
[  921.544393] processor:SDP group:1, rdma_group:1 available
[  921.544695] Enter: dla_read_config
[  921.564934] Exit: dla_read_config
[  921.565282] Exit: dla_prepare_operation status=0
[  921.565614] Enter: dla_program_operation
[  921.565908] Program SDP operation index 10 ROI 0 Group[1]
[  921.567333] no desc get due to index==-1
[  921.567643] no desc get due to index==-1
[  921.567973] no desc get due to index==-1
[  921.568243] no desc get due to index==-1
[  921.568491] no desc get due to index==-1
[  921.568771] no desc get due to index==-1
[  921.569049] Enter: dla_op_programmed
[  921.569299] Exit: dla_op_programmed
[  921.569545] Exit: dla_program_operation status=0
[  921.569825] Enter: dla_enable_operation
[  921.570114] Enable SDP operation index 10 ROI 0
[  921.570584] Enter: dla_op_enabled
[  921.570836] Update dependency operation index 9 ROI 0 DEP_COUNT=2
[  921.571331] Exit: dla_op_enabled
[  921.571566] Exit: dla_enable_operation status=0
[  921.571896] Exit: dla_submit_operation
[  921.572169] Exit: dla_dequeue_operation
[  921.572475] Enter: dla_free_op_desc op desc index 7 ROI 0
[  921.573704] Exit: dla_free_op_desc
[  921.574107] Exit:dla_op_completion processor SDP group1 status=0
[  921.574509] Exit:dla_handle_events, ret:0
[  921.574788] Enter:dla_handle_events, processor:PDP
[  921.575102] Exit:dla_handle_events, ret:0
[  921.575417] Enter:dla_handle_events, processor:CDP
[  921.575772] Exit:dla_handle_events, ret:0
[  921.576063] Enter:dla_handle_events, processor:RUBIK
[  921.576493] Exit:dla_handle_events, ret:0
[  921.618195] Enter:dla_handle_events, processor:BDMA
[  921.618719] Exit:dla_handle_events, ret:0
[  921.619003] Enter:dla_handle_events, processor:Convolution
[  921.619361] Exit:dla_handle_events, ret:0
[  921.619627] Enter:dla_handle_events, processor:SDP
[  921.619980] Handle op complete event, processor SDP group 0
[  921.620325] Enter:dla_op_completion processor SDP group0
[  921.620643] Completed SDP operation index 8 ROI 0
[  921.620962] Update dependency operation index 9 ROI 0 DEP_COUNT=1
[  921.621298] enable Convolution in dla_update_dependency as depdency are resolved
[  921.621742] Enter: dla_enable_operation
[  921.622030] Enable Convolution operation index 9 ROI 0
[  921.622754] Enter: dla_op_enabled
[  921.622992] Exit: dla_op_enabled
[  921.623242] Exit: dla_enable_operation status=0
[  921.623517] 9 HWLs done, totally 11 layers
[  921.623800] Enter: dla_dequeue_operation
[  921.624109] exit SDP as there's no further operation
[  921.624434] Exit: dla_dequeue_operation
[  921.624723] Enter: dla_free_op_desc op desc index 8 ROI 0
[  921.632210] Exit: dla_free_op_desc
[  921.632538] Exit:dla_op_completion processor SDP group0 status=0
[  921.633017] Exit:dla_handle_events, ret:0
[  921.633285] Enter:dla_handle_events, processor:PDP
[  921.633623] Exit:dla_handle_events, ret:0
[  921.633936] Enter:dla_handle_events, processor:CDP
[  921.634230] Exit:dla_handle_events, ret:0
[  921.634538] Enter:dla_handle_events, processor:RUBIK
[  921.634884] Exit:dla_handle_events, ret:0
[  921.729713] Enter:dla_handle_events, processor:BDMA
[  921.730064] Exit:dla_handle_events, ret:0
[  921.730313] Enter:dla_handle_events, processor:Convolution
[  921.730624] Handle cdma data done event, processor Convolution group 1
[  921.730982] Exit:dla_handle_events, ret:0
[  921.731216] Enter:dla_handle_events, processor:SDP
[  921.731508] Exit:dla_handle_events, ret:0
[  921.731753] Enter:dla_handle_events, processor:PDP
[  921.732690] Exit:dla_handle_events, ret:0
[  921.733298] Enter:dla_handle_events, processor:CDP
[  921.733856] Exit:dla_handle_events, ret:0
[  921.734257] Enter:dla_handle_events, processor:RUBIK
[  921.734780] Exit:dla_handle_events, ret:0
[  921.824017] Enter:dla_handle_events, processor:BDMA
[  921.824481] Exit:dla_handle_events, ret:0
[  921.824880] Enter:dla_handle_events, processor:Convolution
[  921.825223] Handle cdma weight done event, processor Convolution group 1
[  921.825657] Handle op complete event, processor Convolution group 1
[  921.826017] Enter:dla_op_completion processor Convolution group1
[  921.826408] Completed Convolution operation index 9 ROI 0
[  921.826794] 10 HWLs done, totally 11 layers
[  921.827124] Enter: dla_dequeue_operation
[  921.827427] exit Convolution as there's no further operation
[  921.829234] Exit: dla_dequeue_operation
[  921.829584] Exit:dla_op_completion processor Convolution group1 status=0
[  921.831171] Exit:dla_handle_events, ret:0
[  921.831595] Enter:dla_handle_events, processor:SDP
[  921.832001] Handle op complete event, processor SDP group 1
[  921.832935] Enter:dla_op_completion processor SDP group1
[  921.833356] Completed SDP operation index 10 ROI 0
[  921.833772] 11 HWLs done, totally 11 layers
[  921.834095] Enter: dla_free_op_desc op desc index 9 ROI 0
[  921.834850] Exit: dla_free_op_desc
[  921.835126] Enter: dla_free_op_desc op desc index 10 ROI 0
[  921.836225] Exit: dla_free_op_desc
[  921.836676] Exit:dla_op_completion processor SDP group1 status=0
[  921.837071] Exit:dla_handle_events, ret:0
[  921.837314] Enter:dla_handle_events, processor:PDP
[  921.837648] Exit:dla_handle_events, ret:0
[  921.837893] Enter:dla_handle_events, processor:CDP
[  921.838175] Exit:dla_handle_events, ret:0
[  921.838677] Enter:dla_handle_events, processor:RUBIK
[  921.839245] Exit:dla_handle_events, ret:0
[  921.846392] reset engine done
Work Found!
Work Done
Test pass
2.21729e-05 0.96875 0.000106931 0.00115108 0.0072403 0.00423813 5.74589e-05 0.0171356 1.49012e-06 0.00135899

However, when running on AWS FPGA: The sanity test works:

./nvdla_runtime --loadable kmd/CDP/CDP_L0_0_small_fbuf 
creating new runtime context...
Emulator starting
submitting tasks...
[   52.041893] Enter:dla_read_network_config
[   52.043159] Exit:dla_read_network_config status=0
[   52.043501] Enter: dla_initiate_processors
[   52.045698] Enter: dla_submit_operation
[   52.046013] Prepare CDP operation index 0 ROI 0 dep_count 0
[   52.046355] Enter: dla_prepare_operation
[   52.046843] processor:CDP group:1, rdma_group:1 available
[   52.047238] Enter: dla_read_config
[   52.056823] Exit: dla_read_config
[   52.057142] Exit: dla_prepare_operation status=0
[   52.057519] Enter: dla_program_operation
[   52.057790] Program CDP operation index 0 ROI 0 Group[1]
[   52.058285] Enter: dla_cdp_program
[   52.058447] Enter: processor_cdp_program
[   52.083128] Exit: processor_cdp_program
[   52.083173] Exit: dla_cdp_program
[   52.083484] no desc get due to index==-1
[   52.083969] no desc get due to index==-1
[   52.085321] no desc get due to index==-1
[   52.085611] no desc get due to index==-1
[   52.085860] no desc get due to index==-1
[   52.086105] no desc get due to index==-1
[   52.086362] no desc get due to index==-1
[   52.086624] Enter: dla_op_programmed
[   52.086990] Exit: dla_op_programmed
[   52.087233] Exit: dla_program_operation status=0
[   52.087561] Enter: dla_enable_operation
[   52.087876] Enable CDP operation index 0 ROI 0
[   52.090056] Enter: dla_cdp_enable
[   52.090436] Exit: dla_cdp_enable
[   52.090693] Enter: dla_op_enabled
[   52.090956] Exit: dla_op_enabled
[   52.091184] Exit: dla_enable_operation status=0
[   52.091488] Exit: dla_submit_operation
[   52.091848] Enter: dla_dequeue_operation
[   52.092149] exit CDP as there's no further operation
[   52.094140] Exit: dla_dequeue_operation
[   52.094426] Exit: dla_initiate_processors status=0
[   52.119887] Enter:dla_handle_events, processor:BDMA
[   52.120342] Exit:dla_handle_events, ret:0
[   52.120629] Enter:dla_handle_events, processor:Convolution
[   52.120956] Exit:dla_handle_events, ret:0
[   52.121203] Enter:dla_handle_events, processor:SDP
[   52.121499] Exit:dla_handle_events, ret:0
[   52.121750] Enter:dla_handle_events, processor:PDP
[   52.122041] Exit:dla_handle_events, ret:0
[   52.122293] Enter:dla_handle_events, processor:CDP
[   52.122606] Handle op complete event, processor CDP group 1
[   52.122994] Enter:dla_op_completion processor CDP group1
[   52.123333] Completed CDP operation index 0 ROI 0
[   52.123704] 1 HWLs done, totally 1 layers
[   52.124024] Enter: dla_free_op_desc op desc index 0 ROI 0
[   52.124820] Exit: dla_free_op_desc
[   52.125086] Exit:dla_op_completion processor CDP group1 status=0
[   52.125464] Exit:dla_handle_events, ret:0
[   52.125722] Enter:dla_handle_events, processor:RUBIK
[   52.126019] Exit:dla_handle_events, ret:0
[   52.129651] reset engine done
Shutdown signal received, exiting
Test pass

However, when running this:

./nvdla_runtime --loadable /mnt/sw/regression/flatbufs/kmd/NN/NN_L0_0_fbuf --i
mage /mnt/sw/regression/images/digits/seven.pgm 
creating new runtime context...
Emulator starting
ppgminfo 1 28 28
pgm2dimg 1 28 28 1 32 896 896
submitting tasks...
Work Found!
Work Done
[  119.544138] Enter:dla_read_network_config
[  119.545407] Exit:dla_read_network_config status=0
[  119.545737] Enter: dla_initiate_processors
[  119.546583] Enter: dla_submit_operation
[  119.546847] Prepare Convolution operation index 0 ROI 0 dep_count 1
[  119.547206] Enter: dla_prepare_operation
[  119.547620] processor:Convolution group:0, rdma_group:0 available
[  119.549943] Enter: dla_read_config
[  119.559254] Exit: dla_read_config
[  119.559558] Exit: dla_prepare_operation status=0
[  119.560866] Enter: dla_program_operation
[  119.561191] Program Convolution operation index 0 ROI 0 Group[0]
[  119.566659] no desc get due to index==-1
[  119.567750] no desc get due to index==-1
[  119.568959] no desc get due to index==-1
[  119.569232] no desc get due to index==-1
[  119.569489] no desc get due to index==-1
[  119.569748] Enter: dla_op_programmed
[  119.570086] Update dependency operation index 3 ROI 0 DEP_COUNT=3
[  119.570442] Update dependency operation index 1 ROI 0 DEP_COUNT=1
[  119.570801] enable SDP in dla_update_dependency as depdency are resolved
[  119.571192] Enter: dla_enable_operation
[  119.571457] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  119.573541] Exit: dla_enable_operation status=0
[  119.573909] Exit: dla_op_programmed
[  119.574148] Exit: dla_program_operation status=0
[  119.574450] Exit: dla_submit_operation
[  119.574802] Enter: dla_dequeue_operation
[  119.575094] Dequeue op from Convolution processor, index=3 ROI=0
[  119.575493] Enter: dla_submit_operation
[  119.575736] Prepare Convolution operation index 3 ROI 0 dep_count 2
[  119.577679] Enter: dla_prepare_operation
[  119.578053] processor:Convolution group:1, rdma_group:0 available
[  119.578410] Enter: dla_read_config
[  119.586814] Exit: dla_read_config
[  119.588296] Exit: dla_prepare_operation status=0
[  119.588566] Enter: dla_program_operation
[  119.588807] Program Convolution operation index 3 ROI 0 Group[1]
[  119.592088] no desc get due to index==-1
[  119.594101] no desc get due to index==-1
[  119.594345] no desc get due to index==-1
[  119.594577] no desc get due to index==-1
[  119.594810] no desc get due to index==-1
[  119.595043] Enter: dla_op_programmed
[  119.595277] Update dependency operation index 6 ROI 0 DEP_COUNT=3
[  119.595611] Update dependency operation index 4 ROI 0 DEP_COUNT=2
[  119.597352] Exit: dla_op_programmed
[  119.597610] Exit: dla_program_operation status=0
[  119.597879] Exit: dla_submit_operation
[  119.598140] Exit: dla_dequeue_operation
[  119.598403] Enter: dla_submit_operation
[  119.598644] Prepare SDP operation index 1 ROI 0 dep_count 0
[  119.598954] Enter: dla_prepare_operation
[  119.599312] processor:SDP group:0, rdma_group:0 available
[  119.599618] Enter: dla_read_config
[  119.609387] Exit: dla_read_config
[  119.609705] Exit: dla_prepare_operation status=0
[  119.610009] Enter: dla_program_operation
[  119.610256] Program SDP operation index 1 ROI 0 Group[0]
[  119.613814] no desc get due to index==-1
[  119.614099] no desc get due to index==-1
[  119.614710] no desc get due to index==-1
[  119.614950] no desc get due to index==-1
[  119.615206] Enter: dla_op_programmed
[  119.615436] Update dependency operation index 4 ROI 0 DEP_COUNT=1
[  119.615787] enable SDP in dla_update_dependency as depdency are resolved
[  119.617562] Enter: dla_enable_operation
[  119.617859] exit dla_enable_operation without actual enable due to processor hasn't been programmed
[  119.618337] Exit: dla_enable_operation status=0
[  119.618597] Exit: dla_op_programmed
[  119.618812] Exit: dla_program_operation status=0
[  119.619086] Enter: dla_enable_operation
[  119.619370] Enable SDP operation index 1 ROI 0
[  119.621260] Enter: dla_op_enabled
[  119.621541] Update dependency operation index 0 ROI 0 DEP_COUNT=1
[  119.621878] enable Convolution in dla_update_dependency as depdency are resolved
[  119.622291] Enter: dla_enable_operation
[  119.622529] Enable Convolution operation index 0 ROI 0
[  119.623333] Enter: dla_op_enabled
[  119.623567] Exit: dla_op_enabled
[  119.623784] Exit: dla_enable_operation status=0
[  119.625652] Exit: dla_op_enabled
[  119.625902] Exit: dla_enable_operation status=0
[  119.626179] Exit: dla_submit_operation
[  119.626414] Enter: dla_dequeue_operation
[  119.626658] Dequeue op from SDP processor, index=4 ROI=0
[  119.626961] Enter: dla_submit_operation
[  119.627199] Prepare SDP operation index 4 ROI 0 dep_count 0
[  119.627511] Enter: dla_prepare_operation
[  119.629923] processor:SDP group:1, rdma_group:1 available
[  119.630280] Enter: dla_read_config
[  119.638245] Exit: dla_read_config
[  119.639510] Exit: dla_prepare_operation status=0
[  119.639784] Enter: dla_program_operation
[  119.640954] Program SDP operation index 4 ROI 0 Group[1]
[  119.642659] no desc get due to index==-1
[  119.642904] no desc get due to index==-1
[  119.643906] no desc get due to index==-1
[  119.645281] no desc get due to index==-1
[  119.645526] Enter: dla_op_programmed
[  119.645754] Update dependency operation index 7 ROI 0 DEP_COUNT=2
[  119.646114] Exit: dla_op_programmed
[  119.646335] Exit: dla_program_operation status=0
[  119.646636] Enter: dla_enable_operation
[  119.646919] Enable SDP operation index 4 ROI 0
[  119.647353] Enter: dla_op_enabled
[  119.647571] Update dependency operation index 3 ROI 0 DEP_COUNT=2
[  119.649504] Exit: dla_op_enabled
[  119.649755] Exit: dla_enable_operation status=0
[  119.650023] Exit: dla_submit_operation
[  119.650258] Exit: dla_dequeue_operation
[  119.650493] Enter: dla_submit_operation
[  119.650759] Prepare PDP operation index 2 ROI 0 dep_count 1
[  119.651072] Enter: dla_prepare_operation
[  119.651413] processor:PDP group:0, rdma_group:0 available
[  119.651721] Enter: dla_read_config
[  119.661678] Exit: dla_read_config
[  119.661985] Exit: dla_prepare_operation status=0
[  119.662275] Enter: dla_program_operation
[  119.662524] Program PDP operation index 2 ROI 0 Group[0]
[  119.662848] group id 0 rdma id 0
[  119.666860] no desc get due to index==-1
[  119.667150] no desc get due to index==-1
[  119.667415] no desc get due to index==-1
[  119.667651] no desc get due to index==-1
[  119.669134] no desc get due to index==-1
[  119.669416] Enter: dla_op_programmed
[  119.669649] Update dependency operation index 5 ROI 0 DEP_COUNT=2
[  119.669986] Exit: dla_op_programmed
[  119.670209] Exit: dla_program_operation status=0
[  119.670485] Exit: dla_submit_operation
[  119.670718] Enter: dla_dequeue_operation
[  119.670959] Dequeue op from PDP processor, index=5 ROI=0
[  119.671273] Enter: dla_submit_operation
[  119.671537] Prepare PDP operation index 5 ROI 0 dep_count 1
[  119.673524] Enter: dla_prepare_operation
[  119.673910] processor:PDP group:1, rdma_group:1 available
[  119.674223] Enter: dla_read_config
[  119.682157] Exit: dla_read_config
[  119.683514] Exit: dla_prepare_operation status=0
[  119.683791] Enter: dla_program_operation
[  119.684979] Program PDP operation index 5 ROI 0 Group[1]
[  119.685326] group id 1 rdma id 1
[  119.687170] no desc get due to index==-1
[  119.687414] no desc get due to index==-1
[  119.687652] no desc get due to index==-1
[  119.689090] no desc get due to index==-1
[  119.689365] no desc get due to index==-1
[  119.689605] no desc get due to index==-1
[  119.689843] Enter: dla_op_programmed
[  119.690068] Exit: dla_op_programmed
[  119.690292] Exit: dla_program_operation status=0
[  119.690554] Exit: dla_submit_operation
[  119.690784] Exit: dla_dequeue_operation
[  119.691040] Exit: dla_initiate_processors status=0
[  119.691402] Enter:dla_handle_events, processor:BDMA
[  119.691762] Exit:dla_handle_events, ret:0
[  119.692107] Enter:dla_handle_events, processor:Convolution
[  119.692440] Handle cdma weight done event, processor Convolution group 0
[  119.692840] Exit:dla_handle_events, ret:0
[  119.693104] Enter:dla_handle_events, processor:SDP
[  119.693386] Exit:dla_handle_events, ret:0
[  119.693628] Enter:dla_handle_events, processor:PDP
[  119.693910] Exit:dla_handle_events, ret:0
[  119.694154] Enter:dla_handle_events, processor:CDP
[  119.694433] Exit:dla_handle_events, ret:0
[  119.694675] Enter:dla_handle_events, processor:RUBIK
[  119.694962] Exit:dla_handle_events, ret:0

It freezes after RUBIK every time. It should be followed by: BDMA, Convolution, SDP, PDP in that order judging by the run in Docker, however, it never reaches that stage. Meanwhile, it does not freeze when it encounters BDMA and Convolution prior to RUBIK event.

I am running this AFI: nv_small_256_full: agfi-0d9cf88f1002a1efb

[nagendra7890]

Hi, @fisherxue I was not able to see the execution time and test result. can you please give some suggestions? https://github.com/nvdla/sw/issues/177#issuecomment-917419962