NNPACK causing segmentation fault

[Shivankit99]

Im using NNPACK to speed up yolov3 on a raspberry pi 3b+. I have tried different repositories that give yolo with NNPACK(digitalbrain79,shizukachan) but all are giving a segmentation fault. I tried testing the model without NNPACK and it seemed to be working just fine. @Maratyszcza Do you think this is an issue with NNPACK or is it something else ?

[Maratyszcza]

Please run your binary under gdb, collect backtrace bt once the program segfaults, and post it here

[Shivankit99]

pi@raspberrypi:~/darknet-nnpack $ gdb --args ./darknet detector test cfg/coco.data cfg/yolov3-tiny.cfg yolov3-tiny.weights data/dog.jpg GNU gdb (Raspbian 8.2.1-2) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabihf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./darknet...done. (gdb) run Starting program: /home/pi/darknet-nnpack/darknet detector test cfg/coco.data cfg/yolov3-tiny.cfg yolov3-tiny.weights data/dog.jpg warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". OpenCV isn't used batch = 1, time_steps = 1, train = 0 layer filters size/strd(dil) input output 0 conv 16 3 x 3/ 1 416 x 416 x 3 -> 416 x 416 x 16 0.150 BF 1 max 2x 2/ 2 416 x 416 x 16 -> 208 x 208 x 16 0.003 BF 2 conv 32 3 x 3/ 1 208 x 208 x 16 -> 208 x 208 x 32 0.399 BF 3 max 2x 2/ 2 208 x 208 x 32 -> 104 x 104 x 32 0.001 BF 4 conv 64 3 x 3/ 1 104 x 104 x 32 -> 104 x 104 x 64 0.399 BF 5 max 2x 2/ 2 104 x 104 x 64 -> 52 x 52 x 64 0.001 BF 6 conv 128 3 x 3/ 1 52 x 52 x 64 -> 52 x 52 x 128 0.399 BF 7 max 2x 2/ 2 52 x 52 x 128 -> 26 x 26 x 128 0.000 BF 8 conv 256 3 x 3/ 1 26 x 26 x 128 -> 26 x 26 x 256 0.399 BF 9 max 2x 2/ 2 26 x 26 x 256 -> 13 x 13 x 256 0.000 BF 10 conv 512 3 x 3/ 1 13 x 13 x 256 -> 13 x 13 x 512 0.399 BF 11 max 2x 2/ 1 13 x 13 x 512 -> 13 x 13 x 512 0.000 BF 12 conv 1024 3 x 3/ 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 13 conv 256 1 x 1/ 1 13 x 13 x1024 -> 13 x 13 x 256 0.089 BF 14 conv 512 3 x 3/ 1 13 x 13 x 256 -> 13 x 13 x 512 0.399 BF 15 conv 255 1 x 1/ 1 13 x 13 x 512 -> 13 x 13 x 255 0.044 BF 16 yolo [yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00 17 route 13 -> 13 x 13 x 256 18 conv 128 1 x 1/ 1 13 x 13 x 256 -> 13 x 13 x 128 0.011 BF 19 upsample 2x 13 x 13 x 128 -> 26 x 26 x 128 20 route 19 8 -> 26 x 26 x 384 21 conv 256 3 x 3/ 1 26 x 26 x 384 -> 26 x 26 x 256 1.196 BF 22 conv 255 1 x 1/ 1 26 x 26 x 256 -> 26 x 26 x 255 0.088 BF 23 yolo [yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00 Total BFLOPS 5.571 avg_outputs = 341534 Loading weights from yolov3-tiny.weights... seen 64, trained: 32013 K-images (500 Kilo-batches_64) Done! Loaded 24 layers from weights-file [New Thread 0x7109c450 (LWP 4253)] [New Thread 0x7089b450 (LWP 4254)] [New Thread 0x7009a450 (LWP 4255)]

Thread 1 "darknet" received signal SIGSEGV, Segmentation fault. 0x76f38dc4 in compute_gemm_convolution_inference.isra () from /usr/local/lib/libnnpack.so (gdb) backtrace

0 0x76f38dc4 in compute_gemm_convolution_inference.isra ()

from /usr/local/lib/libnnpack.so

1 0x00000000 in ?? ()

Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)

@Maratyszcza Thank you for the replies.

[Shivankit99]

@Maratyszcza in case the cpu info is required . processor : 0 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4

processor : 1 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4

processor : 2 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4

processor : 3 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 38.40 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4

Hardware : BCM2835 Revision : a020d3 Serial : 00000000f2e13523 Model : Raspberry Pi 3 Model B Plus Rev 1.3

[Shivankit99]

https://github.com/digitalbrain79/darknet-nnpack @Maratyszcza The link of the repository im using for yolo implementation with NNPACK. Please let me know if your require additional information.

[Shivankit99]

@Maratyszcza Ive updated the software on the rpi as well . Any workarounds ?

[Maratyszcza]

Try to revert NNPACK to commit bda381b2e207230cab2b38be310a17831cfe384a and check if it solves the problem

[Shivankit99]

I tried reverting back, it did not solve the problem . Still getting the same with gdb and bt

[Shivankit99]

@Maratyszcza im getting multiple deprecated threadpool warnings during the make command, although the make is successful always . Could this be a reason for the issue ?

[Maratyszcza]

@Shivankit99 Warnings about the deprecated functions are not the reason for the crash. I ran out of ideas about the possible reason, and don't have time to investigate.