search

snowzach / doods

DOODS - Dedicated Open Object Detection Service
MIT License
306 stars 31 forks source link

panic: runtime error: index out of range [0] with length 0 #10

Closed r1cht3r closed 4 years ago

r1cht3r commented 4 years ago

Encountering the following error when trying to use a model built via AutoML Vision, it works when used with classify_image.py in the https://github.com/google-coral/tflite repo and I have no problem using the default detectors. I've also tried using the amd64, noavx, & latest tags but no luck. Happy to provide any further detail. Thanks!


INFO    detector/detector.go:78 Configured Detector     {"package": "detector", "name": "raccoon", "type": "tflite-edgetpu", "model": "models/raccoon_edgetpu.tflite", "labels": 2, "width": 224, "height": 224}
INFO    server/server.go:273    API Listening   {"package": "server", "address": ":8080", "tls": false}
INFO    server/server.go:137    HTTP Request    {"status": 200, "took": 0.001004527, "request": "/detectors", "method": "GET", "package": "server.request", "request-id": "doods-849c95966b-f2fh6/5IgL5zyRy4-000001", "remote": "10.42.0.1:42076"}
INFO    server/server.go:137    HTTP Request    {"status": 200, "took": 0.000548719, "request": "/detectors", "method": "GET", "package": "server.request", "request-id": "doods-849c95966b-f2fh6/5IgL5zyRy4-000002", "remote": "10.42.0.1:42114"}
INFO    server/server.go:137    HTTP Request    {"status": 200, "took": 0.00054322, "request": "/detectors", "method": "GET", "package": "server.request", "request-id": "doods-849c95966b-f2fh6/5IgL5zyRy4-000003", "remote": "10.42.0.1:42170"}
panic: runtime error: index out of range [0] with length 0

goroutine 98 [running]:
github.com/snowzach/doods/detector/tflite.(*detector).Detect(0xc0000ce2c0, 0xddcec0, 0xc00033cc90, 0xc000033740, 0x0, 0x0, 0x0)
        /build/detector/tflite/detector.go:267 +0x1b85
github.com/snowzach/doods/detector.(*Mux).Detect(0xc00000f260, 0xddcec0, 0xc00033cc90, 0xc000033740, 0xc00000f260, 0xc35401, 0x7f8bac5b64c8)
        /build/detector/detector.go:120 +0xb7
github.com/snowzach/doods/odrpc._Odrpc_Detect_Handler.func1(0xddcec0, 0xc00033cc90, 0xc89ba0, 0xc000033740, 0x13, 0xddcec0, 0xc00033cc90, 0x0)
        /build/odrpc/rpc.pb.go:855 +0x86
github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1(0xddcec0, 0xc00033cc90, 0xc89ba0, 0xc000033740, 0xc00013e0e0, 0xc00013e100, 0xafa19a, 0xc354e0, 0xc00013e120, 0xc00013e0e0)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/auth/auth.go:47 +0x108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xddcec0, 0xc00033cc90, 0xc89ba0, 0xc000033740, 0xc0002ccc00, 0x0, 0xc000362a10, 0x414868)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0xddcec0, 0xc00033cc90, 0xc89ba0, 0xc000033740, 0xc00013e0e0, 0xc00013e100, 0xc000362a80, 0x50d72d, 0xc4c000, 0xc00033cc90)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:34 +0xd5
github.com/snowzach/doods/odrpc._Odrpc_Detect_Handler(0xc14100, 0xc00000f260, 0xddcec0, 0xc00033cc90, 0xc00033a540, 0xc0001d4de0, 0xddcec0, 0xc00033cc90, 0xc0004ae000, 0x1db6d)
        /build/odrpc/rpc.pb.go:857 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001c0180, 0xde4100, 0xc0002ce510, 0xc0002ccc00, 0xc0001d51d0, 0x12b8458, 0x0, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:1007 +0x460
google.golang.org/grpc.(*Server).handleStream(0xc0001c0180, 0xde4100, 0xc0002ce510, 0xc0002ccc00, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:1287 +0xd97
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001b8420, 0xc0001c0180, 0xde4100, 0xc0002ce510, 0xc0002ccc00)
        /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:720 +0xa1```
snowzach commented 4 years ago

Pull the latest image. It should be fixed.

r1cht3r commented 4 years ago

I am past the initial errors but have a fresh set with latest: 

 INFO   detector/detector.go:78 Configured Detector {"package": "detector", "name": "default", "type": "tflite-edgetpu", "model": "models/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 300, "height": 300}
 INFO   detector/detector.go:78 Configured Detector {"package": "detector", "name": "raccoon", "type": "tflite-edgetpu", "model": "models/raccoon_edgetpu.tflite", "labels": 5, "width": 224, "height": 224}
 INFO   detector/detector.go:78 Configured Detector {"package": "detector", "name": "face", "type": "tflite-edgetpu", "model": "models/mobilenet_ssd_v2_face_quant_postprocess_edgetpu.tflite", "labels": 80, "width": 320, "height": 320}
 INFO   server/server.go:274    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.1.3-0-gb0fceac"}
 INFO   server/server.go:138    HTTP Request    {"status": 200, "took": 0.000495601, "request": "/detectors", "method": "GET", "package": "server.request", "request-id": "doods-bc97bc7b-qq4n2/TiIWhEktVc-000061", "remote": "10.42.0.1:48190"}
 INFO   tflite/detector.go:326  Detection Complete  {"package": "detector.tflite", "name": "default", "id": "", "duration": 0.011418942, "detections": 2, "device": {"Type":1,"Path":"/sys/bus/usb/devices/4-4"}}
 INFO   server/server.go:138    HTTP Request    {"status": 200, "took": 0.039045551, "request": "/detect", "method": "POST", "package": "server.request", "request-id": "doods-bc97bc7b-qq4n2/TiIWhEktVc-000062", "remote": "10.42.0.1:58944"}
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x411113]

goroutine 847 [running]:
runtime.throw(0xcb91d6, 0x5)
    /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc00042c340 sp=0xc00042c310 pc=0x436fc2
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:401 +0x3de fp=0xc00042c370 sp=0xc00042c340 pc=0x44c34e
runtime.(*itabTableType).find(0xc000020000, 0xbdd560, 0xb89280, 0x41421e)
    /usr/local/go/src/runtime/iface.go:105 +0x43 fp=0xc00042c378 sp=0xc00042c370 pc=0x411113
runtime.getitab(0xbdd560, 0xb89280, 0x415601, 0xc816e0)
    /usr/local/go/src/runtime/iface.go:54 +0x13b fp=0xc00042c3c8 sp=0xc00042c378 pc=0x410e8b
runtime.assertE2I2(0xbdd560, 0xb89280, 0x1332560, 0x0, 0xc0000f6448, 0x417e66)
    /usr/local/go/src/runtime/iface.go:487 +0x43 fp=0xc00042c3f8 sp=0xc00042c3c8 pc=0x4122e3
go.uber.org/zap.Any(0xcb7c6c, 0x2, 0xb89280, 0x1332560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/go.uber.org/zap@v1.12.0/field.go:220 +0x13f fp=0xc00042d058 sp=0xc00042c3f8 pc=0x7aa27f
go.uber.org/zap.(*SugaredLogger).sweetenFields(0xc0001b6188, 0xc00042d658, 0x7, 0x7, 0xc00027c0b0, 0xc0000f75c8, 0xc0001c8040)
    /go/pkg/mod/go.uber.org/zap@v1.12.0/sugar.go:272 +0x5d3 fp=0xc00042d2e8 sp=0xc00042d058 pc=0x7b1d43
go.uber.org/zap.(*SugaredLogger).log(0xc0001b6188, 0xc0000f7500, 0xcc2b29, 0x12, 0x0, 0x0, 0x0, 0xc0000f7658, 0x7, 0x7)
    /go/pkg/mod/go.uber.org/zap@v1.12.0/sugar.go:234 +0xd4 fp=0xc00042d338 sp=0xc00042d2e8 pc=0x7b1674
go.uber.org/zap.(*SugaredLogger).Infow(...)
    /go/pkg/mod/go.uber.org/zap@v1.12.0/sugar.go:179
github.com/snowzach/doods/detector/tflite.(*detector).Detect(0xc00027c370, 0xddcf00, 0xc000444750, 0xc0001823c0, 0x0, 0x0, 0x0)
    /build/detector/tflite/detector.go:326 +0xf8e fp=0xc00042d858 sp=0xc00042d338 pc=0xab27ee
github.com/snowzach/doods/detector.(*Mux).Detect(0xc0001b4960, 0xddcf00, 0xc000444750, 0xc0001823c0, 0xc0001b4960, 0xc35401, 0x7f8384586180)
    /build/detector/detector.go:120 +0xb7 fp=0xc00042d8a8 sp=0xc00042d858 pc=0xaba347
github.com/snowzach/doods/odrpc._Odrpc_Detect_Handler.func1(0xddcf00, 0xc000444750, 0xc89ba0, 0xc0001823c0, 0x13, 0xddcf00, 0xc000444750, 0x0)
    /build/odrpc/rpc.pb.go:855 +0x86 fp=0xc00042d8f0 sp=0xc00042d8a8 pc=0xa5d2a6
github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1(0xddcf00, 0xc000444750, 0xc89ba0, 0xc0001823c0, 0xc00013e340, 0xc00013e360, 0xafa219, 0xc354e0, 0xc00013e380, 0xc00013e340)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/auth/auth.go:47 +0x108 fp=0xc00042d950 sp=0xc00042d8f0 pc=0xafae08
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xddcf00, 0xc000444750, 0xc89ba0, 0xc0001823c0, 0xc0002f2500, 0x0, 0xc00033fa10, 0x414868)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63 fp=0xc00042d9b0 sp=0xc00042d950 pc=0xafa123
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0xddcf00, 0xc000444750, 0xc89ba0, 0xc0001823c0, 0xc00013e340, 0xc00013e360, 0xc00033fa80, 0x50d72d, 0xc4c000, 0xc000444750)
    /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:34 +0xd5 fp=0xc00042da20 sp=0xc00042d9b0 pc=0xafa305
github.com/snowzach/doods/odrpc._Odrpc_Detect_Handler(0xc14100, 0xc0001b4960, 0xddcf00, 0xc000444750, 0xc00013a540, 0xc0001af7a0, 0xddcf00, 0xc000444750, 0xc0004e2000, 0xba77)
    /build/odrpc/rpc.pb.go:857 +0x14b fp=0xc00042da90 sp=0xc00042da20 pc=0xa5275b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00008a480, 0xde4140, 0xc0001fc120, 0xc0002f2500, 0xc0001afb90, 0x12b8458, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:1007 +0x460 fp=0xc00042de18 sp=0xc00042da90 pc=0xa15240
google.golang.org/grpc.(*Server).handleStream(0xc00008a480, 0xde4140, 0xc0001fc120, 0xc0002f2500, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:1287 +0xd97 fp=0xc00042df48 sp=0xc00042de18 pc=0xa19167
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000314080, 0xc00008a480, 0xde4140, 0xc0001fc120, 0xc0002f2500)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:722 +0xbb fp=0xc00042dfb8 sp=0xc00042df48 pc=0xa263eb
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00042dfc0 sp=0xc00042dfb8 pc=0x466861
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:720 +0xa1

goroutine 1 [chan receive, 3 minutes]:
github.com/snowzach/doods/cmd.glob..func1(0x12c14e0, 0xc0001b41a0, 0x0, 0x2)
    /build/cmd/api.go:46 +0x236
github.com/spf13/cobra.(*Command).execute(0x12c14e0, 0xc0001b4180, 0x2, 0x2, 0x12c14e0, 0xc0001b4180)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x12c1760, 0xc0001ec500, 0xc00010ff20, 0x444bda)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/snowzach/doods/cmd.Execute()
    /build/cmd/root.go:57 +0x31
main.main()
    /build/main.go:8 +0x20

goroutine 6 [syscall, 3 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 18 [chan receive, 3 minutes]:
github.com/snowzach/doods/conf.init.1.func1()
    /build/conf/signal.go:42 +0x3e
created by github.com/snowzach/doods/conf.init.1
    /build/conf/signal.go:40 +0xcc

goroutine 35 [select, 3 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0001bc640)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/balancer_conn_wrappers.go:68 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/balancer_conn_wrappers.go:59 +0x16d

goroutine 36 [chan receive, 3 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0002b0000)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/clientconn.go:1163 +0x6ac
created by google.golang.org/grpc.(*addrConn).connect
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/clientconn.go:800 +0x120

goroutine 37 [chan receive (nil chan), 3 minutes]:
github.com/snowzach/doods/server/rpc.RegisterVersionRPCHandlerFromEndpoint.func1.1(0xddce80, 0xc000040018, 0xc000088a80, 0xc0001b9160, 0x9)
    /build/server/rpc/version.pb.gw.go:56 +0x4c
created by github.com/snowzach/doods/server/rpc.RegisterVersionRPCHandlerFromEndpoint.func1
    /build/server/rpc/version.pb.gw.go:55 +0x1b4

goroutine 38 [select, 3 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0001bc800)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/balancer_conn_wrappers.go:68 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/balancer_conn_wrappers.go:59 +0x16d

goroutine 39 [chan receive, 3 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0002b02c0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/clientconn.go:1163 +0x6ac
created by google.golang.org/grpc.(*addrConn).connect
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/clientconn.go:800 +0x120

goroutine 40 [chan receive (nil chan), 3 minutes]:
github.com/snowzach/doods/odrpc.RegisterOdrpcHandlerFromEndpoint.func1.1(0xddce80, 0xc000040018, 0xc000088e00, 0xc0001b91a0, 0x9)
    /build/odrpc/rpc.pb.gw.go:100 +0x4c
created by github.com/snowzach/doods/odrpc.RegisterOdrpcHandlerFromEndpoint.func1
    /build/odrpc/rpc.pb.gw.go:99 +0x1b4

goroutine 41 [IO wait]:
internal/poll.runtime_pollWait(0x7f838453eea8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0001d8898, 0x72, 0x0, 0x0, 0xcbaf43)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0001d8880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc0001d8880, 0xc000063d58, 0x77a084, 0xc0002920a0)
    /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00028e9a0, 0x5ddcb8d7, 0xc000063d58, 0x4c6a76)
    /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc00028e9a0, 0xc000063da8, 0x18, 0xc00008af00, 0x77959e)
    /usr/local/go/src/net/tcpsock.go:261 +0x47
net/http.(*Server).Serve(0xc000292000, 0xddaa40, 0xc00028e9a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2896 +0x286
github.com/snowzach/doods/server.(*Server).ListenAndServe.func3(0xc0001bc440, 0xddaa40, 0xc00028e9a0, 0xc0001b3680)
    /build/server/server.go:270 +0x5e
created by github.com/snowzach/doods/server.(*Server).ListenAndServe
    /build/server/server.go:269 +0x7b1

goroutine 23 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f838453edd8, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00012a298, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00012a280, 0xc0002f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00012a280, 0xc0002f4000, 0x8000, 0x8000, 0x0, 0x400000801, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00013c298, 0xc0002f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc000172660, 0xc00014e2d8, 0x9, 0x9, 0xc0002b8e5b, 0xe69279de, 0x18)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0xdc7920, 0xc000172660, 0xc00014e2d8, 0x9, 0x9, 0x9, 0xc0002b8e00, 0x414868, 0x20)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00014e2d8, 0x9, 0x9, 0xdc7920, 0xc000172660, 0x0, 0xc000000000, 0xbc1340, 0xc0001b9240)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00014e2a0, 0xc0001b9230, 0xc0001b9230, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000150380)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:1250 +0x183
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:293 +0xd23

goroutine 11 [IO wait]:
internal/poll.runtime_pollWait(0x7f838453ed08, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000100298, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000100280, 0xc0002d4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000100280, 0xc0002d4000, 0x8000, 0x8000, 0x0, 0x400000801, 0xc000000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010068, 0xc0002d4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc00007e960, 0xc0000f2118, 0x9, 0x9, 0xc0002a7af0, 0x7f838402d85d, 0x18)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0xdc7920, 0xc00007e960, 0xc0000f2118, 0x9, 0x9, 0x9, 0xc000067e00, 0x414868, 0x20)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000f2118, 0x9, 0x9, 0xdc7920, 0xc00007e960, 0x0, 0xc000000000, 0xbc1340, 0xc0002a7af0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000f20e0, 0xc0002a7ae0, 0xc0002a7ae0, 0x0, 0x0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0000fe1c0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:1250 +0x183
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:293 +0xd23

goroutine 24 [select, 3 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001386e0, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000172720, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000150380)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:337 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:335 +0xec0

goroutine 12 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000c87d0, 0x1, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00007ea20, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0000fe1c0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:337 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/http2_client.go:335 +0xec0

goroutine 50 [select, 3 minutes]:
golang.org/x/net/http2.(*serverConn).serve(0xc0002c0600)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:861 +0x588
golang.org/x/net/http2.(*Server).ServeConn(0xc0001bc4c0, 0xde2e20, 0xc0000a1c80, 0xc000068ba0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:472 +0x73a
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP(0xdc9e00, 0xc00028e7a0, 0xc0001bc4c0, 0xddad00, 0xc0000f21c0, 0xc0000b6400)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/h2c/h2c.go:87 +0x25d
net/http.serverHandler.ServeHTTP(0xc000292000, 0xddad00, 0xc0000f21c0, 0xc0000b6400)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0002d0000, 0xddce40, 0xc000032840)
    /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 51 [select]:
golang.org/x/net/http2.(*serverConn).serve(0xc0002ec480)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:861 +0x588
golang.org/x/net/http2.(*Server).ServeConn(0xc0001bc4c0, 0xde2e20, 0xc0001cc420, 0xc0002b5ba0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:472 +0x73a
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP(0xdc9e00, 0xc00028e7a0, 0xc0001bc4c0, 0xddad00, 0xc00030e000, 0xc0002f2000)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/h2c/h2c.go:87 +0x25d
net/http.serverHandler.ServeHTTP(0xc000292000, 0xddad00, 0xc00030e000, 0xc0002f2000)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0002d00a0, 0xddce40, 0xc0001c80c0)
    /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 66 [IO wait]:
internal/poll.runtime_pollWait(0x7f838453eb68, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002ce098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002ce080, 0xc0002f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002ce080, 0xc0002f0000, 0x1000, 0x1000, 0x3009, 0x2a85, 0x0)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001d2020, 0xc0002f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc0001cc330, 0xc0002f0000, 0x1000, 0x1000, 0x3009, 0x2a85, 0x8)
    /usr/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).Read(0xc0001ce1e0, 0xc00030e2d8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.(*multiReader).Read(0xc0001c60e0, 0xc00030e2d8, 0x9, 0x9, 0x1, 0xc0002b6d78, 0x4393c3)
    /usr/local/go/src/io/multi.go:26 +0xac
golang.org/x/net/http2/h2c.(*rwConn).Read(0xc0001cc420, 0xc00030e2d8, 0x9, 0x9, 0xc0002b6d78, 0xc0002b6d78, 0x44787c)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/h2c/h2c.go:287 +0x52
io.ReadAtLeast(0x7f8384f230b0, 0xc0001cc420, 0xc00030e2d8, 0x9, 0x9, 0x9, 0x1d89e0c3c3401, 0x0, 0x1)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00030e2d8, 0x9, 0x9, 0x7f8384f230b0, 0xc0001cc420, 0x0, 0xc000000000, 0xc0002b6f28, 0xc0002bc180)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00030e2a0, 0xc0002b6ee0, 0x2, 0x0, 0x1)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*serverConn).readFrames(0xc0002ec480)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:748 +0xa4
created by golang.org/x/net/http2.(*serverConn).serve
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:853 +0x347

goroutine 16 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f838453ec38, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002ce018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002ce000, 0xc000310000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002ce000, 0xc000310000, 0x1000, 0x1000, 0x9, 0xc000068bb8, 0x481e3c)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001d2018, 0xc000310000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc0000a1b90, 0xc000310000, 0x1000, 0x1000, 0x9, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).Read(0xc00007ea80, 0xc0000f2498, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.(*multiReader).Read(0xc00000ea80, 0xc0000f2498, 0x9, 0x9, 0x0, 0xc000068d48, 0x463ce7)
    /usr/local/go/src/io/multi.go:26 +0xac
golang.org/x/net/http2/h2c.(*rwConn).Read(0xc0000a1c80, 0xc0000f2498, 0x9, 0x9, 0xc000068d78, 0xc000068d78, 0x44787c)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/h2c/h2c.go:287 +0x52
io.ReadAtLeast(0x7f8384f230b0, 0xc0000a1c80, 0xc0000f2498, 0x9, 0x9, 0x9, 0x100000099ba78, 0x0, 0x1)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000f2498, 0x9, 0x9, 0x7f8384f230b0, 0xc0000a1c80, 0x0, 0xc000000000, 0xc000068f28, 0xc0000969c0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000f2460, 0xc000068ee0, 0x2, 0x0, 0x1)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*serverConn).readFrames(0xc0002c0600)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:748 +0xa4
created by golang.org/x/net/http2.(*serverConn).serve
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:853 +0x347

goroutine 844 [select]:
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).runStream(0xc0001fc120)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/handler_server.go:394 +0xbf
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).HandleStreams(0xc0001fc120, 0xc0004444e0, 0xcf1010)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/handler_server.go:384 +0x52a
google.golang.org/grpc.(*Server).serveStreams(0xc00008a480, 0xde4140, 0xc0001fc120)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:718 +0xf3
google.golang.org/grpc.(*Server).ServeHTTP(0xc00008a480, 0xdd9f80, 0xc000010028, 0xc0002f2300)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/server.go:770 +0x13f
github.com/snowzach/doods/server.New.func3(0xdd9f80, 0xc000010028, 0xc0002f2300)
    /build/server/server.go:173 +0x5d
net/http.HandlerFunc.ServeHTTP(0xc00028e7a0, 0xdd9f80, 0xc000010028, 0xc0002f2300)
    /usr/local/go/src/net/http/server.go:2007 +0x44
golang.org/x/net/http2.(*serverConn).runHandler(0xc0002ec480, 0xc000010028, 0xc0002f2300, 0xc00013e180)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:2149 +0x9f
created by golang.org/x/net/http2.(*serverConn).processHeaders
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/server.go:1883 +0x4eb

goroutine 856 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0000b6900)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/transport.go:317 +0xcc
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/transport.go:332
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0002ce680, 0xc87020, 0xc0001bc9c0, 0x0, 0xc0004c8000, 0xba77)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/stream.go:871 +0x753
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0002ce680, 0xba77, 0xba77)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/stream.go:736 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0000b8000, 0xc000428948, 0xc000428918, 0xc0004c8000, 0xc000342148)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/stream.go:594 +0x9c
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0000b8000, 0xc87020, 0xc0001bc9c0, 0x0, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/stream.go:735 +0x103
google.golang.org/grpc.invoke(0xddcf00, 0xc000344600, 0xcc372d, 0x13, 0xc89ba0, 0xc0001bc980, 0xc87020, 0xc0001bc9c0, 0xc000088e00, 0xc0001c6220, ...)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/call.go:73 +0x13b
google.golang.org/grpc.(*ClientConn).Invoke(0xc000088e00, 0xddcf00, 0xc000344600, 0xcc372d, 0x13, 0xc89ba0, 0xc0001bc980, 0xc87020, 0xc0001bc9c0, 0xc0001c6220, ...)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/call.go:37 +0x1b3
github.com/snowzach/doods/odrpc.(*odrpcClient).Detect(0xc0001b62b0, 0xddcf00, 0xc000344600, 0xc0001bc980, 0xc0001c6220, 0x2, 0x2, 0xc0001c6180, 0xddcf00, 0xc000344600)
    /build/odrpc/rpc.pb.go:772 +0xcf
github.com/snowzach/doods/odrpc.request_Odrpc_Detect_0(0xddcf00, 0xc000344600, 0xddecc0, 0x1331370, 0xdd9580, 0xc0001b62b0, 0xc0000b6800, 0xc000344540, 0x6, 0x1331301, ...)
    /build/odrpc/rpc.pb.gw.go:49 +0x272
github.com/snowzach/doods/odrpc.RegisterOdrpcHandlerClient.func2(0x7f8384f23130, 0xc0001bc7c0, 0xc0000b6800, 0xc000344540)
    /build/odrpc/rpc.pb.gw.go:170 +0x220
github.com/grpc-ecosystem/grpc-gateway/runtime.(*ServeMux).ServeHTTP(0xc000201650, 0x7f8384f23130, 0xc0001bc7c0, 0xc0000b6800)
    /go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway@v1.12.1/runtime/mux.go:243 +0xbd9
github.com/snowzach/doods/server.(*Server).ListenAndServe.func2(0x7f8384f23130, 0xc0001bc7c0, 0xc0000b6800)
    /build/server/server.go:258 +0x48
net/http.HandlerFunc.ServeHTTP(...)
    /usr/local/go/src/net/http/server.go:2007
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00013b2c0, 0x7f8384f23130, 0xc0001bc7c0, 0xc0000b6800)
    /go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:431 +0x23f
net/http.HandlerFunc.ServeHTTP(0xc0001b35d0, 0x7f8384f23130, 0xc0001bc7c0, 0xc0000b6800)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/snowzach/doods/server.New.func2.1(0xddad00, 0xc0000f2540, 0xc0000b6800)
    /build/server/server.go:113 +0x263
net/http.HandlerFunc.ServeHTTP(0xc00028e7c0, 0xddad00, 0xc0000f2540, 0xc0000b6800)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/go-chi/render.SetContentType.func1.1(0xddad00, 0xc0000f2540, 0xc0000b6700)
    /go/pkg/mod/github.com/go-chi/render@v1.0.1/content_type.go:52 +0x1a0
net/http.HandlerFunc.ServeHTTP(0xc00028e7e0, 0xddad00, 0xc0000f2540, 0xc0000b6700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/go-chi/chi/middleware.Recoverer.func1(0xddad00, 0xc0000f2540, 0xc0000b6700)
    /go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/middleware/recoverer.go:35 +0xbb
net/http.HandlerFunc.ServeHTTP(0xc00028e800, 0xddad00, 0xc0000f2540, 0xc0000b6700)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/go-chi/chi/middleware.RequestID.func1(0xddad00, 0xc0000f2540, 0xc0000b6600)
    /go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/middleware/request_id.go:72 +0x1dc
net/http.HandlerFunc.ServeHTTP(0xc00028e820, 0xddad00, 0xc0000f2540, 0xc0000b6600)
    /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00013b2c0, 0xddad00, 0xc0000f2540, 0xc0000b6300)
    /go/pkg/mod/github.com/go-chi/chi@v4.0.2+incompatible/mux.go:82 +0x2b2
github.com/snowzach/doods/server.New.func3(0xddad00, 0xc0000f2540, 0xc0000b6300)
    /build/server/server.go:175 +0x95
net/http.HandlerFunc.ServeHTTP(0xc00028e7a0, 0xddad00, 0xc0000f2540, 0xc0000b6300)
    /usr/local/go/src/net/http/server.go:2007 +0x44
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP(0xdc9e00, 0xc00028e7a0, 0xc0001bc4c0, 0xddad00, 0xc0000f2540, 0xc0000b6300)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20191105084925-a882066a44e0/http2/h2c/h2c.go:98 +0x44b
net/http.serverHandler.ServeHTTP(0xc000292000, 0xddad00, 0xc0000f2540, 0xc0000b6300)
    /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0000b43c0, 0xddce40, 0xc0001bc1c0)
    /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 857 [IO wait]:
internal/poll.runtime_pollWait(0x7f838453ea98, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0002ce418, 0x72, 0x0, 0x1, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002ce400, 0xc000344281, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0002ce400, 0xc000344281, 0x1, 0x1, 0x1, 0xc00043bfa8, 0x9a4deb)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001d2030, 0xc000344281, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).backgroundRead(0xc000344270)
    /usr/local/go/src/net/http/server.go:677 +0x58
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go/src/net/http/server.go:673 +0xd4

goroutine 845 [select]:
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).HandleStreams.func1(0xc000202180, 0xc0001fc120, 0xc0001b2180)
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/handler_server.go:312 +0xfc
created by google.golang.org/grpc/internal/transport.(*serverHandlerTransport).HandleStreams
    /go/pkg/mod/google.golang.org/grpc@v1.25.0/internal/transport/handler_server.go:311 +0xef
jasonramey commented 4 years ago

Any further detail I can provide here? Thanks.

snowzach commented 4 years ago

I will try to take a look tonight.

snowzach commented 4 years ago

Okay, see if you can pull and give it a shot.

r1cht3r commented 4 years ago

Still errors in a similar way. I've tried changing up the source image, verifying the default detectors work with it, and changing the detection data to the health-check you've mentioned in other issues. Same segfaults. doods.log

snowzach commented 4 years ago

@r1cht3r what kernel version are you running on the host?

r1cht3r commented 4 years ago

5.2.11-arch1-1-ARCH (amd64/intel) initially, I've also tested on 4.19.75-v7l+ (arm/raspberry pi 4) as well.

snowzach commented 4 years ago

Okay, I just found a bug report relating to kernel 5.2 and this kind of error which is why I asked. If it's getting the exact same crash on raspberry Pi as well, then I suspect it has something to do with your model. I think it's the racoons model. Did you train that one? Are you able to share it so I can try to reproduce?

r1cht3r commented 4 years ago

Sure thing, I'm using the image set from here https://github.com/datitran/raccoon_dataset though I trained the model from scratch (tagged images by hand) and added a few other labels (skunk/dog/person/cat), this was all generated in AutoML Vision on GCP and the tflite/labels files were created there. I've tried both single-label/multi-label classification models. Thanks! raccoon_labels.txt raccoon_edgetpu.tflite.zip

snowzach commented 4 years ago

I was able to spend a little time on this last night and I squished a couple issues that were absolutely on my side. I believe those to all be fixed in the library now.

At this point, I honestly believe this is a bug somewhere in the edgetpu library somewhere or in the tensorflow compatibility with the edgetpu library.

When I use any of the original models from the edgetpu site https://coral.ai/models/ it runs fine, no issues and chugs along for days at a time without errors.

I saw and downloaded a new model I found for the pixel 4 here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md I grabbed the ssd_mobilenet_edgetpu_coco model. When I run that continuously, about 10% of the time it returns success but the detection count number is garbage. It now returns an empty detection set and an error in the result.

When I use your raccoon model, it works occasionally but most of the time segfaults inside of the C code somewhere. If I get time, I may try to replicate it just in C code and see if I can produce something I can hand off to the tensorflow developers or the edgetpu guys.

I assume this model works okay with python code. I'll keep trying to figure out what's wrong as I have time.

r1cht3r commented 4 years ago

Yep, works fine with python. Thank you for your diligence!

snowzach commented 4 years ago

Hey @r1cht3r the issue is that the racoon model you provided only has a score and no bounding boxes returned. This is not (yet) compatible with Doods. I will work on adding this. As well, the other model I was using has a totally different output format also which is why it was also having issues. I am going to close this an open another issue adding support for different output formats.

snowzach commented 4 years ago

See #12