[BUG] interface conversion: interface {} is nil, not int64

[Cyberes]

After about a month of collecting data it becomes impossible to load the metrics due to a 500 internal server error.

This has been a long-running issue and affects on Docker as well. I also tried building from source using commit 4b1d9dc2d3f5388440a6746c3bdce2b8e2bee91e. The only way to fix this is to nuke the InfluxDB database.

Metrics seem to be collected fine (as seen in the "last updated" field next to each disk), so the issue must be with displaying them as an error is thrown when I click a drive to view its metrics.

Log Files

Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: 2023/10/02 16:04:21 [Recovery] 2023/10/02 - 16:04:21 panic recovered:
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: interface conversion: interface {} is nil, not int64
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /usr/local/go/src/runtime/iface.go:263 (0x40dba4)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]:         panicdottypeE: panic(&TypeAssertionError{iface, have, want, ""})
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/models/measurements/smart.go:69 (0x83b6ed)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/database/scrutiny_repository_device_smart_attributes.go:54 (0xc22ebb)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/web/handler/get_device_details.go:27 (0xdbdeca)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd821d3)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/recovery.go:83 (0xd821c1)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd77f0a)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/config.go:11 (0xdc4944)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdc33b9)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/repository.go:29 (0xdc33a7)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdc23b4)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/logger.go:56 (0xdc2399)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd81208)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:409 (0xd80e62)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:367 (0xd809b3)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /usr/local/go/src/net/http/server.go:2938 (0x77b6ad)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]:         serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /usr/local/go/src/net/http/server.go:2009 (0x777593)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]:         (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]: /usr/local/go/src/runtime/asm_amd64.s:1650 (0x46b0a0)
Oct 02 16:04:21 scrutiny scrutiny-web[1318491]:         goexit: BYTE        $0x90        // NOP

InfluxDB is created with:

docker run -d --restart=always -p 8086:8086 \
  --network="host" \
  -v /var/docker-data/influxdb2:/var/lib/influxdb2 \
  --name influxdb \
  influxdb:2.2

I run the web dashboard in an LXC container which all my hosts send their data to.

[Cyberes]

@AnalogJ I know you're busy with Fasten, but can you take a look at this issue? It's been a recurring problem and is probably a bug due to the traceback.

[AnalogJ]

hey @Cyberes I think I fixed this in the beta branch. Can you test the beta-omnibus docker image?

[Cyberes]

Great! I compiled the web binary from the beta branch but got this when loading /web/:

Oct 17 22:14:04 scrutiny scrutiny-web[1904255]: time="2023-10-17T22:14:04-06:00" level=info msg="172.0.1.1 - scrutiny [17/Oct/2023:22:14:04 -0600] \"GET /web/\" 200 7467 \"\" \"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0\" (1ms)" clientIP=xxx.xxx.xxx.xxx hostname=scrutiny latency=1 method=GET path=/web/ referer= respLength=7467 statusCode=200 type=web userAgent="Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0"
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: 2023/10/17 22:14:07 [Recovery] 2023/10/17 - 22:14:07 panic recovered:
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: runtime error: invalid memory address or nil pointer dereference
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /usr/local/go/src/runtime/panic.go:261 (0x44f0b7)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]:         panicmem: panic(memoryError)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /usr/local/go/src/runtime/signal_unix.go:861 (0x44f085)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]:         sigpanic: panicmem()
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/webapp/backend/pkg/database/scrutiny_repository.go:436 (0xc213fe)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/webapp/backend/pkg/web/handler/get_devices_summary.go:14 (0xdbe8de)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd82653)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/recovery.go:83 (0xd82641)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd7838a)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/config.go:11 (0xdc4dc4)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdc3839)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/repository.go:29 (0xdc3827)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdc2834)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/webapp/backend/pkg/web/middleware/logger.go:56 (0xdc2819)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xd81688)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:409 (0xd812e2)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /home/username/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:367 (0xd80e33)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /usr/local/go/src/net/http/server.go:2938 (0x77b6ad)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]:         serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /usr/local/go/src/net/http/server.go:2009 (0x777593)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]:         (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: /usr/local/go/src/runtime/asm_amd64.s:1650 (0x46b0a0)
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]:         goexit: BYTE        $0x90        // NOP
Oct 17 22:14:07 scrutiny scrutiny-web[1904255]: 

If you really want me to try it with Docker, let me know how I can use the beta scrutiny:master-web

[Cyberes]

@AnalogJ I'm currently in a situation where I need to view my hard drive metrics so I'd have to reset the InfluxDB database. I'm open to other solutions if there's another patch or workaround you'd like me to try.

[ovizii]

I'm getting this error:

scrutiny  |
scrutiny  |
scrutiny  | 2024/02/16 22:26:06 [Recovery] 2024/02/16 - 22:26:06 panic recovered:
scrutiny  | runtime error: invalid memory address or nil pointer dereference
scrutiny  | /usr/local/go/src/runtime/panic.go:260 (0x44d01c)
scrutiny  | /usr/local/go/src/runtime/signal_unix.go:841 (0x44cfec)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/database/scrutiny_repository.go:436 (0xc61e34)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/handler/get_devices_summary.go:14 (0xe0b988)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdcd0da)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/recovery.go:83 (0xdcd0c6)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe0f6be)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/config.go:11 (0xe0f6a5)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe10c1e)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/repository.go:29 (0xe10c05)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe0fc13)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/logger.go:56 (0xe0fbee)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdcc029)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:409 (0xdcbc77)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:367 (0xdcb793)
scrutiny  | /usr/local/go/src/net/http/server.go:2936 (0x7969d5)
scrutiny  | /usr/local/go/src/net/http/server.go:1995 (0x791ef1)
scrutiny  | /usr/local/go/src/runtime/asm_amd64.s:1598 (0x469280)
scrutiny  |

with:

services:
  scrutiny:
    image: ghcr.io/analogj/scrutiny:master-omnibus

Its been running for quite some time just fine, actually, I managed to access the web interface of scrutiny, delete an old HD, got asked if I was sure as I was going to lose historical data and then everything hang with this error. Restarting the container does no good.

[Cyberes]

Sounds like my issue.

[AnalogJ]

we need copies of the smartctl json output to run against our tests and determine which key is failing to de-serialize.

[Cyberes]

I re-created my database and it's been working for a couple months without any issues, so I don't have anything to give you right now. But if the error crops back up I'll certainly share the JSON.

As a side note, I replaced a few disks on a couple of my servers and I may have replaced the disk that was returning the bad JSON.

[ovizii]

I can make a copy of my scrutiny data by the weekend and then try to trigger the problem again by deleting an old HD from scrutiny's GUI. I'm just not sure where to find the "smartctl json output” you mentioned.

[ovizii]

OK, I made a copy of the InfluxDB and my scrutiny.db, went to the scrutiny GUI, deleted a device, refreshed and the GUI hangs.

I copied the docker logs and have removed my email and pushbullet secret and am happy to share it. Its quite long, I'll have to see if I can attach it here.

I can see the deletion in the logs, followed by a few other things happening in the background and then:

scrutiny  | 2024/03/20 16:11:45 [Recovery] 2024/03/20 - 16:11:45 panic recovered:
scrutiny  | runtime error: invalid memory address or nil pointer dereference
scrutiny  | /usr/local/go/src/runtime/panic.go:260 (0x44cffc)
scrutiny  | /usr/local/go/src/runtime/signal_unix.go:841 (0x44cfcc)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/database/scrutiny_repository.go:436 (0xc615d4)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/handler/get_devices_summary.go:14 (0xe0be68)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdcd07a)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/recovery.go:83 (0xdcd066)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe0fbfe)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/config.go:11 (0xe0fbe5)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe1115e)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/repository.go:29 (0xe11145)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xe10153)
scrutiny  | /go/src/github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware/logger.go:56 (0xe1012e)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/context.go:161 (0xdcbfc9)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:409 (0xdcbc17)
scrutiny  | /go/src/github.com/analogj/scrutiny/vendor/github.com/gin-gonic/gin/gin.go:367 (0xdcb733)
scrutiny  | /usr/local/go/src/net/http/server.go:2936 (0x7961d5)
scrutiny  | /usr/local/go/src/net/http/server.go:1995 (0x7916f1)
scrutiny  | /usr/local/go/src/runtime/asm_amd64.s:1598 (0x469260)

logs.log