Crashing with "illegal proof range"

multikatt commented 1 year ago

Current Behavior

I've had hostd crash twice, both times when i've been away from the server and thus not been interacting with it.

sia@host:~/hostd$ ./bin/hostd
2023/06/05 09:40:04 hostd %(describe:tags=true)
2023/06/05 09:40:04 Network Mainnet
Enter API password:
Enter wallet seed:
2023/06/05 09:40:30 api listening on: 127.0.0.1:9980
2023/06/05 09:40:30 p2p listening on: localhost:9981
2023/06/05 09:40:30 rhp2 listening on: [::]:9982
2023/06/05 09:40:30 rhp3 TCP listening on: [::]:9983
2023/06/05 09:40:30 rhp3 WebSocket listening on: [::]:9984
2023/06/05 09:40:30 host public key: ed25519:2a6a5a98bec1210e5fc8fb0f1050a58707480119a5c5ceb758b9bfec395d95d0
panic: BuildProof: illegal proof range

goroutine 5039864 [running]:
go.sia.tech/core/rhp/v2.BuildProof(0xd4b7ca70f0b79cb3?, 0x8f627ec3e9d97f7?, 0x1577785da0074625?, 0xab4041290521e8c6?)
        go.sia.tech/core@v0.1.12-0.20230529164041-6347a98003be/rhp/v2/merkle.go:262 +0x2a5
go.sia.tech/hostd/rhp/v3.(*programExecutor).executeReadSector(0xc0000c2e00, 0xc00533c400)
        go.sia.tech/hostd/rhp/v3/execute.go:263 +0x5ee
go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram.func1()
        go.sia.tech/hostd/rhp/v3/execute.go:531 +0x365
created by go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram
        go.sia.tech/hostd/rhp/v3/execute.go:503 +0xc5

2023/06/10 02:01:18 hostd %(describe:tags=true)
2023/06/10 02:01:18 Network Mainnet
Enter API password:
Enter wallet seed:
2023/06/10 02:01:55 api listening on: 127.0.0.1:9980
2023/06/10 02:01:55 p2p listening on: localhost:9981
2023/06/10 02:01:55 rhp2 listening on: [::]:9982
2023/06/10 02:01:55 rhp3 TCP listening on: [::]:9983
2023/06/10 02:01:55 rhp3 WebSocket listening on: [::]:9984
2023/06/10 02:01:55 host public key: ed25519:2a6a5a98bec1210e5fc8fb0f1050a58707480119a5c5ceb758b9bfec395d95d0
panic: BuildProof: illegal proof range

goroutine 190590 [running]:
go.sia.tech/core/rhp/v2.BuildProof(0xec97ecdb71183cb?, 0xa14a7756df49a436?, 0xd442b1cb1f820128?, 0x24d0dfd4f29afd1f?)
        go.sia.tech/core@v0.1.12-0.20230529164041-6347a98003be/rhp/v2/merkle.go:262 +0x2a5
go.sia.tech/hostd/rhp/v3.(*programExecutor).executeReadSector(0xc001bc2700, 0xc004da6460)
        go.sia.tech/hostd/rhp/v3/execute.go:263 +0x5ee
go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram.func1()
        go.sia.tech/hostd/rhp/v3/execute.go:531 +0x365
created by go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram
        go.sia.tech/hostd/rhp/v3/execute.go:503 +0xc5

Expected Behavior

-

Steps to Reproduce

No response

Version

Built from db8f71b4980885f8b6a1e7e7350d9febb86312cd

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Ubuntu 22.04

Anything else?

The build is a few days old, i've run a git pull and will run from e62eda3595bee9d9047868e115bdb374797b50ee now. I'll update if i see it happen again.

n8maninger commented 1 year ago

Thank you for the stack trace! Can you try 646b44805e565e522b81d7e5dbb4c898e4d3d002 and report if it happens again?

multikatt commented 1 year ago

Alright, i'll give it a try. I had no improvements with https://github.com/SiaFoundation/hostd/commit/e62eda3595bee9d9047868e115bdb374797b50ee, in fact i've had three crashes of the above type since updating. I've also had two of the following, but i dont know if they're related at all. I'll report back how https://github.com/SiaFoundation/hostd/commit/646b44805e565e522b81d7e5dbb4c898e4d3d002 works for me.

Jun 11 20:28:48 host hostd[264686]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 11 20:28:48 host hostd[264686]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0xbc829e]
Jun 11 20:28:48 host hostd[264686]: goroutine 2316 [running]:
Jun 11 20:28:48 host hostd[264686]: go.sia.tech/hostd/host/contracts.(*ContractUpdater).SectorRoot(...)
Jun 11 20:28:48 host hostd[264686]:         go.sia.tech/hostd/host/contracts/contracts.go:305
Jun 11 20:28:48 host hostd[264686]: go.sia.tech/hostd/rhp/v3.(*programExecutor).executeReadOffset(0xc000648000, 0xc00069a750)
Jun 11 20:28:48 host hostd[264686]:         go.sia.tech/hostd/rhp/v3/execute.go:207 +0x35e
Jun 11 20:28:48 host hostd[264686]: go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram.func1()
Jun 11 20:28:48 host hostd[264686]:         go.sia.tech/hostd/rhp/v3/execute.go:529 +0x33a
Jun 11 20:28:48 host hostd[264686]: created by go.sia.tech/hostd/rhp/v3.(*programExecutor).executeProgram
Jun 11 20:28:48 host hostd[264686]:         go.sia.tech/hostd/rhp/v3/execute.go:503 +0xc5

n8maninger commented 1 year ago

That's a different issue. Fixed in 885c3ac5fa24b93a71f2046a5b0b4ca8a6f3e312

multikatt commented 1 year ago

It's been 10 hours since i started running https://github.com/SiaFoundation/hostd/commit/e62eda3595bee9d9047868e115bdb374797b50ee and about six since https://github.com/SiaFoundation/hostd/commit/885c3ac5fa24b93a71f2046a5b0b4ca8a6f3e312. I haven't seen any new instances of the errors since i did their respective updates. So things look really good so far!

n8maninger commented 1 year ago

Great! I’ll leave this open for a few more days in case you have any other random crashes.

SiaFoundation / hostd