search

SiaFoundation / renterd

A renter for Sia
https://sia.tech/software/renterd
MIT License
68 stars 20 forks source link

"VerifySectorRangeProof: number of roots does not match range" after upgrade to cf362bb #897

Closed artur9010 closed 10 months ago

artur9010 commented 10 months ago

Current Behavior

After upgrading and restarting renterd i saw some VerifySectorRangeProof errors in worker logs

Using RENTERD_BUS_REMOTE_ADDR environment variable
Using RENTERD_BUS_API_PASSWORD environment variable
Using RENTERD_WORKER_API_PASSWORD environment variable
Using RENTERD_WORKER_ENABLED environment variable
Using RENTERD_WORKER_ID environment variable
Using RENTERD_AUTOPILOT_ENABLED environment variable
2024-01-12T13:03:05Z    INFO    renterd/main.go:427 renterd {"version": "cf362bb", "network": "Zen Testnet", "commit": "cf362bb", "buildDate": "2024-01-12T12:35:31Z"}
2024-01-12T13:03:05Z    INFO    renterd/main.go:492 connecting to remote bus at http://renterd-bus:9980/api/bus
2024-01-12T13:03:05Z    INFO    renterd/main.go:598 api: Listening on [::]:9980
2024-01-12T13:03:05Z    INFO    renterd/main.go:602 s3: Listening on [::]:8080
2024-01-12T13:03:05Z    INFO    renterd/main.go:609 bus: Listening on localhost:9981
2024-01-12T13:12:42Z    INFO    s3  s3/s3.go:62 LIST BUCKET
2024-01-12T13:12:42Z    INFO    s3  s3/s3.go:62 bucketname:juicefsprefix:prefix:"myjfs-zstd/meta/"page:{Marker: HasMarker:true MaxKeys:1000}
2024-01-12T13:12:43Z    INFO    s3  s3/s3.go:62 CREATE OBJECT:juicefsmyjfs-zstd/meta/dump-2024-01-12-131242.json.gz
http: panic serving 10.42.2.108:55296: VerifySectorRangeProof: number of roots does not match range
goroutine 4801 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xf46bc0?, 0x19f0760?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
go.sia.tech/core/rhp/v2.VerifySectorRangeProof({0x20e31a0, 0x0, 0x0}, {0xc000910000?, 0x20?, 0x20?}, 0xc0004d8490?, 0xbcafd525ad0b066?, 0x40?, {0x5, ...})
    /go/pkg/mod/go.sia.tech/core@v0.1.12-0.20231211182757-77190f04f90b/rhp/v2/merkle.go:395 +0x30a
go.sia.tech/renterd/worker.(*worker).fetchContractRoots(_, _, _, {0x1, 0x1400000, 0x6540, 0x1400000, {0xc000498498, 0x12}, 0x37e1d800000, ...})
    /renterd/worker/rhpv2.go:601 +0xbfe
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1.1(_, {{{0x24, 0xc3, 0x8a, 0xa1, 0xfa, 0xb, 0xf6, 0x2e, 0x70, ...}, ...}, ...}, ...)
    /renterd/worker/rhpv2.go:291 +0x2f0
go.sia.tech/renterd/worker.(*worker).withRevisionV2(0xc0002746c0?, {0x19f2400?, 0x19fe938?}, 0x0?, 0xc0001b2000, {0x95, 0x77, 0xd, 0x3b, 0x9b, ...}, ...)
    /renterd/worker/rhpv2.go:695 +0x742
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1(0x22ef09b3b0d7795?)
    /renterd/worker/rhpv2.go:280 +0xdc
go.sia.tech/renterd/worker.(*worker).withTransportV2(0x19fa900?, {0x19fa8c8?, 0xc0003876b0}, {0x95, 0x77, 0xd, 0x3b, 0x9b, 0xf0, 0x2e, ...}, ...)
    /renterd/worker/rhpv2.go:639 +0x26f
go.sia.tech/renterd/worker.(*worker).PruneContract.func1()
    /renterd/worker/rhpv2.go:279 +0xdd
go.sia.tech/renterd/worker.(*worker).withContractLock(0xc0000f3d40, {0x19fa8c8?, 0xc0003876b0?}, {0x24, 0xc3, 0x8a, 0xa1, 0xfa, 0xb, 0xf6, ...}, ...)
    /renterd/worker/contract_lock.go:73 +0xcb
go.sia.tech/renterd/worker.(*worker).PruneContract(0x10affbf?, {0x19fa8c8?, 0xc0003876b0?}, {0xc00050a1e0?, 0xb79acdd?}, {0x95, 0x77, 0xd, 0x3b, 0x9b, ...}, ...)
    /renterd/worker/rhpv2.go:278 +0x125
go.sia.tech/renterd/worker.(*worker).rhpPruneContractHandlerPOST(0xc0000f3d40, {{0x19f8e70, 0xc0000d8620}, 0xc000351b00, {0xc0001181a0, 0x1, 0x1}})
    /renterd/worker/worker.go:546 +0x84b
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.1({0x19f8e70, 0xc0000d8620}, 0xc000351b00)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:202 +0xba
net/http.HandlerFunc.ServeHTTP(0x19fa8c8?, {0x19f8e70?, 0xc0000d8620?}, 0x19ef750?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.func1.1({0x19f8e70, 0xc0000d8620}, 0xc000351a00)
    /renterd/worker/interactions.go:31 +0x151
net/http.HandlerFunc.ServeHTTP(0xc000351900?, {0x19f8e70?, 0xc0000d8620?}, 0x20e31a0?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.2({{0x19f8e70, 0xc0000d8620}, 0xc000351900, {0xc0001181a0, 0x1, 0x1}})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:205 +0xe2
go.sia.tech/jape.Mux.adaptor.func2({0x19f8e70?, 0xc0000d8620?}, 0xc0000dc700?, {0xc0001181a0?, 0x0?, 0xc0003d97d0?})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:164 +0x64
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001a8f00, {0x19f8e70, 0xc0000d8620}, 0xc000351900)
    /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb
go.sia.tech/jape.BasicAuth.func1.1({0x19f8e70, 0xc0000d8620}, 0x19f3060?)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:219 +0x99
net/http.HandlerFunc.ServeHTTP(0x19f3060?, {0x19f8e70?, 0xc0000d8620?}, 0xc0000f7a70?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.main.main.workerAuth.func9.func10({0x19f8e70, 0xc0000d8620}, 0xc000351900)
    /renterd/cmd/renterd/main.go:713 +0xf2
net/http.HandlerFunc.ServeHTTP(0xc0003d99a0?, {0x19f8e70?, 0xc0000d8620?}, 0xc0003d99b8?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.treeMux.ServeHTTP({{0x19f20c0?, 0xc000277860?}, 0x0?}, {0x19f8e70, 0xc0000d8620}, 0xc000351900)
    /renterd/cmd/renterd/web.go:28 +0x27a
main.treeMux.ServeHTTP({{0x0?, 0x0?}, 0xc000277290?}, {0x19f8e70, 0xc0000d8620}, 0xc000351900)
    /renterd/cmd/renterd/web.go:23 +0x249
net/http.serverHandler.ServeHTTP({0x19f61e8?}, {0x19f8e70?, 0xc0000d8620?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0000f73b0, {0x19fa8c8, 0xc0001100f0})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 6
    /usr/local/go/src/net/http/server.go:3086 +0x5cb
http: panic serving 10.42.2.108:55300: VerifySectorRangeProof: number of roots does not match range
goroutine 4866 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xf46bc0?, 0x19f0760?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
go.sia.tech/core/rhp/v2.VerifySectorRangeProof({0x20e31a0, 0x0, 0x0}, {0xc00096a000?, 0x20?, 0x20?}, 0xc0002ca490?, 0x8694549ffd7bbcc?, 0x40?, {0x48, ...})
    /go/pkg/mod/go.sia.tech/core@v0.1.12-0.20231211182757-77190f04f90b/rhp/v2/merkle.go:395 +0x30a
go.sia.tech/renterd/worker.(*worker).fetchContractRoots(_, _, _, {0x1, 0x1400000, 0x6540, 0x1400000, {0xc000700780, 0x1d}, 0x1e317800000, ...})
    /renterd/worker/rhpv2.go:601 +0xbfe
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1.1(_, {{{0x87, 0x9e, 0xa8, 0xa0, 0xd0, 0x3e, 0xed, 0xf2, 0xa9, ...}, ...}, ...}, ...)
    /renterd/worker/rhpv2.go:291 +0x2f0
go.sia.tech/renterd/worker.(*worker).withRevisionV2(0xc000700660?, {0x19f2400?, 0x19fe938?}, 0x0?, 0xc0001b20f0, {0x70, 0x7d, 0x18, 0x9d, 0xfd, ...}, ...)
    /renterd/worker/rhpv2.go:695 +0x742
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1(0x9e4608fd9d187d70?)
    /renterd/worker/rhpv2.go:280 +0xdc
go.sia.tech/renterd/worker.(*worker).withTransportV2(0x19fa900?, {0x19fa8c8?, 0xc00033a6c0}, {0x70, 0x7d, 0x18, 0x9d, 0xfd, 0x8, 0x46, ...}, ...)
    /renterd/worker/rhpv2.go:639 +0x26f
go.sia.tech/renterd/worker.(*worker).PruneContract.func1()
    /renterd/worker/rhpv2.go:279 +0xdd
go.sia.tech/renterd/worker.(*worker).withContractLock(0xc0000f3d40, {0x19fa8c8?, 0xc00033a6c0?}, {0x87, 0x9e, 0xa8, 0xa0, 0xd0, 0x3e, 0xed, ...}, ...)
    /renterd/worker/contract_lock.go:73 +0xcb
go.sia.tech/renterd/worker.(*worker).PruneContract(0x10affbf?, {0x19fa8c8?, 0xc00033a6c0?}, {0xc0007000a0?, 0xb79acdd?}, {0x70, 0x7d, 0x18, 0x9d, 0xfd, ...}, ...)
    /renterd/worker/rhpv2.go:278 +0x125
go.sia.tech/renterd/worker.(*worker).rhpPruneContractHandlerPOST(0xc0000f3d40, {{0x19f8e70, 0xc0000f80e0}, 0xc000400200, {0xc00023c020, 0x1, 0x1}})
    /renterd/worker/worker.go:546 +0x84b
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.1({0x19f8e70, 0xc0000f80e0}, 0xc000400200)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:202 +0xba
net/http.HandlerFunc.ServeHTTP(0x19fa8c8?, {0x19f8e70?, 0xc0000f80e0?}, 0x19ef750?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.func1.1({0x19f8e70, 0xc0000f80e0}, 0xc000400100)
    /renterd/worker/interactions.go:31 +0x151
net/http.HandlerFunc.ServeHTTP(0xc000400000?, {0x19f8e70?, 0xc0000f80e0?}, 0x20e31a0?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.2({{0x19f8e70, 0xc0000f80e0}, 0xc000400000, {0xc00023c020, 0x1, 0x1}})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:205 +0xe2
go.sia.tech/jape.Mux.adaptor.func2({0x19f8e70?, 0xc0000f80e0?}, 0xc0000c2200?, {0xc00023c020?, 0x0?, 0xc0003d97d0?})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:164 +0x64
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001a8f00, {0x19f8e70, 0xc0000f80e0}, 0xc000400000)
    /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb
go.sia.tech/jape.BasicAuth.func1.1({0x19f8e70, 0xc0000f80e0}, 0x19f3060?)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:219 +0x99
net/http.HandlerFunc.ServeHTTP(0x19f3060?, {0x19f8e70?, 0xc0000f80e0?}, 0x20d7d88?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.main.main.workerAuth.func9.func10({0x19f8e70, 0xc0000f80e0}, 0xc000400000)
    /renterd/cmd/renterd/main.go:713 +0xf2
net/http.HandlerFunc.ServeHTTP(0xc0003d9928?, {0x19f8e70?, 0xc0000f80e0?}, 0xc000623400?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.treeMux.ServeHTTP({{0x19f20c0?, 0xc000277860?}, 0x0?}, {0x19f8e70, 0xc0000f80e0}, 0xc000400000)
    /renterd/cmd/renterd/web.go:28 +0x27a
main.treeMux.ServeHTTP({{0x0?, 0x0?}, 0xc000277290?}, {0x19f8e70, 0xc0000f80e0}, 0xc000400000)
    /renterd/cmd/renterd/web.go:23 +0x249
net/http.serverHandler.ServeHTTP({0x19f61e8?}, {0x19f8e70?, 0xc0000f80e0?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0000f7440, {0x19fa8c8, 0xc0001100f0})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 6
    /usr/local/go/src/net/http/server.go:3086 +0x5cb
2024-01-12T13:36:19Z    DEBUG   worker.renterd-worker-2 worker/rhpv2.go:336 deleting 67524 contract roots (263.8 GiB)   {"hk": "ed25519:eb51aa3633a87d04acf7a8446a1f842e45db88b37e31e6b353fde74c141576ee", "fcid": "fcid:df429e7fa0458def0a0f62fb1f0a2de741c3374e6fbe3497ec4900b476943ef8"}
2024-01-12T13:36:19Z    DEBUG   worker.renterd-worker-2 worker/rhpv2.go:377 starting batch 1/1 of size 67524
2024-01-12T13:36:25Z    DEBUG   worker.renterd-worker-2 worker/rhpv2.go:379 processing batch 1/1 of size 67524 took 5.343201041s    {"cost": "~2.213 mS"}
http: panic serving 10.42.2.108:42598: VerifySectorRangeProof: number of roots does not match range
goroutine 4897 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xf46bc0?, 0x19f0760?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
go.sia.tech/core/rhp/v2.VerifySectorRangeProof({0x20e31a0, 0x0, 0x0}, {0xc000438000?, 0x20?, 0x20?}, 0xc000434490?, 0x73f2a4f5902c1e1?, 0x40?, {0x87, ...})
    /go/pkg/mod/go.sia.tech/core@v0.1.12-0.20231211182757-77190f04f90b/rhp/v2/merkle.go:395 +0x30a
go.sia.tech/renterd/worker.(*worker).fetchContractRoots(_, _, _, {0x1, 0x1400000, 0x6540, 0x1400000, {0xc0002748c0, 0x1b}, 0x13fb9d000000, ...})
    /renterd/worker/rhpv2.go:601 +0xbfe
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1.1(_, {{{0x12, 0xe3, 0x51, 0xdb, 0xd5, 0x73, 0x9, 0xab, 0x2a, ...}, ...}, ...}, ...)
    /renterd/worker/rhpv2.go:291 +0x2f0
go.sia.tech/renterd/worker.(*worker).withRevisionV2(0xc000274740?, {0x19f2400?, 0x19fe938?}, 0x0?, 0xc0002700f0, {0xa6, 0x71, 0x14, 0x40, 0x31, ...}, ...)
    /renterd/worker/rhpv2.go:695 +0x742
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1(0x39c26531401471a6?)
    /renterd/worker/rhpv2.go:280 +0xdc
go.sia.tech/renterd/worker.(*worker).withTransportV2(0x19fa900?, {0x19fa8c8?, 0xc00033a840}, {0xa6, 0x71, 0x14, 0x40, 0x31, 0x65, 0xc2, ...}, ...)
    /renterd/worker/rhpv2.go:639 +0x26f
go.sia.tech/renterd/worker.(*worker).PruneContract.func1()
    /renterd/worker/rhpv2.go:279 +0xdd
go.sia.tech/renterd/worker.(*worker).withContractLock(0xc0000f3d40, {0x19fa8c8?, 0xc00033a840?}, {0x12, 0xe3, 0x51, 0xdb, 0xd5, 0x73, 0x9, ...}, ...)
    /renterd/worker/contract_lock.go:73 +0xcb
go.sia.tech/renterd/worker.(*worker).PruneContract(0x10affbf?, {0x19fa8c8?, 0xc00033a840?}, {0xc000700140?, 0xb79acdd?}, {0xa6, 0x71, 0x14, 0x40, 0x31, ...}, ...)
    /renterd/worker/rhpv2.go:278 +0x125
go.sia.tech/renterd/worker.(*worker).rhpPruneContractHandlerPOST(0xc0000f3d40, {{0x19f8e70, 0xc0000f80e0}, 0xc000140700, {0xc0001ee280, 0x1, 0x1}})
    /renterd/worker/worker.go:546 +0x84b
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.1({0x19f8e70, 0xc0000f80e0}, 0xc000140700)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:202 +0xba
net/http.HandlerFunc.ServeHTTP(0x19fa8c8?, {0x19f8e70?, 0xc0000f80e0?}, 0x19ef750?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.func1.1({0x19f8e70, 0xc0000f80e0}, 0xc000140600)
    /renterd/worker/interactions.go:31 +0x151
net/http.HandlerFunc.ServeHTTP(0xc000140500?, {0x19f8e70?, 0xc0000f80e0?}, 0x20e31a0?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.2({{0x19f8e70, 0xc0000f80e0}, 0xc000140500, {0xc0001ee280, 0x1, 0x1}})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:205 +0xe2
go.sia.tech/jape.Mux.adaptor.func2({0x19f8e70?, 0xc0000f80e0?}, 0xc0000c2280?, {0xc0001ee280?, 0x0?, 0xc0004377d0?})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:164 +0x64
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001a8f00, {0x19f8e70, 0xc0000f80e0}, 0xc000140500)
    /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb
go.sia.tech/jape.BasicAuth.func1.1({0x19f8e70, 0xc0000f80e0}, 0x19f3060?)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:219 +0x99
net/http.HandlerFunc.ServeHTTP(0x19f3060?, {0x19f8e70?, 0xc0000f80e0?}, 0x20d7d88?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.main.main.workerAuth.func9.func10({0x19f8e70, 0xc0000f80e0}, 0xc000140500)
    /renterd/cmd/renterd/main.go:713 +0xf2
net/http.HandlerFunc.ServeHTTP(0xc000437950?, {0x19f8e70?, 0xc0000f80e0?}, 0xc0004379b8?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.treeMux.ServeHTTP({{0x19f20c0?, 0xc000277860?}, 0x0?}, {0x19f8e70, 0xc0000f80e0}, 0xc000140500)
    /renterd/cmd/renterd/web.go:28 +0x27a
main.treeMux.ServeHTTP({{0x0?, 0x0?}, 0xc000277290?}, {0x19f8e70, 0xc0000f80e0}, 0xc000140500)
    /renterd/cmd/renterd/web.go:23 +0x249
net/http.serverHandler.ServeHTTP({0x19f61e8?}, {0x19f8e70?, 0xc0000f80e0?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc000503560, {0x19fa8c8, 0xc0001100f0})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 6
    /usr/local/go/src/net/http/server.go:3086 +0x5cb
http: panic serving 10.42.2.108:44136: VerifySectorRangeProof: number of roots does not match range
goroutine 4949 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xf46bc0?, 0x19f0760?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
go.sia.tech/core/rhp/v2.VerifySectorRangeProof({0x20e31a0, 0x0, 0x0}, {0xc00081e000?, 0x20?, 0x20?}, 0xc0003c0490?, 0xcf23ae4384ed484?, 0x40?, {0x89, ...})
    /go/pkg/mod/go.sia.tech/core@v0.1.12-0.20231211182757-77190f04f90b/rhp/v2/merkle.go:395 +0x30a
go.sia.tech/renterd/worker.(*worker).fetchContractRoots(_, _, _, {0x1, 0x1400000, 0x6540, 0x1400000, {0xc000044810, 0x15}, 0x35de2c00000, ...})
    /renterd/worker/rhpv2.go:601 +0xbfe
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1.1(_, {{{0xea, 0x22, 0xb7, 0x6f, 0x75, 0xc5, 0x90, 0x51, 0x60, ...}, ...}, ...}, ...)
    /renterd/worker/rhpv2.go:291 +0x2f0
go.sia.tech/renterd/worker.(*worker).withRevisionV2(0xc000138480?, {0x19f2400?, 0x19fe938?}, 0x0?, 0xc0001b2000, {0x48, 0x8c, 0x63, 0x89, 0xb, ...}, ...)
    /renterd/worker/rhpv2.go:695 +0x742
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1(0x9e559d0b89638c48?)
    /renterd/worker/rhpv2.go:280 +0xdc
go.sia.tech/renterd/worker.(*worker).withTransportV2(0x19fa900?, {0x19fa8c8?, 0xc00033b1a0}, {0x48, 0x8c, 0x63, 0x89, 0xb, 0x9d, 0x55, ...}, ...)
    /renterd/worker/rhpv2.go:639 +0x26f
go.sia.tech/renterd/worker.(*worker).PruneContract.func1()
    /renterd/worker/rhpv2.go:279 +0xdd
go.sia.tech/renterd/worker.(*worker).withContractLock(0xc0000f3d40, {0x19fa8c8?, 0xc00033b1a0?}, {0xea, 0x22, 0xb7, 0x6f, 0x75, 0xc5, 0x90, ...}, ...)
    /renterd/worker/contract_lock.go:73 +0xcb
go.sia.tech/renterd/worker.(*worker).PruneContract(0x10affbf?, {0x19fa8c8?, 0xc00033b1a0?}, {0xc00014a228?, 0xb79acdd?}, {0x48, 0x8c, 0x63, 0x89, 0xb, ...}, ...)
    /renterd/worker/rhpv2.go:278 +0x125
go.sia.tech/renterd/worker.(*worker).rhpPruneContractHandlerPOST(0xc0000f3d40, {{0x19f8e70, 0xc0000f8460}, 0xc000140f00, {0xc0001ee7c0, 0x1, 0x1}})
    /renterd/worker/worker.go:546 +0x84b
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.1({0x19f8e70, 0xc0000f8460}, 0xc000140f00)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:202 +0xba
net/http.HandlerFunc.ServeHTTP(0x19fa8c8?, {0x19f8e70?, 0xc0000f8460?}, 0x19ef750?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.func1.1({0x19f8e70, 0xc0000f8460}, 0xc000140e00)
    /renterd/worker/interactions.go:31 +0x151
net/http.HandlerFunc.ServeHTTP(0xc000161500?, {0x19f8e70?, 0xc0000f8460?}, 0x20e31a0?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.2({{0x19f8e70, 0xc0000f8460}, 0xc000161500, {0xc0001ee7c0, 0x1, 0x1}})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:205 +0xe2
go.sia.tech/jape.Mux.adaptor.func2({0x19f8e70?, 0xc0000f8460?}, 0xc0000c2480?, {0xc0001ee7c0?, 0x0?, 0xc0006b87d0?})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:164 +0x64
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001a8f00, {0x19f8e70, 0xc0000f8460}, 0xc000161500)
    /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb
go.sia.tech/jape.BasicAuth.func1.1({0x19f8e70, 0xc0000f8460}, 0x19f3060?)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:219 +0x99
net/http.HandlerFunc.ServeHTTP(0x19f3060?, {0x19f8e70?, 0xc0000f8460?}, 0xc000502d80?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.main.main.workerAuth.func9.func10({0x19f8e70, 0xc0000f8460}, 0xc000161500)
    /renterd/cmd/renterd/main.go:713 +0xf2
net/http.HandlerFunc.ServeHTTP(0x7f7c3c3ad1e0?, {0x19f8e70?, 0xc0000f8460?}, 0xc000101c00?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.treeMux.ServeHTTP({{0x19f20c0?, 0xc000277860?}, 0x0?}, {0x19f8e70, 0xc0000f8460}, 0xc000161500)
    /renterd/cmd/renterd/web.go:28 +0x27a
main.treeMux.ServeHTTP({{0x0?, 0x0?}, 0xc000277290?}, {0x19f8e70, 0xc0000f8460}, 0xc000161500)
    /renterd/cmd/renterd/web.go:23 +0x249
net/http.serverHandler.ServeHTTP({0x19f61e8?}, {0x19f8e70?, 0xc0000f8460?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0000f67e0, {0x19fa8c8, 0xc0001100f0})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 6
    /usr/local/go/src/net/http/server.go:3086 +0x5cb
http: panic serving 10.42.2.108:35278: VerifySectorRangeProof: number of roots does not match range
goroutine 5210 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0xf46bc0?, 0x19f0760?})
    /usr/local/go/src/runtime/panic.go:920 +0x270
go.sia.tech/core/rhp/v2.VerifySectorRangeProof({0x20e31a0, 0x0, 0x0}, {0xc0007d4000?, 0x20?, 0x20?}, 0xc0002d8490?, 0x15967f553c948c0?, 0x40?, {0x12, ...})
    /go/pkg/mod/go.sia.tech/core@v0.1.12-0.20231211182757-77190f04f90b/rhp/v2/merkle.go:395 +0x30a
go.sia.tech/renterd/worker.(*worker).fetchContractRoots(_, _, _, {0x1, 0x1400000, 0x6540, 0x1400000, {0xc000138720, 0x1b}, 0xe8be9400000, ...})
    /renterd/worker/rhpv2.go:601 +0xbfe
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1.1(_, {{{0x6b, 0x78, 0xdd, 0x43, 0xc5, 0x70, 0x1d, 0xf4, 0xac, ...}, ...}, ...}, ...)
    /renterd/worker/rhpv2.go:291 +0x2f0
go.sia.tech/renterd/worker.(*worker).withRevisionV2(0xc000138600?, {0x19f2400?, 0x19fe938?}, 0x0?, 0xc0001b2000, {0x9, 0xa5, 0x97, 0x2a, 0x8f, ...}, ...)
    /renterd/worker/rhpv2.go:695 +0x742
go.sia.tech/renterd/worker.(*worker).PruneContract.func1.1(0xf771e88f2a97a509?)
    /renterd/worker/rhpv2.go:280 +0xdc
go.sia.tech/renterd/worker.(*worker).withTransportV2(0x19fa900?, {0x19fa8c8?, 0xc000402750}, {0x9, 0xa5, 0x97, 0x2a, 0x8f, 0xe8, 0x71, ...}, ...)
    /renterd/worker/rhpv2.go:639 +0x26f
go.sia.tech/renterd/worker.(*worker).PruneContract.func1()
    /renterd/worker/rhpv2.go:279 +0xdd
go.sia.tech/renterd/worker.(*worker).withContractLock(0xc0000f3d40, {0x19fa8c8?, 0xc000402750?}, {0x6b, 0x78, 0xdd, 0x43, 0xc5, 0x70, 0x1d, ...}, ...)
    /renterd/worker/contract_lock.go:73 +0xcb
go.sia.tech/renterd/worker.(*worker).PruneContract(0x10affbf?, {0x19fa8c8?, 0xc000402750?}, {0xc0001380c0?, 0xb79acdd?}, {0x9, 0xa5, 0x97, 0x2a, 0x8f, ...}, ...)
    /renterd/worker/rhpv2.go:278 +0x125
go.sia.tech/renterd/worker.(*worker).rhpPruneContractHandlerPOST(0xc0000f3d40, {{0x19f8e70, 0xc0000f80e0}, 0xc0004d8800, {0xc0001ee100, 0x1, 0x1}})
    /renterd/worker/worker.go:546 +0x84b
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.1({0x19f8e70, 0xc0000f80e0}, 0xc0004d8800)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:202 +0xba
net/http.HandlerFunc.ServeHTTP(0x19fa8c8?, {0x19f8e70?, 0xc0000f80e0?}, 0x19ef750?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.func1.1({0x19f8e70, 0xc0000f80e0}, 0xc0004d8700)
    /renterd/worker/interactions.go:31 +0x151
net/http.HandlerFunc.ServeHTTP(0xc000554300?, {0x19f8e70?, 0xc0000f80e0?}, 0x20e31a0?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
go.sia.tech/renterd/worker.interactionMiddleware.Adapt.func2.2({{0x19f8e70, 0xc0000f80e0}, 0xc000554300, {0xc0001ee100, 0x1, 0x1}})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:205 +0xe2
go.sia.tech/jape.Mux.adaptor.func2({0x19f8e70?, 0xc0000f80e0?}, 0xc0000c2400?, {0xc0001ee100?, 0x0?, 0xc0003247d0?})
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:164 +0x64
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0001a8f00, {0x19f8e70, 0xc0000f80e0}, 0xc000554300)
    /go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x7eb
go.sia.tech/jape.BasicAuth.func1.1({0x19f8e70, 0xc0000f80e0}, 0x19f3060?)
    /go/pkg/mod/go.sia.tech/jape@v0.11.1/server.go:219 +0x99
net/http.HandlerFunc.ServeHTTP(0x19f3060?, {0x19f8e70?, 0xc0000f80e0?}, 0x20d7d88?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.main.main.workerAuth.func9.func10({0x19f8e70, 0xc0000f80e0}, 0xc000554300)
    /renterd/cmd/renterd/main.go:713 +0xf2
net/http.HandlerFunc.ServeHTTP(0xc000324950?, {0x19f8e70?, 0xc0000f80e0?}, 0xc0002bc800?)
    /usr/local/go/src/net/http/server.go:2136 +0x29
main.treeMux.ServeHTTP({{0x19f20c0?, 0xc000277860?}, 0x0?}, {0x19f8e70, 0xc0000f80e0}, 0xc000554300)
    /renterd/cmd/renterd/web.go:28 +0x27a
main.treeMux.ServeHTTP({{0x0?, 0x0?}, 0xc000277290?}, {0x19f8e70, 0xc0000f80e0}, 0xc000554300)
    /renterd/cmd/renterd/web.go:23 +0x249
net/http.serverHandler.ServeHTTP({0x19f61e8?}, {0x19f8e70?, 0xc0000f80e0?}, 0x6?)
    /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc00048b170, {0x19fa8c8, 0xc0001100f0})
    /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 6
    /usr/local/go/src/net/http/server.go:3086 +0x5cb

10.42.2.108 is autopilot, so I also looked in autopilot logs

2024-01-12T13:35:39Z    DEBUG   autopilot.contractor    autopilot/contractor.go:203 fetched 22 contracts from the worker, took 2m9.81647405s
2024-01-12T13:35:39Z    INFO    autopilot.contractor    autopilot/contractor.go:965 revision broadcast completed    {"successful": 0, "failed": 0}
2024-01-12T13:35:39Z    DEBUG   autopilot.contractor    autopilot/contractor.go:1260    selected 3 (potentially) usable hosts for scoring out of 26 {"excluded": 22, "notcompletedscan": 1, "used": 22}
2024-01-12T13:35:39Z    DEBUG   autopilot.contractor    autopilot/contractor.go:1293    scored 0 unused hosts out of 3, took 814.978µs  {"zeroscore": 1, "unusable": 3, "used": 22}
2024-01-12T13:35:39Z    WARN    autopilot.contractor    autopilot/contractor.go:1201    min host score is set to the smallest non-zero float because there are no candidate hosts
2024-01-12T13:35:39Z    DEBUG   autopilot.contractor    autopilot/contractor.go:617 running contract checks
2024-01-12T13:35:40Z    ERROR   autopilot.contractor    autopilot/contractor.go:707 could not fetch price table for host ed25519:87e90594ac909076f6473fbebfe278fa6794fe0bb7b3b093b0792e5a5b734239: failed to fetch price table for host ed25519:87e90594ac909076f6473fbebfe278fa6794fe0bb7b3b093b0792e5a5b734239: could not get price table: PriceTable: DialStream: could not dial transport: dial tcp 185.200.116.211:9809: connect: connection refused (198.474953ms)
2024-01-12T13:35:40Z    INFO    autopilot.contractor    autopilot/contractor.go:732 unusable host   {"hk": "ed25519:87e90594ac909076f6473fbebfe278fa6794fe0bb7b3b093b0792e5a5b734239", "fcid": "fcid:ae9d1f60e0984b234a5257d629319d20f1ff45236ba7b5d336a3ec6e2113b4f9", "reasons": ["host is offline"]}
2024-01-12T13:35:43Z    ERROR   autopilot.contractor    autopilot/contractor.go:707 could not fetch price table for host ed25519:9d0dc5d8a28332254debcf81c83ba429283293cc9647f96cd1071dad34717aab: failed to fetch price table for host ed25519:9d0dc5d8a28332254debcf81c83ba429283293cc9647f96cd1071dad34717aab: could not get price table: PriceTable: DialStream: could not dial transport: dial tcp 173.73.170.76:9883: connect: no route to host (3.439255152s)
2024-01-12T13:35:43Z    INFO    autopilot.contractor    autopilot/contractor.go:732 unusable host   {"hk": "ed25519:9d0dc5d8a28332254debcf81c83ba429283293cc9647f96cd1071dad34717aab", "fcid": "fcid:7e192c21b937d9b66d391d7c94168bdb2df1be0b6e61a1d6fb3d324d54ef2e21", "reasons": ["host is offline"]}
2024-01-12T13:36:13Z    ERROR   autopilot.contractor    autopilot/contractor.go:707 could not fetch price table for host ed25519:63cc01b3e375fba5edbe9c7943d605f269598ec70cc86a52af41eff70660791b: failed to fetch price table for host ed25519:63cc01b3e375fba5edbe9c7943d605f269598ec70cc86a52af41eff70660791b: could not get price table: PriceTable: DialStream: could not dial transport: dial tcp 80.161.50.205:9883: i/o timeout (30.005207047s)
2024-01-12T13:36:15Z    INFO    autopilot.contractor    autopilot/contractor.go:769 unusable contract   {"hk": "ed25519:e3e710c1ea553729ef932b17fbec98e1e6acba484797624d06fb275a10f886da", "fcid": "fcid:0a72a21254824ad824ab6b875756bde1a4d0c8c534bfba605e1b1408cbc0e2a5", "reasons": ["host has redundant IP"], "refresh": false, "renew": false, "recoverable": false}
2024-01-12T13:36:15Z    INFO    autopilot.contractor    autopilot/contractor.go:769 unusable contract   {"hk": "ed25519:6fdce0663adbff9dd0ced4360721c8aea53433cab61c72c2f0082d3a87085400", "fcid": "fcid:824dc03215b5f29d53c5e650509965b374c0f9cad3a539df985df9d2bd62cf25", "reasons": ["host has redundant IP"], "refresh": false, "renew": false, "recoverable": false}
2024-01-12T13:36:15Z    INFO    autopilot.contractor    autopilot/contractor.go:769 unusable contract   {"hk": "ed25519:a95bfd42307179f0affc1512b7014f0c50ab4d208d02f8e18d2ead8bf9b08edb", "fcid": "fcid:1b5f625ade5a0e579fba412d77f08ea6de44ecb975e2c84ef83f3d5b91267836", "reasons": ["host has redundant IP"], "refresh": false, "renew": false, "recoverable": false}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:639 contracts checks completed  {"contracts": 22, "notfound": 0, "usedKeepLeeway": 2, "toKeep": 17, "toArchive": 0, "toRefresh": 0, "toRenew": 0}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:1040    run contracts refreshes {"torefresh": 0, "budget": "~143.7 KS"}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:1046    contracts refreshes completed   {"refreshed": 0, "budget": "~143.7 KS"}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:805 run contract formations {"usedHosts": 22, "required": 30, "missing": 13, "budget": "~143.7 KS"}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:825 looking for 15 candidate hosts
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:834 no candidate hosts found    {"offline": 1, "lowscore": 1, "notacceptingcontracts": 2}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:813 contract formations completed   {"formed": 0, "budget": "~143.7 KS"}
2024-01-12T13:36:15Z    DEBUG   autopilot.contractor    autopilot/contractor.go:502 contractset after maintenance   {"formed": 0, "renewed": 0, "refreshed": 0, "contracts": 17, "added": 0, "removed": 0}
2024-01-12T13:36:15Z    INFO    autopilot   autopilot/autopilot.go:299  autopilot iteration ended
2024-01-12T13:36:15Z    INFO    autopilot.contractor    autopilot/contract_pruning.go:127   performing contract pruning
2024-01-12T13:36:15Z    INFO    autopilot.migrator  autopilot/migrator.go:121   performing migrations
2024-01-12T13:36:15Z    DEBUG   autopilot.migrator  autopilot/migrator.go:207   recomputed slab health in 104.688037ms
2024-01-12T13:36:15Z    DEBUG   autopilot.migrator  autopilot/migrator.go:215   0 potential slabs fetched for migration
2024-01-12T13:36:15Z    DEBUG   autopilot.migrator  autopilot/migrator.go:250   0 slabs to migrate
2024-01-12T13:36:17Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:24c38aa1fa0bf62e707fbcf0c1e630707e36b3ea5b47c4df60dc4335b3b62485, host ed25519:95770d3b9bf02e025bb7f76715cccf943974e73c480d81e1d9b480845bca1989 version 1.6.0, err: Post "http://renterd-worker-2.renterd-worker:9980/api/worker/rhp/contract/fcid:24c38aa1fa0bf62e707fbcf0c1e630707e36b3ea5b47c4df60dc4335b3b62485/prune": EOF
2024-01-12T13:36:18Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:879ea8a0d03eedf2a917879b2b7dfb95aa7930998c6c8e80c3bae07bfc3a0eba, host ed25519:707d189dfd08469efebf1a51338b3d73eed4ac1b56213a0ed6521179f60a7b71 version 1.6.0, err: Post "http://renterd-worker-2.renterd-worker:9980/api/worker/rhp/contract/fcid:879ea8a0d03eedf2a917879b2b7dfb95aa7930998c6c8e80c3bae07bfc3a0eba/prune": EOF
2024-01-12T13:36:25Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:df429e7fa0458def0a0f62fb1f0a2de741c3374e6fbe3497ec4900b476943ef8, host ed25519:eb51aa3633a87d04acf7a8446a1f842e45db88b37e31e6b353fde74c141576ee version 1.6.0, err: failed to prune contract fcid:df429e7fa0458def0a0f62fb1f0a2de741c3374e6fbe3497ec4900b476943ef8; couldn't read signature response, err: ReadResponse: host internal error
2024-01-12T13:36:25Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:12e351dbd57309ab2a47e493e68656240b7f0f3170856f6593de1a0dcd59d3d7, host ed25519:a67114403165c23973fd34048ce18c4c1cb9155f6dddf41f8cb007fea3a6fc0f version 1.6.0, err: Post "http://renterd-worker-2.renterd-worker:9980/api/worker/rhp/contract/fcid:12e351dbd57309ab2a47e493e68656240b7f0f3170856f6593de1a0dcd59d3d7/prune": EOF
2024-01-12T13:36:26Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:ea22b76f75c5905160c29e916de8fbd36ed3a428b92e0ef2ff0cb8a7deb29031, host ed25519:488c63890b9d559e705bcbc142243caf09d3d216af22c131ab9e019356e8ff6e version 1.6.0, err: Post "http://renterd-worker-2.renterd-worker:9980/api/worker/rhp/contract/fcid:ea22b76f75c5905160c29e916de8fbd36ed3a428b92e0ef2ff0cb8a7deb29031/prune": EOF
2024-01-12T13:36:26Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:7e0f137a05451e62e90a93f0dd0569dd791a13ec37a1a35404333426b8e1510b, host ed25519:075c746dd0e85eea7aeb05a5c4f37c2af35e4a8c48493fccec285ead4dbe5c49 version 1.6.0, err: failed to prune contract fcid:7e0f137a05451e62e90a93f0dd0569dd791a13ec37a1a35404333426b8e1510b; couldn't read sector roots response: ReadResponse: EOF
2024-01-12T13:38:35Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:2ad6ad1de4bc022f335bf76bbba0886ca163864aa688407339332cb35cf92031, host ed25519:63cc01b3e375fba5edbe9c7943d605f269598ec70cc86a52af41eff70660791b version 1.6.0, err: failed to prune contract fcid:2ad6ad1de4bc022f335bf76bbba0886ca163864aa688407339332cb35cf92031; dial tcp 80.161.50.205:9882: connect: connection timed out
2024-01-12T13:38:36Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:6b78dd43c5701df4ac25e2e515f8a824a92eff7f943b883e3cb5ea3023daabe8, host ed25519:09a5972a8fe871f765a52729fab15a78a2e0a7d85a8b614de3f4295f2a87292b version 1.6.0, err: Post "http://renterd-worker-2.renterd-worker:9980/api/worker/rhp/contract/fcid:6b78dd43c5701df4ac25e2e515f8a824a92eff7f943b883e3cb5ea3023daabe8/prune": EOF
2024-01-12T13:38:36Z    ERROR   autopilot.contractor    autopilot/contract_pruning.go:153   contract fcid:0f1180fe6741c7798ff90ba167c5cf260f86e72b6da854e86eeada665d08653a, host ed25519:a3aef7db34413e0f1d33140c3cab0a0113317d55b99df6c77a38fbc7a11d2ae1 version 1.6.0, err: failed to prune contract fcid:0f1180fe6741c7798ff90ba167c5cf260f86e72b6da854e86eeada665d08653a; unexpected revision number, 7597!=7600
2024-01-12T13:38:36Z    INFO    autopilot.contractor    autopilot/contract_pruning.go:180   pruned 0 (0 B) from 9 contracts
2024-01-12T14:03:29Z    INFO    autopilot   autopilot/autopilot.go:205  autopilot iteration starting
2024-01-12T14:03:29Z    DEBUG   autopilot.scanner   autopilot/scanner.go:243    updated timeout is lower than min timeout, 0s<10s
2024-01-12T14:03:29Z    INFO    autopilot   autopilot/autopilot.go:247  using worker renterd-worker-2 for iteration
2024-01-12T14:03:29Z    INFO    autopilot.contractor    autopilot/contractor.go:523 performing wallet maintenance
2024-01-12T14:03:29Z    DEBUG   autopilot.contractor    autopilot/contractor.go:587 no wallet maintenance needed, plenty of outputs available (361>=30)
2024-01-12T14:03:29Z    INFO    autopilot.contractor    autopilot/contractor.go:156 performing contract maintenance
2024-01-12T14:03:29Z    DEBUG   autopilot.contractor    autopilot/contractor.go:191 contract set 'autopilot' holds 17 contracts

Expected Behavior

x

Steps to Reproduce

No response

Version

cf362bb

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Ubuntu 22.04, kubernetes

Autopilot Config

{ "contracts": { "set": "autopilot", "amount": 30, "allowance": "143692570742000000000000000000", "period": 6048, "renewWindow": 2016, "download": 7000000000000, "upload": 7000000000000, "storage": 4000000000000, "prune": true }, "hosts": { "allowRedundantIPs": false, "maxDowntimeHours": 336, "minRecentScanFailures": 10, "scoreOverrides": null } }

Bus Config

{ "default": "autopilot" } { "hostBlockHeightLeeway": 6, "maxContractPrice": "15000000000000000000000000", "maxDownloadPrice": "3551431456000000000000000000", "maxRPCPrice": "1000000000000000000000", "maxStoragePrice": "288857732948", "maxUploadPrice": "1077494143666666666666666667", "migrationSurchargeMultiplier": 10, "minAccountExpiry": 86400000000000, "minMaxCollateral": "10000000000000000000000000", "minMaxEphemeralAccountBalance": "1000000000000000000000000", "minPriceTableValidity": 300000000000 } { "minShards": 2, "totalShards": 6 } { "enabled": true, "slabBufferMaxSizeSoft": 4294967296 }

Contract Set Contracts

17

Anything else?

No response

ChrisSchinnerl commented 10 months ago

@peterjan we should probably check n matches rootsResp.SectorRoots after reading the response from the host. This seems like it might be due to the host not returning the expected number of roots.