search

SiaFoundation / renterd

A renter for Sia
https://sia.tech/software/renterd
MIT License
66 stars 19 forks source link

autopilot is trying to migrate slabs to host with full disk #925

Closed artur9010 closed 4 months ago

artur9010 commented 7 months ago

Current Behavior

autopilot is trying to migrate all of slabs to host with full disk

2024-02-01T21:56:18Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-1: migration 67/112 failed, key: key:ec48be72f05574899db428572dfc8155ba7a9681f7038be7896bc038c672bca6, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
2024-02-01T21:56:22Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-0: migration 68/112 failed, key: key:f346fee153a788edc0c707ccc16fd05acb572f9e71603a7ad2bfe8e2e2ff40e7, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
2024-02-01T21:56:27Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-2: migration 69/112 failed, key: key:f50db018d41b3937aabbd1fb4e0bc2afa247b20152cfa296a0fe321915f46494, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
2024-02-01T21:56:29Z    INFO    autopilot.accounts  autopilot/accounts.go:161   Successfully funded account {"account": "ed25519:6bf736169e8f8f29cc99fc73409437c18521129ecd4f27a2658a1d69e55769fa", "host": "ed25519:c101ca41273de728bd51c76c003c2d9ff93cba713c2c57850c5e50a66a7a8791", "balance": "1 SC"}
2024-02-01T21:56:31Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-1: migration 70/112 failed, key: key:9ad78cb5737562fa9737cc8850dc9b5b4d4ad13525fc9789461bc2e237b64516, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
2024-02-01T21:56:33Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-0: migration 71/112 failed, key: key:7f669c825103db099cb4266e45bffaf883459dd08ac6fdb54da8991e8abf09aa, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
2024-02-01T21:56:38Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-2: migration 72/112 failed, key: key:f0d9005e8fe50ad64f7312aaf9a9d49566d54796cf6e4af18b2907764b455918, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed
2024-02-01T21:56:46Z    ERROR   autopilot.migrator  autopilot/migrator.go:158   renterd-worker-1: migration 73/112 failed, key: key:f550f7e4a33b7ff61a4695b31a2497f804ce2c9875fc1b0b919e69e66e450f4c, health: 0.75, overpaid: false, err: failed to migrate slab; failed to upload slab for migration: failed to upload slab: launched=9 uploaded=0 remaining=2 inflight=0 pending=2 uploaders=9 errors=9
63cc01b3: failed to upload sector to contract fcid:17d54655ca2b69d50fde6ceb6d3b49ceb17dc4849c25f855ef17872cccf5349b, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: failed to store sector: failed to write sector data: failed to write sector to index 114085: write G:\hostd-zen-931GB-B.dat: Der er ikke tilstrækkelig plads på disken.
87e90594: failed to upload sector to contract fcid:aafbbc3f0fdc425a5c8e9f91bd1dc241c028617fd17844ebcba4f841b6187586, err: AppendSector: proof verification failed
09a5972a: failed to upload sector to contract fcid:e68fbbd3f962b183614fa7c79ab73f650f142a22233e058ae41a0685b67e056b, err: AppendSector: proof verification failed
eb51aa36: failed to upload sector to contract fcid:1b41dbc343040af1881cf1579ac71778465ceef4b17d08f19a55a066ebeef8c3, err: AppendSector: failed to execute instruction "AppendSector": failed to write sector: transaction failed (1): failed to get empty location: not enough storage
a6711440: failed to upload sector to contract fcid:186fb3a683c1d4f65a75b3e13fe9055770d72850eab9a976c2b025628e444ddf, err: AppendSector: proof verification failed
075c746d: failed to upload sector to contract fcid:defc5b69983c18d542062705c18914658edd491c31b7d3f7789f2840487bd1ec, err: AppendSector: proof verification failed
707d189d: failed to upload sector to contract fcid:3bf78cfc3b6179ecee04994c65d1b8a11e5540f42a7d59567b82e68e73935130, err: AppendSector: proof verification failed
488c6389: failed to upload sector to contract fcid:d3e809b486937508808feca121ab01528d9e894df0f7156601020b7a7755e97e, err: AppendSector: proof verification failed
95770d3b: failed to upload sector to contract fcid:97aca224e7c0fad4b5a598b1cfb7dccde10be792b87cb3492468a4065ca13f6f, err: AppendSector: proof verification failed

response from host: image

Expected Behavior

x

Steps to Reproduce

No response

Version

402398b, zen

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Kubernetes, https://github.com/artur9010/charts/tree/master/renterd

Autopilot Config

probably not necessary

Bus Config

probably not necessary

Contract Set Contracts

probably not necessary

Anything else?

No response

ChrisSchinnerl commented 6 months ago

When we upload we try all hosts. It shouldn't cause uploads to fail. However, I'll add this to the backlog since we might want to change the code slightly to not bother hosts that ran out of storage too much.

peterjan commented 4 months ago

@ChrisSchinnerl upload penalty should de-prioritise hosts with full disks. Not sure if we need to do more than that. The issue described here is likely due to a lack of contracts in general and just testnet hosts being testnet hosts.

ChrisSchinnerl commented 4 months ago

Closing this for now since we PJ is right and we confirmed that with additional testing around host penalties recently.