search

autonomys / subspace

Subspace Network reference implementation
https://subspace.network
378 stars 243 forks source link

Farmer hangs after Ctrl-C when there are invalid chunk errors #3178

Open teor2345 opened 2 days ago

teor2345 commented 2 days ago

Setup

On macOS 13.7 on M1 Max, I ran the following commands:

./subspace-node run --chain devnet --base-path ~/subspace-nets/devnet-2024-10-24/ --blocks-pruning archive --state-pruning archive --sync full --bootstrap-nodes /dns/bootstrap-0.devnet.subspace.foundation/tcp/30333/p2p/12D3KooWKd4qisMjuBXU4YvpbJgepiX278iumqMjCC3GiUjjZEZS --dsn-bootstrap-nodes /dns/bootstrap-0.devnet.subspace.foundation/tcp/30533/p2p/12D3KooWSKAQm66N7jQme72obbPPquBSZuSqMyx5mH5ki1aZ4pGv --pot-external-entropy 0xddca5a66 --farmer
./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:RPC_PORT

This bug happens with the binaries from:

Root Cause

I understand the network is partly shut down, and the farmer's storage might be corrupted. But I'm not sure how the corruption happened, I was running the devnet binaries as part of the test network. Unfortunately I don't have logs of the corruption itself, because my terminal only saves a few thousand lines.

Whatever the root cause was, the farmer shouldn't hang when it's newly started, even if its storage is corrupt.

Errors

There were a bunch of "invalid chunk" errors, so I pressed Ctrl-C:

2024-10-28T02:30:46.377291Z  INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=4403
2024-10-28T02:30:50.237170Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=628 error=Invalid chunk at location 20532296 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:52.110828Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:30:56.068184Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=983 error=Invalid chunk at location 20530024 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T02:31:09.391617Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...

But the node didn't exit after I pressed Ctrl-C, or when I used kill (SIGTERM) on its pid:

2024-10-28T02:31:13.727143Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar
2024-10-28T02:31:15.589845Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar

I had to use kill -KILL to get it to exit.

Where it's hanging

The farmer is hanging somewhere in the piece reading code, but I'm not sure exactly which function is blocking it exiting.

Here's what I got from flamegraph --pid PID after the Ctrl-C when the farmer didn't shut down: flamegraph

Full Logs

The full logs are:

``` $ ./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:49918 Exit 137 2024-10-28T02:30:42.546960Z INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:49918 2024-10-28T02:30:42.559884Z INFO subspace_farmer::node_client::caching_proxy_node_client: Downloading all segment headers from node... 2024-10-28T02:30:42.560240Z INFO subspace_farmer::node_client::caching_proxy_node_client: Downloaded all segment headers from node successfully 2024-10-28T02:30:42.563752Z INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv protocol_version=/subspace/2/1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d 2024-10-28T02:30:42.567371Z INFO libp2p_swarm: local_peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T02:30:42.998612Z INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-9), .. }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-4), .. }] 2024-10-28T02:30:43.006612Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while 2024-10-28T02:30:43.006756Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents 2024-10-28T02:30:43.014136Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method 2024-10-28T02:30:45.547367Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector 2024-10-28T02:30:45.562124Z INFO {farm_index=0}: subspace_farmer::commands::farm: Farm 0: 2024-10-28T02:30:45.562136Z INFO {farm_index=0}: subspace_farmer::commands::farm: ID: 01JARGR52SKF93EY07J9C6DJRA 2024-10-28T02:30:45.562141Z INFO {farm_index=0}: subspace_farmer::commands::farm: Genesis hash: 0x1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d 2024-10-28T02:30:45.562143Z INFO {farm_index=0}: subspace_farmer::commands::farm: Public key: 0x189db863c1905d14e524456c7496766ddd80da2856b24dd21627b5d745d52344 2024-10-28T02:30:45.562149Z INFO {farm_index=0}: subspace_farmer::commands::farm: Allocated space: 9.3 GiB (10.0 GB) 2024-10-28T02:30:45.562174Z INFO {farm_index=0}: subspace_farmer::commands::farm: Directory: /Users/hyper/subspace-nets/devnet-2024-10-24/farm-0 2024-10-28T02:30:45.562381Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)... 2024-10-28T02:30:45.562423Z INFO subspace_farmer::farmer_cache: Initializing piece cache 2024-10-28T02:30:45.564209Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully 2024-10-28T02:30:45.564706Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments 2024-10-28T02:30:45.564873Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications 2024-10-28T02:30:45.564883Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::reward_signing: Subscribing to reward signing notifications 2024-10-28T02:30:45.565309Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T02:30:45.565566Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T02:30:45.565577Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.222/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T02:30:45.565586Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.2/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T02:30:46.071369Z INFO subspace_farmer::farmer_cache: Synchronizing piece cache 2024-10-28T02:30:46.073043Z INFO {farm_index=0}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) 2024-10-28T02:30:46.102827Z INFO subspace_farmer::farmer_cache: Finished piece cache synchronization 2024-10-28T02:30:46.377291Z INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=4403 2024-10-28T02:30:50.237170Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=628 error=Invalid chunk at location 20532296 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:30:52.110828Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:30:56.068184Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=983 error=Invalid chunk at location 20530024 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar ^C2024-10-28T02:31:09.391617Z INFO subspace_farmer::utils: Received SIGINT, shutting down farmer... 2024-10-28T02:31:13.727143Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:31:15.589845Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:31:27.168930Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=447 error=Invalid chunk at location 15399502 s-bucket 24581 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:31:29.018152Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=860 error=Invalid chunk at location 20529955 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:31:40.799672Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=605 error=Invalid chunk at location 20529791 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:31:48.523526Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=298 error=Invalid chunk at location 20529593 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:32:32.737403Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=531 error=Invalid chunk at location 20530366 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:32:44.742552Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=574 error=Invalid chunk at location 15397684 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:32:50.752305Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=304 error=Invalid chunk at location 20530222 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:32:56.726401Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=652 error=Invalid chunk at location 20530447 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:33:02.759542Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=936 error=Invalid chunk at location 15399186 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:33:12.671331Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=9 error=Invalid chunk at location 20529393 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:33:18.631802Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=717 error=Invalid chunk at location 20532353 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:34:04.455366Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=316 error=Invalid chunk at location 15398798 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:34:08.345956Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=794 error=Invalid chunk at location 15396527 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:34:17.977849Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 20530090 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:34:25.793158Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=57 error=Invalid chunk at location 20531930 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:34:31.909144Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=979 error=Invalid chunk at location 20530020 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:35:03.696272Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=934 error=Invalid chunk at location 15397911 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:35:13.484399Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=863 error=Invalid chunk at location 20531191 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:35:45.555993Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=559 error=Invalid chunk at location 20529758 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:35:46.076149Z WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers. 2024-10-28T02:35:47.469890Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=400 error=Invalid chunk at location 20532152 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:36:07.091913Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=669 error=Invalid chunk at location 20531688 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:36:55.045260Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=141 error=Invalid chunk at location 20530125 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:36:56.909655Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=18 error=Invalid chunk at location 20529399 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:36:58.861221Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=117 error=Invalid chunk at location 15398021 s-bucket 24579 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:37:04.721454Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=635 error=Invalid chunk at location 20531062 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T02:37:10.664913Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=219 error=Invalid chunk at location 20532664 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar ```
teor2345 commented 2 days ago

Attempted Fixes

This issue still happens after the code changes in:

Further Analysis

I added some logs to see where the hang was:

2024-10-28T03:04:35.739207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=651 error=Invalid chunk at location 20531677 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar
^C2024-10-28T03:04:39.384097Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...
2024-10-28T03:04:39.384111Z  INFO subspace_farmer::commands::farm: signal select branch
2024-10-28T03:04:39.384113Z  INFO subspace_farmer::commands::farm: end of async fn farm()
2024-10-28T03:05:10.236106Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=671 error=Invalid chunk at location 15396453 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar

The farmer is blocked at the end of the farm() function. None of the logs I added in main() got logged.

teor2345 commented 2 days ago

Aha, it did exit, it just took 17 minutes.

Full logs (including logs that show where in the code it hung):

``` $ ./subspace-farmer farm --reward-address 5Dsysn9Pu9fLwNEau17JpqJ1tDRtRRiXBKy9GWWxUezu1BFq path=~/subspace-nets/devnet-2024-10-24/farm-0,size=10G --node-rpc-url ws://127.0.0.1:49918 Exit 137 2024-10-28T03:04:27.869201Z INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:49918 2024-10-28T03:04:27.879595Z INFO subspace_farmer::node_client::caching_proxy_node_client: Downloading all segment headers from node... 2024-10-28T03:04:27.879993Z INFO subspace_farmer::node_client::caching_proxy_node_client: Downloaded all segment headers from node successfully 2024-10-28T03:04:27.883274Z INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv protocol_version=/subspace/2/1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d 2024-10-28T03:04:27.886818Z INFO libp2p_swarm: local_peer_id=12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T03:04:28.311078Z INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-9), .. }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-4), .. }] 2024-10-28T03:04:28.318777Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while 2024-10-28T03:04:28.318931Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents 2024-10-28T03:04:28.326807Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method 2024-10-28T03:04:30.687798Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector 2024-10-28T03:04:30.701671Z INFO {farm_index=0}: subspace_farmer::commands::farm: Farm 0: 2024-10-28T03:04:30.701682Z INFO {farm_index=0}: subspace_farmer::commands::farm: ID: 01JARGR52SKF93EY07J9C6DJRA 2024-10-28T03:04:30.701698Z INFO {farm_index=0}: subspace_farmer::commands::farm: Genesis hash: 0x1666067b80762b374703f5267360f8c63c0a8c068579b63c7455c6b2302ec57d 2024-10-28T03:04:30.701701Z INFO {farm_index=0}: subspace_farmer::commands::farm: Public key: 0x189db863c1905d14e524456c7496766ddd80da2856b24dd21627b5d745d52344 2024-10-28T03:04:30.701727Z INFO {farm_index=0}: subspace_farmer::commands::farm: Allocated space: 9.3 GiB (10.0 GB) 2024-10-28T03:04:30.701744Z INFO {farm_index=0}: subspace_farmer::commands::farm: Directory: /Users/hyper/subspace-nets/devnet-2024-10-24/farm-0 2024-10-28T03:04:30.701868Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)... 2024-10-28T03:04:30.701924Z INFO subspace_farmer::farmer_cache: Initializing piece cache 2024-10-28T03:04:30.703654Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully 2024-10-28T03:04:30.703933Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments 2024-10-28T03:04:30.703949Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications 2024-10-28T03:04:30.703956Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::reward_signing: Subscribing to reward signing notifications 2024-10-28T03:04:30.705087Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T03:04:30.705350Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T03:04:30.705358Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.222/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T03:04:30.705370Z INFO subspace_farmer::commands::shared::network: DSN listening on /ip4/192.168.18.2/tcp/30533/p2p/12D3KooWFib6tNB2BQVXom6veQA2Lf8WjbNePQpPETeGSc1oy9rv 2024-10-28T03:04:31.209924Z INFO subspace_farmer::farmer_cache: Synchronizing piece cache 2024-10-28T03:04:31.211675Z INFO {farm_index=0}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) 2024-10-28T03:04:31.896383Z WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers. 2024-10-28T03:04:31.922698Z INFO subspace_farmer::farmer_cache: Finished piece cache synchronization 2024-10-28T03:04:32.047669Z INFO {farm_index=0}:{sector_index=7}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=1059 2024-10-28T03:04:35.739207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=651 error=Invalid chunk at location 20531677 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar ^C2024-10-28T03:04:39.384097Z INFO subspace_farmer::utils: Received SIGINT, shutting down farmer... 2024-10-28T03:04:39.384111Z INFO subspace_farmer::commands::farm: signal select branch 2024-10-28T03:04:39.384113Z INFO subspace_farmer::commands::farm: end of async fn farm() 2024-10-28T03:05:10.236106Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=671 error=Invalid chunk at location 15396453 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:05:15.744994Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=582 error=Invalid chunk at location 20530402 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:05:19.355233Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=442 error=Invalid chunk at location 15397603 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:05:40.039400Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=14 error=Invalid chunk at location 15396673 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:06:13.850488Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=574 error=Invalid chunk at location 15397684 s-bucket 24578 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:06:40.465975Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=714 error=Invalid chunk at location 20530482 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:01.897407Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=979 error=Invalid chunk at location 20530020 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:05.792917Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=814 error=Invalid chunk at location 20529929 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:09.533264Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=446 error=Invalid chunk at location 20529684 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:13.242498Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=501 error=Invalid chunk at location 20529720 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:20.760735Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=863 error=Invalid chunk at location 20531191 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:45.220739Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=577 error=Invalid chunk at location 20529770 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:07:48.921442Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=605 error=Invalid chunk at location 20529791 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:08:00.354067Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=404 error=Invalid chunk at location 20529659 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:08:05.904692Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=874 error=Invalid chunk at location 20533071 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:08:17.274189Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=447 error=Invalid chunk at location 20531550 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:09:09.340978Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=45 error=Invalid chunk at location 20529418 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:09:11.128233Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=678 error=Invalid chunk at location 23096087 s-bucket 36864 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:09:31.906446Z WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers. 2024-10-28T03:09:35.754257Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=141 error=Invalid chunk at location 20530125 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:09:45.278837Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=558 error=Invalid chunk at location 20531015 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:09:58.439013Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=117 error=Invalid chunk at location 20532595 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:10:17.214293Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=366 error=Invalid chunk at location 15396908 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:10:20.777816Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=852 error=Invalid chunk at location 20530572 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:03.752059Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=202 error=Invalid chunk at location 20532028 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:07.375453Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=316 error=Invalid chunk at location 20530868 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:14.808446Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=95 error=Invalid chunk at location 15396723 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:20.290224Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=936 error=Invalid chunk at location 15399186 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:22.062909Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=298 error=Invalid chunk at location 20529593 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:31.383476Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=134 error=Invalid chunk at location 15398682 s-bucket 24580 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:11:51.795447Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=473 error=Invalid chunk at location 20534691 s-bucket 32776 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:12:04.896285Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=219 error=Invalid chunk at location 20532664 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:12:21.747393Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=59 error=Invalid chunk at location 15396700 s-bucket 24577 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:12:25.429656Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=34 error=Invalid chunk at location 20531914 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:12:38.485213Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=377 error=Invalid chunk at location 20530265 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:12:40.243353Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=569 error=Invalid chunk at location 20531023 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:10.285007Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 20530090 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:13.862207Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=403 error=Invalid chunk at location 20531523 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:15.639197Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=848 error=Invalid chunk at location 20530569 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:26.886856Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:34.381042Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=860 error=Invalid chunk at location 20529955 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:41.992782Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=18 error=Invalid chunk at location 15396060 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:47.416684Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=151 error=Invalid chunk at location 20530131 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:49.178926Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=57 error=Invalid chunk at location 20531930 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:13:50.972383Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=304 error=Invalid chunk at location 15396236 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:14:31.803047Z WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers. 2024-10-28T03:14:53.193743Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=73 error=Invalid chunk at location 20530082 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:02.531185Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=829 error=Invalid chunk at location 16680023 s-bucket 26624 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:04.306124Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=292 error=Invalid chunk at location 20530215 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:13.655777Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=144 error=Invalid chunk at location 20532611 s-bucket 32773 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:19.137320Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=2 error=Invalid chunk at location 20530039 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:35.892909Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=74 error=Invalid chunk at location 20530083 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:15:54.633768Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=717 error=Invalid chunk at location 20532353 s-bucket 32772 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:16:05.829172Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=756 error=Invalid chunk at location 15398436 s-bucket 24579 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:16:07.632074Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=791 error=Invalid chunk at location 20529916 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:16:11.339153Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=150 error=Invalid chunk at location 20530130 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:16:18.701387Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=474 error=Invalid chunk at location 21812639 s-bucket 34816 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:16:28.044645Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=755 error=Invalid chunk at location 20530507 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:17:09.497806Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=452 error=Invalid chunk at location 20530318 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:17:13.174638Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=837 error=Invalid chunk at location 20533671 s-bucket 32774 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:17:24.471791Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=669 error=Invalid chunk at location 20531688 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:17:31.978087Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=20 error=Invalid chunk at location 20530051 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:17:37.540352Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=9 error=Invalid chunk at location 20529393 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:18:17.353410Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=575 error=Invalid chunk at location 20531629 s-bucket 32771 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:18:45.386698Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=859 error=Invalid chunk at location 20531189 s-bucket 32770 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:17.863804Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=114 error=Invalid chunk at location 20529465 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:25.187593Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=648 error=Invalid chunk at location 20534151 s-bucket 32775 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:31.809430Z WARN libp2p_kad::behaviour: Failed to trigger bootstrap: No known peers. 2024-10-28T03:19:34.725782Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=308 error=Invalid chunk at location 20529600 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:42.094973Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=120 error=Invalid chunk at location 15396121 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:43.850573Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=133 error=Invalid chunk at location 20530120 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:47.487999Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=338 error=Invalid chunk at location 20529619 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:51.310132Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=234 error=Invalid chunk at location 20529545 s-bucket 32768 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:53.129580Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=84 error=Invalid chunk at location 15396100 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:19:58.533113Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=354 error=Invalid chunk at location 20530252 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:20:00.309980Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=837 error=Invalid chunk at location 20533671 s-bucket 32774 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:20:11.397075Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=355 error=Invalid chunk at location 20530253 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:20:47.089882Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=402 error=Invalid chunk at location 20530282 s-bucket 32769 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:21:27.083450Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::piece_reader: Failed to read piece from sector sector_index=6 piece_offset=311 error=Invalid chunk at location 15396239 s-bucket 24576 encoded true, possible disk corruption: Invalid scalar 2024-10-28T03:21:32.579313Z INFO subspace_farmer: end of block with defer!() and farm() calls 2024-10-28T03:21:32.579328Z INFO subspace_farmer: spawning watchdog 2024-10-28T03:21:32.579355Z INFO subspace_farmer: end of tokio::main() ```