Open AnnaShaleva opened 4 hours ago
One more example:
2024-10-24 06:10:41.740
Successfully uploaded batch of blocks: from 1960000 to 1969999
2024-10-24 06:14:17.712 Processing batch from 1980000 to 1989999
2024-10-24 06:14:17.712 Successfully uploaded batch of blocks: from 1970000 to 1979999
2024-10-24 06:18:11.192 Processing batch from 1990000 to 1999999
2024-10-24 06:18:11.192 Successfully uploaded batch of blocks: from 1980000 to 1989999
2024-10-24 06:21:47.636 Processing batch from 2000000 to 2009999
2024-10-24 06:21:47.636 Successfully uploaded batch of blocks: from 1990000 to 1999999
2024-10-24 06:25:07.098 Processing batch from 2010000 to 2019999
2024-10-24 06:25:07.098 Successfully uploaded batch of blocks: from 2000000 to 2009999
2024-10-24 06:29:13.781 Processing batch from 2020000 to 2029999
2024-10-24 06:29:13.781 Successfully uploaded batch of blocks: from 2010000 to 2019999
2024-10-24 06:33:18.434 Processing batch from 2030000 to 2039999
2024-10-24 06:33:18.434 Successfully uploaded batch of blocks: from 2020000 to 2029999
2024-10-24 06:36:05.190 upload error: failed to initiate object upload: connection: no healthy client
2024-10-24 06:37:08.145 Chain block height: 6235291
2024-10-24 06:54:58.627 Processing batch from 0 to 9999
2024-10-24 06:54:58.627 First block of latest incomplete batch uploaded to NeoFS container: 0
2024-10-24 07:00:32.594 Processing batch from 10000 to 19999
2024-10-24 07:00:32.594 Successfully uploaded batch of blocks: from 0 to 9999
2024-10-24 07:05:19.672 Processing batch from 20000 to 29999
2024-10-24 07:05:19.672 Successfully uploaded batch of blocks: from 10000 to 19999
2024-10-24 07:08:32.479 Processing batch from 30000 to 39999
2024-10-24 07:08:32.479 Successfully uploaded batch of blocks: from 20000 to 29999
2024-10-24 07:10:25.644 Processing batch from 40000 to 49999
2024-10-24 07:10:25.644 Successfully uploaded batch of blocks: from 30000 to 39999
2024-10-24 07:10:48.870 upload error: failed to initiate object upload: connection: no healthy client
2024-10-24 07:11:50.608 Chain block height: 6235419
But sometimes it works differently (logs are from the same mainnet service):
2024-10-24 06:37:08.145 Chain block height: 6235291
2024-10-24 06:54:58.627 Processing batch from 0 to 9999
2024-10-24 06:54:58.627 First block of latest incomplete batch uploaded to NeoFS container: 0
2024-10-24 07:00:32.594 Processing batch from 10000 to 19999
2024-10-24 07:00:32.594 Successfully uploaded batch of blocks: from 0 to 9999
2024-10-24 07:05:19.672 Processing batch from 20000 to 29999
2024-10-24 07:05:19.672 Successfully uploaded batch of blocks: from 10000 to 19999
2024-10-24 07:08:32.479 Processing batch from 30000 to 39999
2024-10-24 07:08:32.479 Successfully uploaded batch of blocks: from 20000 to 29999
2024-10-24 07:10:25.644 Processing batch from 40000 to 49999
2024-10-24 07:10:25.644 Successfully uploaded batch of blocks: from 30000 to 39999
2024-10-24 07:10:48.870 upload error: failed to initiate object upload: connection: no healthy client
2024-10-24 07:11:50.608 Chain block height: 6235419
2024-10-24 07:18:58.201 failed to fetch the latest missing block index from container: search of index files failed for batch with indexes from 2480000 to 2489999: failed to initiate object search: session: init session: status: code = 1024 message = connection to the RPC node has been lost
2024-10-24 07:20:01.123 Chain block height: 6235449
2024-10-24 07:29:07.555 Processing batch from 40000 to 49999
2024-10-24 07:29:07.555 First block of latest incomplete batch uploaded to NeoFS container: 40000
2024-10-24 07:29:45.742 Processing batch from 50000 to 59999
2024-10-24 07:29:45.742 Successfully uploaded batch of blocks: from 40000 to 49999
2024-10-24 07:30:14.433 Processing batch from 60000 to 69999
2024-10-24 07:30:14.433 Successfully uploaded batch of blocks: from 50000 to 59999
2024-10-24 07:31:04.556 Processing batch from 70000 to 79999
2024-10-24 07:31:04.556 Successfully uploaded batch of blocks: from 60000 to 69999
2024-10-24 07:31:42.554 upload error: failed to initiate object upload: connection: no healthy client
2024-10-24 07:32:44.920 Chain block height: 6235496
can be connected #3615
Well, of course it's a bug in fetchLatestMissingBlockIndex only if our batches are full and don't have gaps.
can be connected https://github.com/nspcc-dev/neo-go/issues/3615
Check currently uploaded data for N3 mainet. See if there are gaps in batches.
Checked, depends on #3615 resolution.
To resolve this issue, we need to adopt the SEARCH completeness marker once https://github.com/nspcc-dev/neofs-node/issues/2721 implemented.
Current Behavior
Some blocks are uploaded to NeoFS, then restart of the script happens. The script starts to upload from 0 block, not from the latest incomplete batch:
The pattern repeats.
Expected Behavior
Reupload must happen starting from latest incomplete batch.
Possible Solution
Find the problem in
fetchLatestMissingBlockIndex
, fix it.