search

nspcc-dev / neo-go

Go Node and SDK for the Neo blockchain
MIT License
123 stars 79 forks source link

`upload-bin` fails to upload index files #3658

Closed AnnaShaleva closed 4 days ago

AnnaShaleva commented 2 weeks ago 

2024-11-02 11:03:54.592 Chain block height: 6284055
2024-11-02 11:03:56.388 Chain block height: 4902100
2024-11-02 11:03:58.730 Chain block height: 6621405
2024-11-02 11:04:01.577 Chain block height: 4504542
2024-11-02 11:06:50.445 Uploading index files...
2024-11-02 11:06:50.445 First block of latest incomplete batch uploaded to NeoFS container: 6250000
2024-11-02 11:06:56.839 Uploading index files...
2024-11-02 11:06:56.839 First block of latest incomplete batch uploaded to NeoFS container: 4460000
2024-11-02 11:07:49.082 Uploading index files...
2024-11-02 11:07:49.082 First block of latest incomplete batch uploaded to NeoFS container: 4870000
2024-11-02 11:08:53.590 Uploading index files...
2024-11-02 11:08:53.590 First block of latest incomplete batch uploaded to NeoFS container: 6590000
2024-11-02 13:13:45.533 Index file 0 generated, checking for the missing blocks...
2024-11-02 13:13:45.562 0 missing block(s) processed for index file 0, uploading index file...
2024-11-02 13:13:52.846 Index file 0 generated, checking for the missing blocks...
2024-11-02 13:13:57.482 Uploaded index file 0
2024-11-02 13:14:13.310 Index file 0 generated, checking for the missing blocks...
2024-11-02 13:14:19.667 failed to upload index files: block 89052 is missing from the storage
2024-11-02 13:14:22.703 Index file 0 generated, checking for the missing blocks...
2024-11-02 13:14:40.243 failed to upload index files: block 91326 is missing from the storage
2024-11-02 13:15:21.224 Chain block height: 6284540
2024-11-02 13:15:42.791 Chain block height: 4505042
2024-11-02 13:15:50.891 failed to upload index files: block 27475 is missing from the storage
2024-11-02 13:16:52.950 Chain block height: 4902596
2024-11-02 13:18:49.521 Uploading index files...

Blocks 91326 (FS testnet), 89052 (mainnet) and 27475 (testnet) can't be found in the storage. Check what's wrong with them or is it just a small number of retries.

AnnaShaleva commented 2 weeks ago

Within the scope of this issue let's move maxRetries constant to the CLI parameter.

AliceInHunterland commented 2 weeks ago

Block 91326 (FS testnet) exisits.:

(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object search --ttl 1 -r st2.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --filters 'Block EQ 91326' --cid FidCcVzUxzY3PeUksCnnpUQ8h8PJeYPessX8FRMuveSy

Found 1 objects.
HwkGuEX5bra2et7pm2CD2fs4vi9oqfrSrKjkX4GBc6m5
AliceInHunterland commented 2 weeks ago

89052 (mainnet) - not found.

(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object search --ttl 1 -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --filters 'Block EQ 89052' --cid 7a1cn9LNmAcHjESKWxRGG7RSZ55YHJF6z2xDLTCuTZ6c -t 60s

Found 0 objects.
(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object search --ttl 1 -r st2.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --filters 'Block EQ 89052' --cid 7a1cn9LNmAcHjESKWxRGG7RSZ55YHJF6z2xDLTCuTZ6c -t 60s

Found 0 objects.
(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object search --ttl 1 -r st3.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --filters 'Block EQ 89052' --cid 7a1cn9LNmAcHjESKWxRGG7RSZ55YHJF6z2xDLTCuTZ6c -t 60s

Found 0 objects.
(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object search --ttl 1 -r st4.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --filters 'Block EQ 89052' --cid 7a1cn9LNmAcHjESKWxRGG7RSZ55YHJF6z2xDLTCuTZ6c -t 60s

Found 0 objects.
AnnaShaleva commented 2 weeks ago

Block 91326 (FS testnet) exisits.:

Can you GET/HEAD it?

89052 (mainnet) - not found.

What if try to request more times?

AliceInHunterland commented 2 weeks ago

Can you GET/HEAD it?

i dont know OID, so I cant

AliceInHunterland commented 2 weeks ago

What if try to request more times?

Yet nothing

AnnaShaleva commented 2 weeks ago

Block 91326 (FS testnet) exisits.:

i dont know OID, so I cant

You have it, it's HwkGuEX5bra2et7pm2CD2fs4vi9oqfrSrKjkX4GBc6m5. The question is about existing object.

AliceInHunterland commented 2 weeks ago

HwkGuEX5bra2et7pm2CD2fs4vi9oqfrSrKjkX4GBc6m5

with this one everything is okay, its correct and returned. GET/HEAD several times from different nodes and with/without --ttl 1

AliceInHunterland commented 2 weeks ago

For --cid 7a1cn9LNmAcHjESKWxRGG7RSZ55YHJF6z2xDLTCuTZ6c checked from 0 to 350k received such results. (96 blocks)

Block 89052 is missing from the storage
Block 89131 is missing from the storage
Block 89151 is missing from the storage
Block 89152 is missing from the storage
Block 89160 is missing from the storage
Block 89171 is missing from the storage
Block 89180 is missing from the storage
Block 89238 is missing from the storage
Block 89260 is missing from the storage
Block 89263 is missing from the storage
Block 89278 is missing from the storage
Block 89280 is missing from the storage
Block 89318 is missing from the storage
Block 89323 is missing from the storage
Block 89327 is missing from the storage
Block 89329 is missing from the storage
Block 89332 is missing from the storage
Block 89333 is missing from the storage
Block 89338 is missing from the storage
Block 89352 is missing from the storage
Block 89373 is missing from the storage
Block 89380 is missing from the storage
Block 89385 is missing from the storage
Block 89389 is missing from the storage
Block 89392 is missing from the storage
Block 89398 is missing from the storage
Block 89400 is missing from the storage
Block 89408 is missing from the storage
Block 89413 is missing from the storage
Block 89443 is missing from the storage
Block 89445 is missing from the storage
Block 89493 is missing from the storage
Block 89494 is missing from the storage
Block 89500 is missing from the storage
Block 89503 is missing from the storage
Block 89520 is missing from the storage
Block 89533 is missing from the storage
Block 89547 is missing from the storage
Block 89553 is missing from the storage
Block 89554 is missing from the storage
Block 89558 is missing from the storage
Block 89560 is missing from the storage
Block 89567 is missing from the storage
Block 89580 is missing from the storage
Block 89585 is missing from the storage
Block 89586 is missing from the storage
Block 89589 is missing from the storage
Block 89598 is missing from the storage
Block 89599 is missing from the storage
Block 89600 is missing from the storage
Block 89606 is missing from the storage
Block 89628 is missing from the storage
Block 89633 is missing from the storage
Block 89636 is missing from the storage
Block 89638 is missing from the storage
Block 89645 is missing from the storage
Block 89655 is missing from the storage
Block 89664 is missing from the storage
Block 89668 is missing from the storage
Block 89674 is missing from the storage
Block 89685 is missing from the storage
Block 89687 is missing from the storage
Block 89693 is missing from the storage
Block 89694 is missing from the storage
Block 89698 is missing from the storage
Block 89708 is missing from the storage
Block 89709 is missing from the storage
Block 89716 is missing from the storage
Block 89725 is missing from the storage
Block 89733 is missing from the storage
Block 89739 is missing from the storage
Block 89745 is missing from the storage
Block 89746 is missing from the storage
Block 89749 is missing from the storage
Block 89756 is missing from the storage
Block 89802 is missing from the storage
Block 89804 is missing from the storage
Block 89817 is missing from the storage
Block 89824 is missing from the storage
Block 89839 is missing from the storage
Block 89846 is missing from the storage
Block 89858 is missing from the storage
Block 89861 is missing from the storage
Block 89862 is missing from the storage
Block 89864 is missing from the storage
Block 89884 is missing from the storage
Block 89897 is missing from the storage
Block 89901 is missing from the storage
Block 89921 is missing from the storage
Block 89924 is missing from the storage
Block 89956 is missing from the storage
Block 89958 is missing from the storage
Block 89959 is missing from the storage
Block 89961 is missing from the storage
Block 89966 is missing from the storage
Block 89997 is missing from the storage
AliceInHunterland commented 2 weeks ago 

2024-11-05 13:39:12.765 | Uploaded index file 26 |  
-- | -- | --
  |   | 2024-11-05 13:39:11.254 | 0 missing block(s) processed for index file 26, uploading index file... |  
  |   | 2024-11-05 13:39:11.228 | Index file 26 generated, checking for the missing blocks... |  
  |   | 2024-11-05 12:46:05.259 | Uploading index files... |  
  |   | 2024-11-05 12:46:05.259 | First block of latest incomplete batch uploaded to NeoFS container: 6590000 |  
  |   | 2024-11-05 12:43:02.745 | Chain block height: 6638208 |  
  |   | 2024-11-05 12:41:59.568 | failed to upload index files: block 3349950 is missing from the storage |  
  |   | 2024-11-05 06:40:39.327 | Index file 26 generated, checking for the missing blocks... |  
  |   | 2024-11-05 05:17:15.621 | Uploading index files... |  
  |   | 2024-11-05 05:17:15.621 | First block of latest incomplete batch uploaded to NeoFS container: 6580000 |  
  |   | 2024-11-05 05:12:19.370 | Chain block height: 6636493 |  
  |   | 2024-11-05 05:10:47.285 | failed to upload index files: block 3378959 is missing from the storage |  
  |   | 2024-11-05 00:53:58.817 | Index file 26 generated, checking for the missing blocks... |  
  |   | 2024-11-05 00:07:24.780 | Uploaded index file 25

After 3 times all blocks were found. Depends on https://github.com/nspcc-dev/neofs-node/issues/2721.

AnnaShaleva commented 2 weeks ago

For the record: some blocks are really missing from NeoFS. We have https://github.com/nspcc-dev/neo-go/pull/3660/commits/18ed3bb3e8e75efceef832c050d7e6c15a3e53d2 that should solve this issue, but we're not 100% sure if it helps, hence let's keep this issue open, test the updated uploading script on mainnet one more time and check if it solves the problem with gaps in blocks uploading.

AliceInHunterland commented 1 week ago

It seems to be still relevant, based on new uploader runs:

2024-11-11 07:18:23.373 | failed to upload index files: index file 0: block 1618 is missing from the storage |  
2024-11-11 07:17:54.825 | Index file 0: fetching missing block 1618
roman-khimov commented 1 week ago

How does it really happen? You're uploading an object for block 1618 and... get an error? get a successful reply? don't get any reply? Can we get more data from logs in this case?

AliceInHunterland commented 1 week ago

How does it really happen? You're uploading an object for block 1618 and... get an error? get a successful reply? don't get any reply? Can we get more data from logs in this case?

From the logs, it was successfully uploaded, we didn't have any related errors, that could explain it.

roman-khimov commented 1 week ago

And we don't know its OID? I think we need some verbose mode with more data in this case to trace the problem. Seems like it happens often enough and we don't need to wait for full upload cycle to complete.

AliceInHunterland commented 1 week ago

And we don't know its OID? I think we need some verbose mode with more data in this case to trace the problem. Seems like it happens often enough and we don't need to wait for full upload cycle to complete.

No OID. https://github.com/nspcc-dev/neo-go/issues/3655 will be as a verbose mode - it will put OIDs directly into the index file. Or we can try to write everything in logs to see it faster -200k blocks will be enough to catch.

roman-khimov commented 1 week ago

Log line is much easier to add, let's add it and then try a new upload, see where it fails and then take a look at it from the NeoFS side. Either it loses an object that was accepted (bad), or it reports an incorrect status for failed upload (bad). Or something else. Maybe related to https://github.com/nspcc-dev/neofs-node/issues/2975, maybe not.

AnnaShaleva commented 4 days ago

The problem is on NeoFS side, some objects are silently failed to be uploaded. Fixed by https://github.com/nspcc-dev/neofs-node/pull/3014, hence I consider this issue as not planned, no actions required from NeoGo side.