nspcc-dev / neo-go

Go Node and SDK for the Neo blockchain
MIT License
123 stars 79 forks source link

Do not reupload existing blocks to NeoFS block storage #3652

Open AnnaShaleva opened 2 weeks ago

AnnaShaleva commented 2 weeks ago

Is your feature request related to a problem? Please describe.

Currently upload-bin finds latest incomplete batch of blocks and starts reuploading from the first block in this batch. Due to https://github.com/nspcc-dev/neo-go/issues/3647#issuecomment-2444172577 objects with the same block index but different OIDs may be created using this way of uploading.

Describe the solution you'd like

Don't upload the same block twice. Instead, search for the latest incomplete batch, then search for the missing block indexes in this batch and upload only missing blocks.

Describe alternatives you've considered

Leave it as is. It works, but a lot of useless duplicating objects are created which makes SEARCH request slower.

AliceInHunterland commented 1 day ago

By the way, we finally found duplicates at the end. example:

Chain block height: 6689772
Uploading index files...
Searching for blocks within the range 6680000 6690000, found 14742
Duplicate block 6686840 with oid saved 381yPLFRr9PN1wWKDiLmaR7QvjhxvvYNXFNo7KeGDPfb and new one 5sPVCKbNZwS27w1a61qDqPyXxNfEf4PKpgCvdmuwbisa

and objects differ with checksums - which means its different payloads (blocks):

(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object get --file ./res2.bin -r apollo7.morphbits.io:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid ANMHF1LcEhf4LsFQRCafst5PdHN1YTvhaZYk1EhLKMPH -t 60s --oid 92vfVNYQgF3gZyJyDKJDGhrkVHas2DYR6jdqSWMvumF7

 697 / 697 [=============================================================================================================================================================================] 100.00% 0s
[./res2.bin] Object successfully saved
ID: 92vfVNYQgF3gZyJyDKJDGhrkVHas2DYR6jdqSWMvumF7
CID: ANMHF1LcEhf4LsFQRCafst5PdHN1YTvhaZYk1EhLKMPH
Owner: NSqo6fy6uGRTLQUofh8VkB1r6G7VbMnFmM
CreatedAt: 1
Size: 697
HomoHash: 5bdefd06efbbf0c66003c05dcfaa340441bc09bb06c8d6f1f2ae1291b3fc42ff0dbad5dfa481ebeaa5fcd5f39732379d7cf962a9c33de229abb29ce4462344ae
Checksum: 7ba30f2ba341a7e55cf9d3a1f1334d19e3f75ec1199b39a4225d447c83a27487
Type: REGULAR
Attributes:
  Block=6686383
  Primary=4
  Hash=97d6402f58445fc6cff40c80ea46c52af26e2971d2ba26637b87a551c2565d8f
  PrevHash=0aed6b9903e34194759eee7953f2c471ca20aae46f1e999e9e4431bac2b11316
  Timestamp=1731570803990 (56841-04-17 20:59:50 +0300 MSK)
ID signature:
  public key: 037902671970564dae4366aab65fe1da930c41ce12b1908f99f3e7fa3843f3996d
  signature: 9a11fdd7997e0d0b2bfe21f93ca815b683d442c64b90da8c70ced5ac320dfe953fb1f490a52bd9d217628892d9d490738c6832e0306eaae9b28f2cdaf6e2b5a7
(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object get --file ./res1.bin -r apollo7.morphbits.io:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid ANMHF1LcEhf4LsFQRCafst5PdHN1YTvhaZYk1EhLKMPH -t 60s --oid H5U4i6PxYAwAZHVuEzFKJzs2gJMno6e8zxCqxDiJB2ia

 697 / 697 [=============================================================================================================================================================================] 100.00% 0s
[./res1.bin] Object successfully saved
ID: H5U4i6PxYAwAZHVuEzFKJzs2gJMno6e8zxCqxDiJB2ia
CID: ANMHF1LcEhf4LsFQRCafst5PdHN1YTvhaZYk1EhLKMPH
Owner: NSqo6fy6uGRTLQUofh8VkB1r6G7VbMnFmM
CreatedAt: 1
Size: 697
HomoHash: 48d156668021420a6e1dbe2a30986b701cccb098333b95976887623b7d6706173016f309d3c5b8dc6c8efe6fabda0fde66f6a62872d93a25b93b114fa45d0963
Checksum: 717166bec84eba2f53fde4bf6b6a400527b95996c79aee6f93adb7baf8a677ea
Type: REGULAR
Attributes:
  Block=6686383
  Primary=4
  Hash=97d6402f58445fc6cff40c80ea46c52af26e2971d2ba26637b87a551c2565d8f
  PrevHash=0aed6b9903e34194759eee7953f2c471ca20aae46f1e999e9e4431bac2b11316
  Timestamp=1731570803990 (56841-04-17 20:59:50 +0300 MSK)
ID signature:
  public key: 037902671970564dae4366aab65fe1da930c41ce12b1908f99f3e7fa3843f3996d
  signature: c989a391f3d0bb33e7b3ad93e5d41afce92957927190e0d1b001bfc57de13ea6b6792a632027e74b004b06bf9b4ec922a430e323ae85986004cb0f69d7a2c4b4