search

nspcc-dev / neo-go

Go Node and SDK for the Neo blockchain
MIT License
123 stars 79 forks source link

BlockUploader fails due to timeouts #3620

Closed AnnaShaleva closed 1 month ago

AnnaShaleva commented 1 month ago

Current Behavior

Oct 17 10:57:34 kaede systemd[1]: Stopped neogo-block-fetcher-testnet.service - NeoGo Block Fetcher testnet.
Oct 17 10:57:34 kaede systemd[1]: Started neogo-block-fetcher-testnet.service - NeoGo Block Fetcher testnet.
Oct 17 10:57:35 kaede neogo-block-fetcher[2991350]: Chain block height: 4816438
Oct 17 10:58:11 kaede neogo-block-fetcher[2991350]: failed to fetch the latest missing block index from container: search of index files failed for batch with indexes from 4820000 to 4830000: error during object IDs iteration: context deadline exceeded
Oct 17 10:58:11 kaede systemd[1]: neogo-block-fetcher-testnet.service: Main process exited, code=exited, status=1/FAILURE
Oct 17 10:58:11 kaede systemd[1]: neogo-block-fetcher-testnet.service: Failed with result 'exit-code'.

And also https://github.com/nspcc-dev/neo-go/pull/3582#discussion_r1782183181 and https://github.com/nspcc-dev/neo-go/pull/3582#discussion_r1792899198

Expected Behavior

Timeouts themselves are quite native, but they break the application and seamless script work is impossible due to timeouts. They must be customizable and we should now where exactly they happen to be able to increase them.

Possible Solution

Find the reason. Increase timeout or move it to the command setting.

AnnaShaleva commented 1 month ago

@AliceInHunterland need to check this: https://github.com/nspcc-dev/neo-go/blob/14ea5a8d3271df25394c6fc5aac84f1bf8398264/cli/util/uploader.go#L85

It may be too strict default pool options.

AnnaShaleva commented 1 month ago

Object search initialization always passes, but the iteration itself fails. May be it's something connected with stream timeout.

AliceInHunterland commented 1 month ago

Also got timeout at approximately same time

(base) ekaterinapavlova@MacBook-Air-4 neo-go % ./bin/neo-go util upload-bin --cid HmP48JvCcmiEbJemG7jFNEg2qkdCTx4thD3EwWUi288Q --wallet-config ./wallet-config.yml --block-attribute block --index-attribute index --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.storage.fs.neo.org:8080 -fsr st2.storage.fs.neo.org:8080 -fsr st3.storage.fs.neo.org:8080 -fsr st4.storage.fs.neo.org:8080

Chain block height: 384000
failed to fetch the latest missing block index from container: search of index files failed for batch with indexes from 390000 to 400000: error during object IDs iteration: context deadline exceeded
(base) ekaterinapavlova@MacBook-Air-4 neo-go % ./bin/neo-go util upload-bin --cid HmP48JvCcmiEbJemG7jFNEg2qkdCTx4thD3EwWUi288Q --wallet-config ./wallet-config.yml --block-attribute block --index-at
tribute index --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.storage.fs.neo.org:8080 -fsr st2.storage.fs.neo.org:8080 -fsr st3.storage.fs.neo.org:8080 -fsr st4.storage.fs.neo.org:8080 -fs
r grpcs://st1.storage.fs.neo.org:8082 

Chain block height: 384000
failed to fetch the latest missing block index from container: search of index files failed for batch with indexes from 390000 to 400000: error during object IDs iteration: context deadline exceeded
AliceInHunterland commented 1 month ago 
(base) ekaterinapavlova@MacBook-Air-4 neo-go %  ./bin/neo-go util upload-bin --cid HmP48JvCcmiEbJemG7jFNEg2qkdCTx4thD3EwWUi288Q --wallet-config ./wallet-config.yml --block-attribute block --index-a
ttribute index --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.storage.fs.neo.org:8080 -fsr st2.storage.fs.neo.org:8080 -fsr st3.storage.fs.neo.org:8080 -fsr st4.storage.fs.neo.org:8080 -f
sr grpcs://st1.storage.fs.neo.org:8082 -fsr grpcs://st2.storage.fs.neo.org:8082 -fsr grpcs://st3.storage.fs.neo.org:8082 -fsr grpcs://st4.storage.fs.neo.org:8082

Chain block height: 4818652
Retrying after error: error during object IDs iteration: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type "text/html"
Retrying after error: error during object IDs iteration: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type "text/html"
Retrying after error: error during object IDs iteration: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html"
AliceInHunterland commented 1 month ago 
(base) ekaterinapavlova@MacBook-Air-4 neofs-node %  ./bin/neofs-cli object search -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV --filters 'block GE 400000' --filters 'block LE 420000'

rpc error: search objects using NeoFS API: rpc error: code = DeadlineExceeded desc = context deadline exceeded
(base) ekaterinapavlova@MacBook-Air-4 neofs-node %  ./bin/neofs-cli object search -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV --filters 'block GE 400000' --filters 'block LE 420000'

rpc error: search objects using NeoFS API: rpc error: code = DeadlineExceeded desc = context deadline exceeded
(base) ekaterinapavlova@MacBook-Air-4 neofs-node %  ./bin/neofs-cli object search -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV --filters 'block GE 400000' --filters 'block LE 420000'

rpc error: search objects using NeoFS API: rpc error: code = DeadlineExceeded desc = context deadline exceeded
(base) ekaterinapavlova@MacBook-Air-4 neofs-node % ./bin/neofs-cli object head -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV --oid DkAsCdWttqtbWkLb6ma7XnRiS9SUDAwwcSTXG5y26dft

ID: DkAsCdWttqtbWkLb6ma7XnRiS9SUDAwwcSTXG5y26dft
CID: 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV
Owner: NhNT6EFqrDu7hAsv2V1J3B7J1DKLvkNGRg
CreatedAt: 26403
Size: 697
HomoHash: 43b3023152b3dfe8c79b7bb296fc227679fa8dfaf782594d34933877b568984311d559814489fa49db8d5fccf50660143909323d516776c1c717532150cb1cb3
Checksum: 2f1fd11816ef03d19b5f89efd8760184eea0e52a95383b6816f99970e6ad16c3
Type: REGULAR
Attributes:
  block=399999
ID signature:
  public key: 02d89e1e06504a2ec3410a9ea801acfa3e45ea5d9ad3fa56ea179638a6dd72b7d2
  signature: 8f37659c42f021f1060f3055062552cee5aceddc8ca00a4b984b52a9984a6337e3ecf087bc4c26c7e1f4210c99bb14301dd3372f49a491f2b80b254b7f6a2d9f
(base) ekaterinapavlova@MacBook-Air-4 neofs-node %  ./bin/neofs-cli object search -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfL
UYYBv1ZNzPG5w7Ra9U1jQgasVayV --filters 'block EQ 399999' 

rpc error: search objects using NeoFS API: rpc error: code = DeadlineExceeded desc = context deadline exceeded
(base) ekaterinapavlova@MacBook-Air-4 neofs-node %  ./bin/neofs-cli object search -r st1.storage.fs.neo.org:8080 -w /Users/ekaterinapavlova/Workplace/neo-go/panelwallet1.json --cid 5zqTAP24qFHp6LfLUYYBv1ZNzPG5w7Ra9U1jQgasVayV --filters 'block EQ 399999'  -t 60s

Found 1 objects.
DkAsCdWttqtbWkLb6ma7XnRiS9SUDAwwcSTXG5y26dft