search

SaitoTech / saito-rust-workspace

Cargo workspace which contains all the rust projects for Saito
4 stars 3 forks source link

Syncing blocks from peer fails. #629

Closed arpee closed 9 months ago

arpee commented 11 months ago

To replicate:

Have the network running with more than a thousand or so blocks in the chain.

Get a clean node (in this case the services node after the reset script is run) and try to sync it.

Some blocks will sync - but after a period the bellow begins appearing in the logs. (This may be caused by timeoutes on the node being synced from).

When this error appears - the node stops trying to sync blocks and eventually forks.

Expected behaviour:

log output:

0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116
0|npm start  | 2023-12-17T23:04:56: failed verifying block buffer with length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/data/block.rs:604
0|npm start  | 2023-12-17T23:04:56: block buffer is invalid to read transaction data. length : 539
0|npm start  | 2023-12-17T23:04:56: WARN saito-core/src/core/verification_thread.rs:116

Edit:

I removed rate limiting on the main node and the above still happened - not main node had topped out at 100% memory usage at this point.


![Image](https://github.com/SaitoTech/saito-rust-workspace/assets/2867779/ffb82d56-58c7-47d6-85d5-9cc358fbfd14)
arpee commented 11 months ago

CORRECTION

I had not removed the rate limiting. When I do - the nodes can sync.

SankaD commented 9 months ago

Couldn't recreate