search

waku-org / nwaku

Waku node and protocol.
Other
201 stars 52 forks source link

bug: intense execution of storage erase in waku-sync #3129

Closed Ivansete-status closed 4 weeks ago

Ivansete-status commented 1 month ago

Problem

Intense waku-sync operation that makes the node irresponsive

Impact

Other protocols, such as store cannot operate as usual.

To reproduce

Enable store-sync in status.staging fleet and then it appears an important number of logs with "storage erase failed"

Screenshots/logs

Evidence of the issue in Kibana: Image

https://kibana.infra.status.im/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2024-10-09T00:19:07.466Z',to:'2024-10-18T08:31:50.066Z'))&_a=(columns:!(logsource,message),filters:!(),hideChart:!f,index:ffcc22b0-0116-11ed-9719-cdd3b483481c,interval:auto,query:(language:kuery,query:'fleet:%20%22status.staging%22%20AND%20message:%20%22storage%20erase%20failed%22'),sort:!(!('@timestamp',desc)))

nwaku version/commit hash

v0.33.1

SionoiS commented 1 month ago

The errors are harmless. They come from trying to erase msgs we don't have.

As for the DB load, we could check if Waku Sync is doing more queries than needed (one every 5m max).

fryorcraken commented 1 month ago

I thought the issue was that all nodes were connecting to the same node (first in the list) to sync?

SionoiS commented 1 month ago

I thought the issue was that all nodes were connecting to the same node (first in the list) to sync?

Already fixed. here #3123

Ivansete-status commented 4 weeks ago

@SionoiS mentioned that this performance impact will be enhanced by waku-sync 2.0