search

Granola-Team / mina-indexer

The Mina Indexer is a re-imagined version of the software collectively called the "Mina archive node."
Apache License 2.0
19 stars 10 forks source link

Problem: `mina-indexer server start` fails to report progress on large startup_dir #538

Closed robinbb closed 7 months ago

robinbb commented 8 months ago

The above command fails to report progress (seems to hang, but consumes CPU) after a few log file entries, ending in the following lines:

2024-03-06T00:58:18.496223Z DEBUG mina_indexer::state: Block not added: db: already seen block (100271, 3NLT52sQ2n4ZR2J1kADKUE773o24HTZkTQYQEED2ftv6bw1Udqxg)
2024-03-06T00:58:18.598968Z DEBUG mina_indexer::state: Block not added: db: already seen block (100272, 3NK6DE5DvwjbW6w664piDGt4MJjFfynKHxt46WgKmrE6JS9c47tG)
2024-03-06T00:58:18.600217Z DEBUG mina_indexer::state: Block not added: db: already seen block (100272, 3NKTjwEbo3YMk8G2AsAopkoTGCdoEK5E7tyDaqJvd4eTYf5AHs4m)
2024-03-06T00:58:18.744043Z DEBUG mina_indexer::state: Block not added: db: already seen block (100272, 3NKkY2QKxkAbuuowqaDB5wxRJXzwUYZ32iNKHfbmgXWPBpmGWcXQ)
2024-03-06T00:58:18.887186Z DEBUG mina_indexer::state: Block not added: db: already seen block (100272, 3NKnbPUaDp1UZsYksA13ogNyXY4RxFTwmFhtByfHHZPb8Q3egnz3)
2024-03-06T00:58:18.990305Z DEBUG mina_indexer::state: Block not added: db: already seen block (100272, 3NL2zjNNFTWHZpGe4qZNwXkgABG6Cy3AzLK6ExnkiRCREUBHcsub)
2024-03-06T00:58:19.137507Z DEBUG mina_indexer::state: Block not added: db: already seen block (100273, 3NKPTM8e8C4Mt7BQykYwiuSr2KkQjdy9MhepRCdhGFLwPfzYaG5w)
2024-03-06T00:58:19.288149Z DEBUG mina_indexer::state: Block not added: db: already seen block (100273, 3NKUaab3R1mcG9caqyUUntpjvYD7VYUJ7rWjTCbgxzob3miG3WHK)
2024-03-06T00:58:19.438165Z DEBUG mina_indexer::state: Block not added: db: already seen block (100273, 3NLHTdvTPXxUn8YFy4z59NxDcX9DYhthFtv8aPNMmpm2pYuA6Tf6)

It takes 746 seconds before any further output appears. That is too long. The users needs to see output - what is being processed.

jhult commented 8 months ago

I will take this one.

@robinbb, is there a server I can test this with (that already has a lot of blocks downloaded)?

Also, "the above command" seems to be missing.

robinbb commented 8 months ago

@jhult Actually, I'm still diagnosing. I'll correct the "above command", too...

robinbb commented 8 months ago

... I added --log-level-file TRACE and --ledger-cadence 1000 and I'm now getting much better results...

robinbb commented 8 months ago

... changed number of open files with ulimit -n 4096 and trying again ....

Isaac-DeFrain commented 7 months ago

Any further progress on this one @robinbb?

robinbb commented 7 months ago

Yes, apologies for lack of updates. Trevor added some logging, but we still do not log any message when we are waiting for Speedb to reorg itself, which takes many hours. So, the problem is less bad, but still bad.

Isaac-DeFrain commented 7 months ago

Should we defer to more specific issues and close this one?

robinbb commented 7 months ago

Yes, this is now a dup of #608