search

kadena-io / chainweb-data

Data ingestion for Chainweb.
BSD 3-Clause "New" or "Revised" License
14 stars 8 forks source link

[WIP] Optimize Backfill and Gaps #48

Closed emmanueldenloye closed 3 years ago

mightybyte commented 3 years ago

When I run this gaps it doesn't seem to feel all the gaps that it says exist.

2021-08-04T20:45:38.919Z [Info] [] Filling 9 gaps and 564 blocks
2021-08-04T20:45:38.919Z [Debug] [] Gaps to fill fromList [(0,[(1853768,1853774),(1854129,1854134),(1854489,1854494),(1854852,1854854),(1855205,1855214),(1855571,1855574),(1855930,1855934),(1856288,1856294),(1856337,1856870)])]
2021-08-04T20:45:39.378Z [Info] [] Filled in 392 missing blocks.

Then when I run it again right away I get this.

2021-08-04T20:47:56.474Z [Info] [] Filling 2 gaps and 178 blocks
2021-08-04T20:47:56.474Z [Debug] [] Gaps to fill fromList [(0,[(1856337,1856510),(1856863,1856870)])]
2021-08-04T20:47:56.706Z [Info] [] Filled in 174 missing blocks.

Comparing the two...

(1856337,1856870) -- from the first run
(1856337,1856510) -- from the second run
(1856863,1856870)

So somehow the whole gaps isn't getting filled. This violates an invariant that I would expect it to have.

mightybyte commented 3 years ago

The nested concurrent traversals here are pretty concerning. When I print the size of the inner structuring we're traversing over concurrently for an empty DB I get this:

2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 2797 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges
2021-08-05T13:38:26.860Z [Debug] [] Traversing concurrently over 5164 ranges

So it seems like this will be creating something along the lines of 80,000 threads. Even though these are lightweight green threads, that seems like WAY too many. I would say a fixed configurable number of worker threads or a simple one thread per chain would make more sense to me.