lightninglabs / taproot-assets

A layer 1 daemon, for the Taproot Assets Protocol specification, written in Go (golang)
MIT License
463 stars 111 forks source link

[enhancement]: optimize performance of DB: batched writes (sqlite), concurrent writes (postgres) #417

Open dstadulis opened 1 year ago

dstadulis commented 1 year ago

Background


To prevent previous dysfunction, write locks were added in https://github.com/lightninglabs/taproot-assets/commit/b190d32087a3d579fbfca4cd06548d59d0b46694#diff-34ab0b3a4593407088a7516e921f22607e4237f236325db502804e33433327b2R303 which ultimately won't be necessary once #261 is fixed. After which, DB writes can be optimized with batched writes (sqlite) or concurrent writes (postgres)

Potential parameters to be chose / Design criteria


Given sqlite, at what which point should the batch write cut off e.g. How long should write requests be collected for the ultimate batched write?

Given postgres,

What interactions are there with the MS-SMT tree level structure and SIMD processing that we might want to leverage for highly paralizable pocket universe load balancer design architectures?

Related issues: https://github.com/lightninglabs/taproot-assets/issues/374

Roasbeef commented 1 year ago

I think let's target this at using the batched postgers queries for things like reading the SMT levels. I think we can modify the recursive call to fetch all the siblings at once.