lightninglabs / taproot-assets

A layer 1 daemon, for the Taproot Assets Protocol specification, written in Go (golang)
MIT License
463 stars 111 forks source link

proofs+tarodb: stop storing proofs files in the DB switch to mmap access to on disk files everywhere #147

Open Roasbeef opened 2 years ago

Roasbeef commented 2 years ago

Today we store proofs in the DB, mainly for simplicity and to ensure we can atomically update proofs when we modify assets. IMO what we really want to do is only store them on disk, them use memory-mapping to access the proofs over contained memory regions. This means we minimize copying, and can pass around pointers in memory to the relevant mmap'd region. On top of this, we'll either want to maintain an external file w/ an index, or maintain the index within the database itself.

dstadulis commented 1 year ago

"memory mapping the subdirectory" would enable the path be an S3 bucket