fluree / db

Fluree database library
https://fluree.github.io/db/
Other
330 stars 21 forks source link

Feature/async db record #779

Closed zonotope closed 1 month ago

zonotope commented 1 month ago

This patch adds 3 new protocols which are meant to describe database interactions and will eventually replace the IFlureeDb protocol, implementations for those protocols for the JsonLdDb record, and a new AsyncDB record implementing those protocols that will eventually be used for index updates.

The first new protocol is a fluree.db.query.exec.where/Matcher protocol to match variables in solutions when executing where clause searches. The second is a fluree.db.json-ld.response/NodeFormatter protocol to format the subject nodes of iris during graph crawls, and the third is a fluree.db.json-ld.transact/Transactable protocol to stage new transactions.

There is a lot more cleanup work to be done to reorganize the namespaces and code to fully remove the IFlureeDb protocol while avoiding cyclic namespace dependencies, and the new Transactable isn't as precise as it could be as it would be best if it were fully isolated from the Matcher protocol. It depends on that protocol for now because of the way some of the reasoner code is organized. I will do that cleanup and reorganization in a subsequent pr, but I put this forward for now because it was the minimum necessary to unlock the index queue work which will come next.

zonotope commented 1 month ago

I see a lot of good refactoring going on here 👍

So, here's how I understand this PR: it's mainly moving things around in anticipation of being able to use the AsyncDB record for indexing, though in this PR we are not yet using it.

Will we start using AsyncDB in place of JsonLdDb everywhere?

Not quite. They should be interchangeable and, if we get the protocols right, we won't notice which one we're actually using.

The motivation behind async db is so we can transparently load a new db in the background while still returning a usable db record immediately. That will allow us to process commits and index updates immediately instead of asynchronously since we don't have to wait for new dbs to load from commits/index updates, and that should eliminate a number of race conditions we've experienced.

But it shouldn't matter if we're using an AsyncDB if the protocols are correct. The -stage method implementation for AsyncDB returns a normal JsonLdDb for example.