search

fluree / db

Fluree database library
https://fluree.github.io/db/
Other
330 stars 21 forks source link

Indexing process state management #771

Closed bplatz closed 3 weeks ago

bplatz commented 1 month ago

This is a temporary fix to solve the problem of bulk loads under high server demand. It should be replaced by work being done by @zonotope which eliminates ever attempting to update a commit file.

If an index finishes, it attempts to update the latest commit. In a high load scenario, by the time it can update the latest commit there is a new commit. This has some bad effects.

We now have a temporary atom state that maintains the latest index, the two main process changes are: 1) Upon index completion, that index atom is updated 2) When writing new commits, that index atom is always consulted for the latest index files 3) When updating an existing commit with a new index, before writing a check is performed if there have been any newer commits and if so, it no longer will attempt an update at all

While this gets things working fairly reliably, there are still some race conditions that exist and this is not by iteself a permanent fix. At the point we no longer try to update commits it is irrelevant however.

bplatz commented 3 weeks ago

Closing because main now has features that address this isssue.