search

cppalliance / NuDB

NuDB: A fast key/value insert-only database for SSD drives in C++11
Boost Software License 1.0
384 stars 59 forks source link

Sporadic Fetch Failures (Multi-Process Support ?) #72

Open movitto opened 5 years ago

movitto commented 5 years ago

Greetings! I've been exploring & diving into this library into the context of reading data directly from a live rippled nodestore. Periodically a fetch operation will sporadically fail with the 'key_not_found' error. Subsequently trying to fetch the same key using a new database connection succeeds. I'm 100% sure the values exist in the DB before starting this process.

After exploring the code my primary thought as to the cause of this is a race condition, where the rippled process is flushing data to the nodestore file just as I'm trying to read the bucket containing the data. This could result in a lookup failure, though admittedly I haven't been able to pinpoint this as being the exact cause. Subsequent lookups of the same key using the same database connection fail until it is closed / reopened, leading me to believe internal in-memory cache buckets are getting out of sync (or similar). Does this sound like it could be the cause of this issue or if not any idea on what could be the problem?

If this is the issue, care would need to be taken to support multi-process access for this to be resolved. Sqlite3 employs on-disk locking solutions to facilitate this, but even there that is less than ideal. Would there be any interest in adding something similar to this library? For the time being, we can live with closing/reopening the db on failure, but it'd be nice to utilize something more robust!

Appreciate any insights.

vinniefalco commented 5 years ago

Hmm... nudb assumes that it has exclusive access to the file. Accessing any of the database files from more than one process at the same time, or from two separate instances of store in the same process, results in undefined behavior.

movitto commented 5 years ago

Gotchya, thanks for the info @vinniefalco. I figured such as I didn't see anything in the code that would sanitize multi-process access. I'll look into how sqlite3 implements this internally and perhaps if something similar could be implemented in NuDB.

vinniefalco commented 5 years ago

The most sane way to do it would be to modify rippled and use the existing instance of the nudb::store to perform your fetches.