search

3-Round-Stones / callimachus

Callimachus is a highly scalable platform for creating and running data-driven websites
Other
95 stars 24 forks source link

Move AliBaba Optimistic SAIL into Sesame Memory/Native Stores #218

Closed catch-point closed 8 years ago

catch-point commented 9 years ago

Callimachus currently uses AliBaba's Optimistic SAIL module to support concurrent transactions (absolutely necessary in a Web context). However, the current implementation cannot be extended to support arbitrary length paths in SPARQL (w/ data consistency) and it has significant performance limitations: i.e. CAR imports in 1.4 exponentially slow down with the number of entries.

With about three days of effort we could combine the optimistic logic into the native stores in Sesame 2.8. This would cut out the slow query rewriting code in the optimistic module that plagues Callimachus' import performance and give some consistency back to our SPARQL engine (wrt transactions).

prototypo commented 9 years ago

I can see where Callimachus would benefit from this. What would the impact be on other Sesame users?

catch-point commented 9 years ago

It would change the store from uses a pessimistic locking mechanism to an optimistic mechanism. Speaking with Jeen, it would be a welcome change. Sesame 2.8 API is aware of different isolation levels, but the store implementations currently can't handle anything beyond READ_COMMIT. This would bring it up to support SNAPSHOT_READ, SNAPSHOT, and SERIALIZABLE with settings that would allow the calling code to choose the appropriate level.

In Callimachus we currently use SNAPSHOT_READ, but once #30 is resolved we could switch to SNAPSHOT or SERIALIZABLE.

prototypo commented 9 years ago

OK, great. Bernadette has given approval for this work. Go for it.

catch-point commented 9 years ago

The MemoryStore has been update[1]. It has taken about 3.5 days, which is more then I estimated. The NativeStore can reuse much of that code, but more work is still needed. Callimachus uses the NativeStore primarily, but it shares a lot of code with the MemoryStore. I'd estimate another 1.5 days to complete this task.

[1] https://bitbucket.org/openrdf/sesame/branch/SES-2222-optimistic?dest=2.8.x