API design

[danielsz]

This is a question rather than an issue.

I was wondering about the design decision to make the core.async explicit rather than implicit in the API.

What are we gaining by writing this:

(<!! (k/assoc-in store ["foo" :bar] {:foo "baz"}))

Instead of:

(k/assoc-in store ["foo" :bar] {:foo "baz"})

Assuming that the latter would do the core.async calls internally.

BTW, I am asking this question naively, without having studied the source code.

[whilo]

One of the main purposes is to be compatible with ClojureScript so that one can write platform-neutral code. Blocking IO is not available on JavaScript, so we use core.async in the first place to allow writing sequential code everywhere. I guess that if you are interested in Clojure support only, you might think that this is redundant code.

Note though that this has a useful sideeffect as well, as you can use https://github.com/replikativ/superv.async to automatically deal with IO errors in your async code. This is otherwise not readily available. A further advantage to use core.async for blocking IO in Clojure is to get a lot more concurrency in a node.js fashion when you use for example https://github.com/mpenet/jet. I haven't done so yet, but plan to do so. The file-store does not use asynchronous IO on the JVM (it blocks the thread) for now, but it should be implemented with FileChannel in java.nio. I just wanted to have a safe simple store first.

So konserve also helps you to write better asynchronous code. We can provide Clojure shortcuts if you think the core.async operations are too verbose in many cases.

[danielsz]

Ah, yes, I see. Thank you for the explanation. It makes sense.

I will get back to you after accumulating some mileage with konserve as to whether or not I think a sparser notation would be beneficial in the context of pure Clojure (blocking I/O).

That wouldn't be the first time a library exposes a synchronous and asynchronous API side by side.

[whilo]

Ok, sure. But by this convenience you lose the portability of your storage IO code, just to repeat :).

[whilo]

Maybe this does not seem important, but most of my code uses two kinds of side-effects: storage IO and network IO. If I can use a cross-platform protocol for base cases (if I don't need some specially tuned DB like Datomic or something with very critical performance, but rather something like a kv-store), I will be able to factor and reuse most of my code later on, no matter where it moves, e.g. Android, iOS with react-native, web browser, Clojure CLR in Unity (if core.async is ported again), embedded JVM etc.

All that is needed is core.async with non-blocking ops (no *!! operators) and some implementation of the protocol. There is a price in the fact that core.async is not as battle-tested as many native storage interfaces and some things can be tricky (good error-handling took me roughly a year to figure out, due to cljs dynamic binding issues), but I think this price is worth paying as one always has to write custom code for performance critical sections. core.async go-routine overhead is especially not a problem performance-wise compared to IO operations. One can built really nice systems on core.async primitives and they are better scalable than blocking threads anyway. Anyhow I am rambling... :)

But I would love to see more complex storage types built with konserve and core.async in a plattform-neutral fashion: I have done a quick dive here: https://github.com/replikativ/durable-persistence but plan to port Hitchhiker trees https://github.com/datacrypt-project/hitchhiker-tree soon. If you have any wishes or contributions, let me know :)

[whilo]

I close this issue for now, feel free to reopen it when you have questions or some ideas about a better API :).

[danielsz]

Sure. I wanted to thank you for the explanation. This was very helpful. I'll be happy to discuss this further after I gain more insights.

[whilo]

Btw. feel free to write a system component for replikativ :).

[danielsz]

I have done so, actually. ;-) At least for Konserve, that's the part I'm interested in at the moment. It's already listed among all the other components in the README, but it will be properly announced with the next release of system.

It's just a start. It supports only the built-in stores for now.

https://github.com/danielsz/system/blob/master/src/system/components/konserve.clj

I believe it will be enhanced in time.

Besides that, I've started using Konserve in my prototypes. I really like it. One thing I was wondering is how hard it would be to graft a datalog query engine onto the stores.

[whilo]

Cool, I added a reference to the component to the README.

Yes, I would like to have an efficient persistent (snapshottable) sorted index implemented on top of konserve. I did a simple experiment here: https://github.com/replikativ/durable-persistence I also tried to port the hitchhiker tree https://github.com/datacrypt-project/hitchhiker-tree , but it is very tightly coupled to redis commands and I couldn't figure it out on a weekend. I would love to have this as a building block to store associative things like DataScript or sophisticated persistent replikativ metadata in konserve. Sadly no-one decided to cooperate in this direction yet, so I have it postponed after I have shown some use cases for replikativ and improved kabel (network lib).

[danielsz]

Thank you for the hitchhiker-tree mention. I had a look a it, and was very impressed indeed. https://github.com/datacrypt-project/hitchhiker-tree/issues/29

[whilo]

Yes, it is impressive. Do you think it would be portable to konserve? Performancewise it will be bad, because all the inlined redis operations will have to become generic, but otherwise it could give a nice persistent datastructure for cljs as well.

[whilo]

I moved this discussion here: https://github.com/replikativ/konserve/issues/4