Vase microservices to use clj-client to connect to peers.

[mik373]

Description

According to Datomic documentation in a microservice environment the peers are recommended to be running outside of the microservice process that way the actual microservice memory footprint can stay small, a peer can be shared amongst microservice instances and not be restarted as often as the microservice instance itself. It looks to me that the vase services do not have the option of using the datomic API client (http://docs.datomic.com/project-setup.html). If that's the case, are there any plans on supporting using the client right out-of-the-box in the future vase versions? If not, that it might be prohibitive for us to use Vase, since we plan on using Ditomic peers.

[mtnygard]

I'm definitely thinking about how we can support both the peer model and the client library. Their APIs are just a bit different, so it's not clear that the same set of literals can work effectively with both.

I'd be interested in your opinion. Given these choices, what makes more sense?

1. We use #vase/query, #vase/transact, etc for their current purpose (i.e., in a peer) and add new ones like #vase.client/query, #vase.client/transact for the client APIs. (This probably also entails making aliases like #vase.peer/query, etc.)
2. We attempt to plaster over the differences with some kind of dynamic namespace resolution. I'm totally waving my hands here, because I'm not sure how this would work in practice.
3. We make two different jars out of Vase. One for use in a peer, one for use with a client.

[hlprmnky]

I went reasonably far down the rabbit hole of producing my own fork of Vase to run on the client API recently - my use case is that I would really like to deploy a Vase service on AWS Lambda. I will share what I found in the hopes that it proves useful.

• It's not too onerous to get the initial connection construction and handling to dispatch on type, in fact I was able to add a client-config-map in the resources EDN file and simply look to see if that key was present to decide whether to make a peer or client connection.
• The place I stopped digging was when I realized that io.rkn/conformity not only assumes a peer connection, but relies on the ability to transact a fn into the database, which is explicitly not provided by the Client API today.

I would suggest, based on what I've learned, that approach 1would be fairly easy to produce in a clean, maintainable way once the Client API supports all the launch-time interaction with the DB that is currently assumed to exist but that until that beautiful future day dawns, 3 is more likely to work without anyone setting their own hair on fire.

[mtnygard]

Thanks, that is very helpful information.

[mik373]

Yes, I agree with 3 being the easiest option. It should also be the default option since that's the way Datomic wants microservices to work with it.

Sorry to be pushy, but is there any timeline on when the support would be implemented. It's just that I would help us plan on when start using Vase in production.

[andrewhr]

Just my 2cc: creating a new set of literals (the proposal number 1) can be easier, but diminishes the fact that data description lifts the abstraction and makes the service agnostic about Datomic implementation. Of course, I cannot say if it this "abstraction layer" is a Vase aspiration neither if it's practical in real world scenarios (any custom interceptor can break it)

[ddeaguiar]

Support for datomic via new tagged literals recently landed master. How to support both the Datomic cloud and on-prem clients needs to be investigated.