search

eBay / akutan

A distributed knowledge graph store
Apache License 2.0
1.65k stars 106 forks source link

Log service #11

Open superfell opened 5 years ago

superfell commented 5 years ago

There should be a server implementation of the logspec service. Beam was moving from using Kafka to using this abstract log service definition for its log. The Kafka client needs some work for production usage.

ghost commented 5 years ago

I am very interested in the abstraction because I would like to support NATS instead of Kafka. Just raising this in case others might be thinking to do it.

superfell commented 5 years ago

There's 2 abstraction points for the log that can be used.

superfell commented 5 years ago

NATS looks interesting, the NATS streaming service looks it could work. However one thing i've seen is that Publishing a message doesn't return its sequence number in the Log. This is problematic as the writers to the log in the API tier need to know the sequence / log index number writes ended up at, for example, transactions are identified by their log seq #. The guid's that the AsyncPublish call expose aren't included in the messages the subscriber sees, so you couldn't even have it use a private subscriber to correlate on the guid to then get the seq#. Its possible i missed something, but looks like it wouldn't be possible to use the NATS streaming service with its current client API as a log for Beam.

superfell commented 5 years ago

Facebook's LogDevice looks like it'd be a good fit for the log service.

joeblew99 commented 5 years ago

@superfell NATS Streaming has those things and i agree they are needed for your use case. Its explained here: https://nats.io/documentation/streaming/nats-streaming-protocol/#PUBACK

Also the GO team are making a Enterprise framework and PUBSUB is in it. https://github.com/google/go-cloud/tree/master/pubsub

joeblew99 commented 5 years ago

AH its does not support ACK and NACK. https://github.com/google/go-cloud/blob/master/pubsub/natspubsub/nats.go#L395 SO basically they have NATS but not NATS Streaming support.

superfell commented 5 years ago

Its good its in the protocol, its unfortunate that its not reflected in the go api i looked at https://github.com/nats-io/go-nats-streaming