PACKAGE: NATS and TiDB for mutation events (CQRS Architecture pattern)

winwisely99 commented 4 years ago

We need to start to get the Storage Layer Infra up and in CI.

NATS and TIDB have fantastic K8 tooling, including operators.

Architecture Patterns

NATS is used to push updates to client modules via grpc. This is why liftbridge is perhaps the best option since it has a native grpc interface.

https://github.com/liftbridge-io/liftbridge

A user can have many devices that are various states of caught up to the latest push events.

A user's new device will need to catch-up from the nothing though. So this means that all state must be stored in tidb.

CDC pattern: So a mutation first is stored in tidb and then the resulting event is pushed onto the correct NATS / liftbridge topic so that all of the users other devices also get the mutation pushed to them. This is basically CDC !

Persistent log pattern: Because NATS does not store events for ever, you can use the Persistent log pattern also, so that all events are forever in the DB, to allow catchup: https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/persistent-event-log

This pattern should work effectively for all modules.

Persistent log pattern is the most simple one.

Code

https://github.com/pingcap/ticdc

I raised an issue to push Change events to NATS or Liftbridge: https://github.com/pingcap/ticdc/issues/201

Note the dependencies: https://github.com/pingcap/ticdc/blob/master/Makefile#L70

See the Network repo for extended make files that setup all these dependencies locally. https://github.com/getcouragenow/network/tree/master/tidb

Helps because you dont need docker at all to play with it.

CockroachDB

TIDB is pretty new. We coudl easily use CockroachDB for now. Good example for it here: https://github.com/dchenk/mazewire/blob/master/Makefile

It also has CDC but pushes to Kafka. Can make it push to NATS i think if we want.

winwisely99 commented 4 years ago

We need to generate a Flutter Client and test it working with Flutter Web ( and hence envoy)

The LiftBridge Proto for clients is here: https://github.com/liftbridge-io/liftbridge-api/blob/master/api.proto

They have a python and go generated client currently.

So in order to help everyone we should fork and add the dart generation to it which is easy.

Then you have the real client code which requires a bit more than just a code generated GRPC.

For Example the Golang one is here and of courses uses the Liftbridge-api code: https://github.com/liftbridge-io/go-liftbridge/blob/master/go.mod#L8

winwisely99 commented 4 years ago

@winwisely112 There is a TIDB Dashboard in incubation: https://github.com/pingcap-incubator/tidb-dashboard :)

am alerting you because you are likely to be the one to do the bring up of tidb on k8

amplify-edge / packages