wojtekmach
commented
2 years ago General Notes
A few years back, I wrote a hacky event-sourcing implementation for our Turtle backend. It has worked for us so far, but I'd like to re-implement some core components as a standalone library.
Having worked on this particular problem already I think you're in a really good position to create a library. At the same time I think you're putting yourself on a journey to build a library (not an easy one at that!) which is a big committment and will take away from your journey of building the application and business.
We have found a nice list of CQRS libraries in Elixir, https://github.com/slashdotdash/awesome-elixir-cqrs#libraries. Have you considered any of them? Are there fundamental problems with them that are best addressed by starting from scratch?
Keeping it as a separate library will make it easier to test, improve, and get help on from folks like yourself :)
Agreed! Just a word of caution, when you start integrating the library into the main project, we'd strongly advice to keep it in the same repo as it is going to be so much more easy to work on both at the same time, no need for cutting and coordinating releases, etc. It can be really simple, just a top-level ./derive directory that is used as a path dependency from the main app: {:derive, path: "derive"}.
Before implementing too much, I'd love to get help iterating on the public API. Please take a look at the tests in
derive_ecto_test.exsand the documentation inDerive.Reducer
Before jumping in to these files, I briefly looked at some of the other internals, the Dispatcher and the EventLog. You might be interested in looking at Registry (the PubSub section) and PartitionSupervisor (will ship in Elixir v1.14) as these should provide great building blocks for what you want to achieve.
test/derive_ecto_test.exs
defmodule User do
use Derive.State.Ecto.Model
schema "users" do
field(:name, :string)
# ...
end
def up do
create table(:users, primary_key: false) do
add(:id, :string, size: 32, primary_key: true)
# ...
end
end
# ...
endWhat's your thought process on coupling the schema with the migration?
def handle_event(%UserCreated{user_id: user_id, name: name, email: email}) do
insert(%User{id: user_id, name: name, email: email})
end
def handle_event(%UserNameUpdated{user_id: user_id, name: name}) do
update([User, user_id], %{name: name})
endI think this is a very nice API.
Derive.Source.EventLog.append(:events, [%UserCreated{id: "1", user_id: "99", name: "John"}])
Derive.Dispatcher.await_processed(dispatcher, [
%UserCreated{id: 12, user_id: 99, name: "John"}
])It is too early for this but eventually perhaps a single function that does both (append and wait for it to be committed) would be helpful.
setup_all do
Derive.State.Ecto.reset_state(Derive.Repo, [User])
# ^^^^^^^^^^^nitpick: you probably did it just to get off the ground quickly but eventually I don't think Derive should ship with a Repo module. Instead I think the consumer of the library should pass their own Repo as this gives them more control.
lib/derive/reducer.ex
@moduledoc """
Defines how a given state is kept up to date based on an event source by a `Derive.Dispatcher`
It happens as follows:
- Events come from a source process defined by `&Derive.Reducer.source/0`
- These are partitioned by `&Derive.Reducer.partition/1` for maximum concurrency
- These events are processed by `&Derive.Reducer.handle_event/1`
which produces 0+ operations that are meant to update some state
- These operations are committed by &Derive.Reducer.commit_operations/1
"""This makes sense to me.
venkatd
Hi @TurtleAI/eds
A few years back, I wrote a hacky event-sourcing implementation for our Turtle backend. It has worked for us so far, but I'd like to re-implement some core components as a standalone library.
Keeping it as a separate library will make it easier to test, improve, and get help on from folks like yourself :)
Goals of the library
Derive.ReducerDesired characteristics
Scalability
As our number of users grow, we'd like to be able to handle the load. In the old implementation, I'm doing some things inefficiently such as redundantly querying the event source for events.
We'd also like to eventually run this in a multi-node environment. Our code in Turtle only works in a single-node environment.
Flexibility
We'd like to have flexibility on:
Good tooling
Once we've built this library, we will want to add some tooling to improve our experience. For example:
Next steps
Before implementing too much, I'd love to get help iterating on the public API. Please take a look at the tests in
derive_ecto_test.exsand the documentation inDerive.ReducerWhen we integrate this library into our Turtle backend, we would essentially define a bunch of reducers that implement the
Derive.Reducerbehavior. Then, we'd like some derived state (postgres tables) to be kept up to date with an event log (postgres table called events)Do you have any questions/feedback on the API? Any undefined behavior we should better define?
Thanks!