last event

[henry-hz]

Dear Supporters, I am trying to get the last event from a stream, once I am saving snapshots into that one. The problem is that I have to read all events from a stream before extracting the last one. Is there any way to read only the last one ? I saw that the C# API has a ReadStreamEventsBackwardAsync, shoud I create a PR adding this message in the protobuf include file ? best, and thanks for sharing this fanstastic lib

[mjaric]

Hi Henry,

We will check if proto file is up to date. Anyhow, is there a reason why you need all snapshots in snapshot stream? You could set metadata to such stream to keep only last event/snapshot.

[burmajam]

Hi Henry,

Please take a look at last commit. There's an update for reading stream backwards with test. Please try it and if everything is ok we'll push it to hex.

Thanks for kind words and reporting issue

[henry-hz]

Hi @mjaric, that's a good idea also, I just googled snapshot+eventstore and got a Greg's advice http://stackoverflow.com/questions/16359330/is-snapshot-supported-from-greg-young-eventstore . But thinking deeper on this option, in case we decide to change the stream version, adding or removing fields on the state, keeping the old snapshots can be nice if the database became too big and events deleted. @burmajam thanks for the fast update :+1: that's amazing. I hope to have a cqrs framework based on extreme and be able to share too :) . I am trying right now

[henry-hz]

It's working well, thanks!

[burmajam]

Hey Henry. I was thinking of some sort of cqrs framework but then I have realized (based on 2 big systems with about 20 apps in them) that that's hard to generalize. The only thing we can do is give some guidelines with some examples. Also I've found out that Aggregate can be generalized, thus I have created GenAggregate for our internal use. Take a look @ https://github.com/burmajam/gen_aggregate.

[mindreframer]

Hello @henry-hz, did you manage to finish your CQRS framework? It would be great to play with it anytime in the future. Love the screenshot, looks really crisp and clear.

[henry-hz]

Hi @mindreframer , thanks :) we are advancing, https://hex.pm/packages/engine , hope in two or three weeks to have some basic functionality. We are finishing the pure data structures based on the https://github.com/slashdotdash/commanded , but still in doubt if we will use his command dispatch approach, or genstage.

[mindreframer]

@henry-hz it looks great! I have seen it recently released on hex.pm!

About command handling: well... gen_stage is a non-trivial piece of software, with many other general-purpose features... I know you are interested in back-pressure for streams... Anyways I would first get clear interface requirements for the command bus / dispatcher, and provide a default dispatcher module. If the need to plug gen_stage arises, somebody could implement an interface conforming adapter for it and be done.

Also, some extra feedback:

• the name "engine" is practically non-googleable and carries little information. An artificial name goes long way, like "sidekiq". Also, it would evoke an emotional response, and this is critical for user adoption.
• you have 3 storage engines as direct non-optional dependencies. I know, it just now for the convenience and I hope they will be separate packages in the future ))))
• try to keep the number of dependencies at minimum, the current mix.lock file looks rather intimidating...

Hope that this does not sound too critical, I am really looking forward to play with "engine" and try it for some real projects.

Is there a place where you guys are discussing future directions? I would suggest gitter.im, really good integration with Github (duh!), and no message number limitations (like in slack). My guess is there will be some people interested in that topic (ES + Elixir seems really like a killer combo :)))

[mjaric]

Hi @henry-hz,

Alongside what mindreframer said here is what you could also consider.

Hide GenServer behavior beneath framework using macros. For instance, you will definitely need a router at the top level of service which hosts aggregates. It can force convention, so you want to force developers to map which command should be handled to which aggregate, so that is the very first level where you are routing commands. BTW,it is Bus you started to work on in your repository. Second router is the one you will need to find concrete aggregate to pass command to it.

Let say this Bus router should look like this (please ignore naming it is just guidance):

defmodule MyApp.ComandRouter do
  use Engine.Routing.CommandRouter # provides macro magic etc.

 @clinet_handler MyApp.Handlers.Client
  cmd MyApp.Commands.CreateClient to: @client_handler 
  cmd MyApp.Commands.UpdateClientEmail to: @client_handler

  @account_handler MyApp.Handlers.Account
  cmd MyApp.Commands.CreateAccount to: @account_handler
  cmd MyApp.Commands.CreditAccount to: @account_handler
  cmd MyApp.Commands.DebitAccount to: @account_handler
  cmd MyApp.Commands.GetAccountBalance to: @account_handler # this is probably not good place for such query :)  
  ...
end

Now, this is just to tell framework what command is going to which group of aggregates. At the same time CommandRouter is mapper for your Bus and it is good place to plug extensions. If you want to allow devs to extend functionalities to this layer of your framework, eg. to set transport for bus like rabbit, websockets, HTTP ... Note that CommandRouter is at the same time a process, so macro magic above should take care about it and registered it as named process so your command routing can start immediately as command arrives to MyApp endpoint. start_link and other event handlers for such behaviors are hidden in macro

Now, the constellation of processes where each handler should be grouped with its supervisor and aggregates it should call when command arrives. This could be done like this:

• Engine.DomainSupervisor it should be there to control all children below
  • Agent/GenServer which should hold the list of aggregate pids, ids (uuids, although I don't think it should be the only option)
  • Handler
  • Aggregates

When Engine app is starting, it should create CommandRouter and while it is starting it should bring up empty constellations of domains defined in CommandRouter, then start recovery (check if there is any events in store which need to be applied to aggregates) or, and probably better approach is to let it be lazy and wait command to fire up and recover aggregate to its latest state. This second approach is better when you need to balance load to more than one erlang node. So here is how supervision tree should look like probably

• EngineAppSupervisor
  • CommandRouter ( accepts commands from outer world and send them to domain handlers)
  • DomainSupervisor[client]
  • DomainRegistry[client]
  • ClientHandler
  • ... client aggregate pids ...
  • DomainSupervisor[account]
  • DomainRegistry[account]
  • AccountHandler
  • ... account aggregate pids ...

Finally, you will probably need something like the Plug module, since it could be very useful to other devs to extend Engine and easy for you to add yours. At least you should wrap caller pid and command and pass it all the way down to the domain handler. this will allow you to keep this part of communication asynchronous and respond to caller from aggregate. Second, if needed, you still can wrap command in transaction, the caller can guard it self with timeout... Milan Burmaja prefer "eventual consistency" which is Ok, I prefer to use transactions to avoid ping pong between domains, like in case when you need to create 2 aggregates in same business transaction. EventStore has acid transactions, it is not hard to wrap them in distributed one. Anyways, you can do things in both ways successfully and I don't see any significant bad effects using either approach, it is just what is your preferred way.

So far, this is what I could share, but if you open any channel please let us know, I'd like to participate as much as I can.

Regards, Milan J.

[mjaric]

One more thing @henry-hz, did you check Milan's B. repo https://github.com/burmajam/gen_aggregate maybe you find it interesting

[henry-hz]

@mindreframer and @mjaric thanks for the valuable and detailed feedback. @mindreframer please, feel free to tell all your feelings about every issue, please. Yes, we should open a gitter channel in the following week, and we should discuss more name options there, and also the ideas and suggestions we received from you. The direction will be to get abstracted, as soon as I organize the requirements in a more concrete implementation. Today I encapsulated the db side-effects for the aggregate, and I hope tomorrow I will abstract and refactor to cover also the process manager. I talked with @burmajam about the gen_aggregate, and it is more suitable for aggregates that self-interact, as it has a very cool buffer implementation, but to use with process managers we can drop the buffer out, once the PM is orchestrating. Note that the https://github.com/slashdotdash/commanded just implemented a middleware as I suggested to copy the idea from exq (queue lib) pipelines, and I just saw and got impressed.

[mindreframer]

@henry-hz crazy good! ))) We just have started with an MVP (in Ruby, to get a better grip with concepts), but might switch to Elixir, when we have a better understanding about requirements. Really hope we could distil some best practices for ES/CQRS in Elixir with those projects.

[henry-hz]

@mindreframer @burmajam please, take a look: https://github.com/slashdotdash/commanded/issues/34