[Question] Event and Audit log aka centralized store of events persisted

[simkimsia]

Hi Harry and Bob 👋

I just finished reading chapters 8, 9, 10, 11.

I have 2 broad questions.

Question 1:

I understand that you treat Events as simple dataclass. I wonder if you would recommend reusing them to build timeline/audit log.

An example is your typical timeline in GitHub issue

See how the commits appear as "events" on the timeline of this issue as well as actual comments.

How would you persist the events in a centralized store and then display them by their associated activities or actions.

I found two schemas https://activitystrea.ms/ and https://schema.org/docs/schemas.html and not sure if they are related to the display of related events.

If I am wrong (which I think I probably am), let me know

Question 2:

I'm already using redis + celery + celery-beat + django-channels in my Django app to do 2 things:

1. run scheduled tasks
2. handle long running tasks triggered by web requests and then responded to via web sockets

I see that on page 168, 170 i.e. https://www.cosmicpython.com/book/chapter_11_external_events.html#_using_a_redis_pubsub_channel_for_integration you're using redis as the adapter around the message bus which according to page 262 in appendix E i.e. https://www.cosmicpython.com/book/appendix_validation.html#_validating_at_the_edge is simply a python class.

Would it be okay that I adopt the same setup?

Or is it crucial that the message bus be a separate piece of technology?

I understand you stated that at MADE, you use Event Store; Kafka or RabbitMQ

does that mean you use these technology as adapter for a message bus python class the way you use Redis in the book? Or does one of these 3 tech replace the python class as well?

Thank you

[hjwp]

re: audit logs or event history, i think it would be worth looking at a pattern called "eventsourcing", whereby the events become the core of the domain, and everything else is derived from them. check out bob's intro video here https://github.com/bobthemighty/eventsourcing-101

[hjwp]

re: scheduled tasks and long-running tasks and the message bus -- i think i'll find it easier to answer your questions in reverse order.

we distinguish between an "external" message bus (or message broker) and the "internal" message bus.

• the external message broker at Made is EventStore. we use primarily for integration between separate services, but sometimes also as a message queue for handing off work between different containers of the same service, so eg for async tasks.

• the internal message bus is pure python, its purpose is to structure the internals of the application, to allow us to define things in terms of commands, events and handlers, and the be able to cleanly separate responsibilities into different chunks of code and different transactions. but the internal message bus is not async.

does that help?

[simkimsia]

Thanks for your detailed response, Harry.

between an "external" message bus

going forward I will always indicate external or internal when I mention message bus.

For external message bus, I wonder do you use it for primarily for publishing events to other services you have no control over? E.g. another team's app/service in MADE or another api outside MADE.

Same question for listening to events, these other services publish.

For internal message bus

so basically this is like yet another pattern similar to repository pattern yes? your purpose is to separate the concerns between the main process for the unit of work (AllocateBatch) and its side effects (notify people after batch allocated or after exception happens when attempt batch allocation)

yes?

[hjwp]

do you use it for primarily for publishing events to other services you have no control over? E.g. another team's app/service in MADE or another api outside MADE.

it's primarily for other services within MADE. the external message bus is still internal to made.com though, it's not something we use for communicating to other companies. at the other end of the scale, we do sometimes use it as a message queue within a single service, as a way of deferring async work.

yes to the second question! the internal message bus is a pattern like repository pattern, a way of structuring your code to help enforce the single responsibility principle.

[simkimsia]

Sorry to belabor the point. Because i suspect I have a similar situation as yours.

it's primarily for other services within MADE

These other services within MADE are NOT under your / your team's control hence the choice of an external message bus.

Yes?

[hjwp]

that's right yes. I mean, they're not completely out of our control, anyone can open up a pull request on any service, but they're not out primary responsibility.

I would say that the reason we use the external message bus for integration isn't so much about control, it's more about wanting the benefits of asynchronous integration, plus using the events' schemas as our integration contracts / public facing apis. if the alternative if integrating via synchronous http api calls between services, that brings a host of problems. the coupling is much tighter. we talk about it a bit in the intro to the external messagebus chapter. I think there are also some pointers to further reading in there? or in the epilogue maybe? there's a book called "patterns of enterprise application integration" iirc...

[simkimsia]

I'm looking at this. https://github.com/johnbywater/eventsourcing to help with an event or audit log.

Am I roughly in the right direction?

[hjwp]

that could help yes! i gather it's quite an opinionated library, so its assumptions may clash with ours... did you check out bob's video and example code?

[Bryant-Yang]

I have taken brief try to event sourcing lib of @johnbywater, and watched the details of how event recorded in the 'stored_events' table. With this pattern we can record every change of a single biz entity.

There are lots of requirements in such scenario in my real work, most of the data seems like 'doc header'(such as purchase plan doc, order doc) + 'line items', all those data involves auditing, modifying. User want to check audit log, data change log, and want to query the details of the data right on event happen. So the event data rows will increase rapidly in db table, possibly encounter performance issue.

It seems like event-sourcing pattern is good at replaying the events, I'm not sure if it's also good at
log querying or data tracing (For me it need to think differently from querying data record to querying event record).

And I found more documents talking about use cqrs based on event sourcing and materialized view: https://docs.microsoft.com/en-us/azure/architecture/patterns/materialized-view.

[simkimsia]

Thanks for sharing the link from azure @Bryant-Yang

Are you familiar with the azure table storage service that article seems to talk abt?

[Bryant-Yang]

Thanks for sharing the link from azure @Bryant-Yang

Are you familiar with the azure table storage service that article seems to talk abt?

No, I haven't use MS tech stack for long time. I just read some common idea of architecture from this site.

[johnbywater]

@hjwp @simkimsia I don't think there is any clash of assumptions between Bob's 101 and the Python eventsourcing library, which supports having separate chunks of code for command, events, and handlers.

The library isn't really so very opinionated, but it does have quite a lot to say :).

It takes a fairly straightforward and standard approach to event sourcing, and most of it is just a collection of suggestions about how the core infrastructure stuff can be used, which again has a variety of offerings, and is intended merely to offer possibilities rather than make a statement of how things should always be done.

If there's one area that is opinionated it's the approach to distributed systems, and this is perhaps the difference of approach with other renderings of the event sourcing story. The library here has a "strong" opinion about reliability, which differs from most other descriptions and frameworks, in that it puts reliability first. Its design for reliability in distributed systems is described by the "process event" pattern, which involves three factors written atomically: tracking record; domain event record; event notification record. That's something you mostly don't see discussed or appearing in most systems. And I've been told this is something that's MADE also doesn't have.

This design also avoids the traditional approach to "sagas" or "process managers" that emit commands that need to be consumed by idempotent command handlers which I feel is a relatively challenging and difficult thing for most developers to understand and do well, and it seems to be a relatively restrictive position to me. The "process event" pattern doesn't have any such restrictions and obtains "exactly once" processing by design.

In my opinion, this "process event" pattern is crucial for obtaining a truly reliable distributed system, however the specialist "domain event stores" (such as EventStore and Axon) don't currently support recording process events. But I don't see why a specialist "process event store" couldn't be built along the same lines (ie ARIES with focus on append-only persistence whilst avoiding b-tree indexing), and it's something I've been discussing with various people.

So it's probably fair to say that the Python eventsourcing library's "default of reliability" for distributed systems currently comes at the expense of not currently having the scalability offered by the specialist event stores, but the core "simple application" stuff in the library can use these specialist event stores and in future I hope this situation will be remedied by the development of a specialist process event store.

But in the meantime, the "opinion" in library with respect to distributed systems is to start by default with a reliable system, and then degrade to the current conventional situation if needed (rather than starting with the conventional level of imperfect reliability, and then not really having a route up to obtain a truly reliable system). I think this is also justified despite the ultimate scalability limits of RDBMS by the fact the most software that is developed probably won't need to scale hugely, and if it does then sharding the state of a system on the user space or group space or document space will probably be satisfactory. And wanting to have both reliability and scalability builds the desire for developing the specialist process event stores that simply don't exist at the moment (there's no technical reason that I have been able to identify why such a DB couldn't be developed, and although I have zero experience developing database systems, I'm very tempted to have a go at prototyping one myself!).

Hope that helps! :))

[simkimsia]

Hi @johnbywater, thanks for this. I'm still learning so I have some clarifying questions.

@hjwp if it's better I take this branch topic elsewhere, let me know.

ARIES with focus on append-only persistence whilst avoiding b-tree indexing

Q1. What is ARIES? I can only find this https://github.com/hyperledger/aries-cloudagent-python

the specialist "domain event stores" (such as EventStore and Axon) don't currently support recording process events.

What's "process event"? What's the difference between domain events and process events?

I use https://martinfowler.com/eaaDev/DomainEvent.html or https://microservices.io/patterns/data/domain-event.html to form my understanding of what a domain event is.

this "process event" pattern is crucial for obtaining a truly reliable distributed system, however the specialist "domain event stores" (such as EventStore and Axon) don't currently support recording process events.

Sorry, this question is not so much targeted at the line I quoted, but more of an elaboration of my personal goal and options before the actual question inspired by this line.

Personally, I am not so much interested in building a reliable distributed system. More in terms of building something that supports "some kind of" audit logging.

The example I use yesterday in my personal email to you is like for e.g. user centric logs and issue centric logs in GitHub.

After our zoom yesterday, and from https://microservices.io/patterns/observability/audit-logging I now realize that true audit logging works best if I adopt a true eventsourcing situation. i.e. events are the source of truth and domain objects are transient. Or more accurately, domain objects are built from the events. Which I think your library adopts as first principle.

Another realization I just had is that I have (previously) mixed up application events aka transactional outbox pattern with eventsourcing pattern. They are actually different because in event sourcing, single source of truth is pure events and application events pattern does not have that same assumption.

And because I started with Django, obviously, my source of truth is currently not events. It's the domain objects.

Now I have two choices:

1. Adopt the "event is single source of truth" attitude in the eventsourcing pattern either by strangler pattern or big bang change all at once.

2. Adopt the application event pattern where I keep domain object as source of truth but also emit domain events.

My goal is to have two kinds of UI logs or timelines.

One user-centric logs and one that's domain-object-centric like the way GitHub has timeline logs for issues and contribution activity logs for users. Of course both logs actually are built up from the domain events either emitted in the application event pattern or in the eventsourcing pattern.

I suppose I cannot use your eventsourcing library for option 2. But can I use your library to adopt option 1 but only for new features in the manner of a strangler pattern?

[hjwp]

@johnbywater thanks for stepping in! for clarification, i meant a clash of assumptions between your library, and the architecture patterns and their implementation that we have in the book (repository, unit of work, message bus, etc).

@simkimsia i wonder if you'd get some value out of trying to implement event sourcing "from scratch", using a simple example like bob's shopping cart one, with no framework at all. then re-implement the same thing with John's library, so you can get a feel for how it all works. maybe that would put you into a good position to decide what to do in your real use case?

by all means do continue the discussion here, I'm curious to see how it all works out. hope you don't mind if i mark the issue "closed" tho.

It would be awesome if you decide to write up a blog post about your experiences at the end of it...

[simkimsia]

Hi @hjwp

did you check out bob's video and example code?

Yes I did. Several times in fact. and I must have contributed more than my fair share of the 666 views it has so far 😆

I even downloaded a copy of the video to watch offline. It's good. Thanks for recommendations!

As far as I can see from Bob's video, esp this diagram

That Bob is using the eventsourcing pattern where the single source of truth is the events itself. And the domain objects are built up from the events (or to be more precise, the events are applied to derive the domain objects).

Yes?

I initially wanted to ask more abt persisting events and eventsourcing in cosmicpython. Then I found this issue https://github.com/cosmicpython/book/issues/240#issuecomment-588258339

options to persist events

And I realize the main options are lined out in a similar fashion as my earlier response to John

1. Use an external eventstore/broker/bus that persists the domain events like in chapter 11.
2. Flip everything to use events as single source of truth as per Bob's video aka eventsourcing pattern. Note the cosmicpython book did not explicitly covers this.
3. Use the application events pattern aka transactional outbox pattern but this may require single commit to ensure the events are saved in the right sequence and within the same atomic transaction.

My goal

Ultimately I just want two kind of UI logs similar to GitHub's issue timeline and user contribution activities,

but I have already started in Django and persisting domain objects in my own native Postgres database.

Then as I re-read chapters 9, 10, 11. I came across an example code in chapter 10 about History Aggregate.

https://www.cosmicpython.com/book/chapter_10_commands.html#_discussion_events_commands_and_error_handling

I was wondering if how do you persist the History Aggregate's following data:

1. History
2. HistoryEntry
3. the events under the History instance ?

[simkimsia]

Sorry I just saw your reply, @hjwp

My previous commented was drafted 99% hours ago but I got called away to a work meeting so just posted it.

if you'd get some value out of trying to implement event sourcing "from scratch", using a simple example like bob's shopping cart one, with no framework at all. then re-implement the same thing with John's library, so you can get a feel for how it all works. maybe that would put you into a good position to decide what to do in your real use case?

Good idea! Will do so.

eans do continue the discussion here, I'm curious to see how it all works out. hope you don't mind if i mark the issue "closed" tho.

yes of course! This is your repo. You have final edits.

It would be awesome if you decide to write up a blog post about your experiences at the end of it...

Haha, I will provided I clear my previous debt first. I think I still owe you a draft post for another topic. So how do you suggest I send you my draft posts?

I can create a private git repo and send my drafts there and invite you or you have other ideas for your workflow?

I like how cosmicpython was put together using asciidoc so happy to do the same.

[hjwp]

re: blog posts, you don't have to wait for me to review things! if you'd like to publish on your own blog first, by all means do that. then we can organise cross-posting onto cosmicpython.com

if you'd like to do the post as a PR to our blog, that's fine too. the blog posts are all here, they're in markdown format (not asciidoc) https://github.com/cosmicpython/cosmicpython.github.io/tree/master/posts

[simkimsia]

re: blog posts, you don't have to wait for me to review things! if you'd like to publish on your own blog first, by all means do that. then we can organise cross-posting onto cosmicpython.com

Ok!

[johnbywater]

@simkimsia

Q1. What is ARIES?

It's basically the standard way to build a database.

https://en.wikipedia.org/wiki/Algorithms_for_Recovery_and_Isolation_Exploiting_Semantics

[johnbywater]

@simkimsia

What's "process event"? What's the difference between domain events and process events?

The difference is that a "domain model event" is the "what happens" when something happens in a domain model; whereas "process event" is the "what happens" in an application that involves something happening in its domain model.

These two things are indistinguishable in the simple case, when a client request invokes a command method which makes something happen in the domain model, and what happens in the model is coded as domain events that are persisted.

However, there are two other factors: event notifications; and tracking records. A "process event" includes the three factors: tracking upstream position; domain events; and downstream event notification. Each of these three factors maybe null, so that writing a process event could involve just writing tracking records; or just writing domain events. You can can't really have notifications with domain events, but you can have domain events without notifications if the domain events should not be propagated for some reason (e.g. PII and data protection).

The event notifications must be written atomically with the domain events to avoid the unreliability of "dual writing". If the domain events are written before a notification is sent to a message queue (which is still something people are doing), or if the notification is send to a message queue before the domain events are written to a database, then it's possible for one thing to happen and not the other. This was described by Vaughn Vernon in his book Implementing Domain Driven Design published 2013, by Martin Kleppmann in a talk called Using logs to build a solid data infrastructure (or: why dual writes are a bad idea) in 2015, by James Roper in talk called From CRUD to Event Sourcing Why CRUD is the wrong approach for microservices in 2017. These three have discussed the same issue on many other occasions, and so have many others.

Propagating state whilst disregarding this concern is broken, and most uses of "message queue" or "message bus" that uses AMQP suffers from this problem.

[1] https://www.goodreads.com/book/show/15756865-implementing-domain-driven-design [2] https://martin.kleppmann.com/2015/05/27/logs-for-data-infrastructure.html [3] https://www.youtube.com/watch?v=holjbuSbv3k

Similarly, tracking records must be written atomically with new domain events that are created when an upstream domain event is processed, to avoid losing track of position and processing upstream state either more than once or less than once. By writing tracking records atomically with new state, we can achieve "exactly once" semantics in the processing of application state. And there isn't really another way of doing it.

Upstream state can, of course, also be projected into non-domain event ORM objects or other data such as a JSON document representing a materialised view. Obviously in this case, since there aren't any domain events, there can't be any event notifications. So ORM objects is perhaps a fourth "factor" of "process events".

Basically, "event" is the answer to the question "what happens?"; "domain event" answers the question of "what happen in a domain model?"; and "process event" answers the question of what happens in an application. By identifying the "process events" and making them atomic "actual occasions", we can accomplish "exactly once" semantics and use this design as the repeatable building block for making reliable distributed systems. Everything is else is essentially a departure from this design, which inevitably introduces unreliability, which may be justified for particular reasons such as extreme scalability or performance given the current state of databases that actually exist at the moment. But in most cases, such departures aren't justified, and in fact most development continues without an appreciation of these considerations, and the reports that "distributed systems are hard" is more or less an expression of failure to grasp what is actually happening and what is needed to create satisfaction.

In other words, "process event" is the real objective satisfaction yearned for by everybody who wants to build distributed systems that won't let you down.

My view is that the reason why people mostly fail to figure this out for themselves is because of the incumbent dominant view that distributed systems are characterised by "passing messages", and furthermore that these "messages" are being passed between "enduring objects" (the typical domain model entity that changes and often represents a physical object encountered in the domain supported by the application). However, these are pre-modern notions that are several degrees of abstraction away from what is required by the situation, and unfortunately suffer from the "fallacy of misplaced concreteness" and critically fail to avoid the "substance-quality" categories that function to restrict thought. And that's the restriction that needs to be set aside before it's possible to think about process events. The process events are the real actual entities that actual systems are built up of. And we have been severely misled by people who didn't know this. These concepts were there all along, explicitly described in modern process philosophy and latent in our field of work due to the application of modern process philosophy in the original pattern language of Christopher Alexander. So there are no excuses, and it's been a multi-trillion dollar mistake. :-))

[johnbywater]

@simkimsia

By the way, the "outbox pattern" is basically a reprise of the "notification log" pattern. These two things are "process events" that involve domain events and event notifications, and by writing these two factors atomically, the state of an application can reliably be propagated. Vaughn Vernon has said that he doesn't like the name "outbox pattern" because a message is normally removed from an outbox when it is sent, whereas a notification log continues to exist regardless of how many times it has been read.

[johnbywater]

@hjwp @simkimsia

Harry - you are in a better position than me to know whether or not your book avoids the dual writing problem, but at least on this page it seems that it doesn't: https://www.cosmicpython.com/book/chapter_08_events_and_message_bus.html

Option 1

   try:
            batchref = product.allocate(line)
            uow.commit()  # Dual write part 1.
            return batchref
        finally:  #(1)
            messagebus.handle(product.events)  #  Dual write part 2.

Option 2

def allocate(
        orderid: str, sku: str, qty: int,
        uow: unit_of_work.AbstractUnitOfWork
) -> str:
    line = OrderLine(orderid, sku, qty)
    with uow:
        product = uow.products.get(sku=line.sku)
        if product is None:
            raise InvalidSku(f'Invalid sku {line.sku}')
        batchref = product.allocate(line)
        uow.commit()  # Dual write part 1.

        if batchref is None:
            messagebus.handle(events.OutOfStock(line.sku))
        return batchref  # Dual write part 2.

Option 3

class AbstractUnitOfWork(abc.ABC):
    ...

    def commit(self):
        self._commit()  # Dual write part 1.
        self.publish_events()  # Dual write part 2.

This isn't in conflict with the Python eventsourcing library, which has a decorator @subscribe_to which can be used to send published events anywhere. For simpler uses, the library uses a simple publish-subscribe mechanism to implement inversion of control from the domain model and infrastructure, so that the domain model publishes events and then a persistence subscriber subscribes and writes events to a database. Dual writing to a message bus can by implemented by subscribing a message bus client after (or before) the persistence subscriber, and then domain events published by a domain model will be written to the event store and then sent to a message bus. The @subscribe_to decorator makes it easy to register handlers. So it's possible to do this "dual writing" mistake in the library, but it isn't recommended.

https://eventsourcing.readthedocs.io/en/stable/topics/projections.html#subscribing-to-events

The Python eventsourcing library instead tends to write the domain events and event notifications atomically to the database. And then the notifications (in a notification log) can be read and consumed in different ways. One thing would be to send them to an AMQP style message bus, but then the problems of out-of-order messages, duplicates, and so on recur, since the ACK of the message bus consumer is effectively another case of dual writing (you have to ACK to the message bus either before or after the consequences of processing the message are recorded but you can't do it atomically).

Of course not all databases are capable of having domain events and event notifications written to them atomically. For example, the "light weight transactions" of Cassandra don't allow for writing to two distinct sequences. But EventStore and Axon both do support this, and this "atomic dual-sequence" writing can be implemented in relational databases (and other databases). You just need two indexes, one to read the domain events of an aggregate and one to read the event notifications of an application. I discussed these concerns carefully in the section "Three options" in the Python eventsourcing documentation:

https://eventsourcing.readthedocs.io/en/stable/topics/notifications.html#three-options

The trouble with the mature RDBMSs is that they use b-tree indexes, which aren't expecting the append-only behaviour, and so need to rebalanced regularly, and it's this rebalancing which tends to get in the way of large volume and large velocity ("big data"). But, as I said before, most applications aren't "big data" applications, so it's fine to use RDBMSs in most cases, and it's probably always a good place to start, since it makes things easy and making things easy at the beginning means you are more likely to be able to maintain development velocity and find an implementation that actually supports delivering some business value (without which the project is dead regardless of how well it would scale in the future).

The Python eventsourcing library tries to maximise developer velocity by providing the "Plain Old Python Objects" infrastructure, which "stores" events in memory, allowing test suites to run in milliseconds. It's much faster even than "in memory" SQLite. CI can then run with a normal database, putting things on disk. And then applications that need to scale can use one of the specialist event stores (e.g. EventStore or Axon).

The "process event" pattern takes this avoidance of dual writing one step further, picking up the "ACK" case I mentioned above, by allowing a tracking record (that has the position in an upstream sequence) to be written along with the domain events and the event notifications, so that one event sourcing application can be projected into another event sourced application, making a distributed system. Just like a domain model, and indeed a simple application can be defined independently of infrastructure, the Python eventsourcing library allows an entire distributed system to be defined independently of both infrastructure, and of the mode of running the system. (With a single threaded runner being the fastest way of running a system, and the most suitable for running a system during development. Other runners, for multi-threaded running, multiprocessing, running with both Ray and Thespian actor model, are included in the library).

For horizontal scaling of a system of computational intensive applications, this "staged" processing can be done with "parallel pipelines" of processing, and the library supports synchronising the pipelines when there are causal dependencies between events processed in different pipelines. I tried to discuss these topics exhaustively on this page of the Python eventsourcing library documentation:

https://eventsourcing.readthedocs.io/en/stable/topics/process.html#process-application

However, as I said, at this time there aren't any specialist "process event stores" that support the required atomic writes, whilst avoiding the use of b-tree indexes. So when using this "system of applications" technique, special attention must be given to sharding the state of the system to achieve high volumes (either by document/user/group or with "time buckets" e.g. accounts for one year). But there are many systems that don't involve massively high volumes that are required to be highly reliable, and since ~80% of software is written to support internal ordinary line-of-business processes, perhaps most applications don't and never will.

Hope that helps! :-)

[simkimsia]

Thanks @johnbywater for your detailed replies.

A "process event" includes the three factors: tracking upstream position; domain events; and downstream event notification.

Have you seen this? https://blog.arkency.com/correlation-id-and-causation-id-in-evented-systems/

And also this process event.. where do you draw the boundaries so that the various downstream events or their notifications are part of a single process event?

And tracking record.. what is an example of that?

This was described by Vaughn Vernon in his book Implementing Domain Driven Design published 2013, by Martin Kleppmann in a talk called Using logs to build a solid data infrastructure (or: why dual writes are a bad idea) in 2015, by James Roper in talk called From CRUD to Event Sourcing Why CRUD is the wrong approach for microservices in 2017. These three have discussed the same issue on many other occasions, and so have many others.

Thanks for the links! I'm getting the red book since even Harry recommended that as well. I only barely finished one chapter of the blue book.

incumbent dominant view that distributed systems are characterised by "passing messages", and furthermore that these "messages" are being passed between "enduring objects" (the typical domain model entity that changes and often represents a physical object encountered in the domain supported by the application).

Yes. the enduring objects passing messages is also how I read the event driven architecture literature in general everywhere.

But the difference between me before yesterday and me today is that, I'm now emotionally aware of this subconscious bias.

And now I can totally see that you take a event first attitude all the time. Which suddenly makes a lot of your writings a lot more comprehensible to me.

For e.g. the diagram from the transactional outbox I link to previously clearly shows that the two things saved in an atomic transaction

thing + message/event is the enduring object + the event of saving/changing it

Whereas the way you write on this topic of outbox/notification log is

These two things are "process events" that involve domain events and event notifications, and by writing these two factors atomically, the state of an application can reliably be propagated.

clearly, event first

domain event + event notification of saving that domain event

Not saying one way is right or wrong. I'm just expressing my relief at having this epiphany now. 🙌 🎉

Took me so long to grok this fundamental assumption in your entire approach and library. I feel like 🤦 myself in the face a million times despite reading your docs so many times.

If it helps other newbies like me to your docs and library, and as you clearly point out that the dominant view is still of enduring objects. I recommend that you put in your documentation on first page, and at the very top a giant warning sign. Something like the below.

You're probably thinking in terms of enduring objects passing messages/events to one another. This library assumes something totally radical. We assume there's no such thing as enduring object. There's only events and their by-products such as event notifications. An object is simply derived by culmination of all the domain events that create it. (Notice how I purposely write this way instead of saying "an object is the culmination of all its events". In this library, object does not own the events. Events own their object) Here, events are first-class citizens conceptually.

You might read this and go yea yea, right, i got it. But no, you haven't really. Because for most people this bias towards enduring object is very deep and subconscious. When you struggle with understanding this library, I can safely say that chances are 99 out of 100, you're still thinking in an objects-first world view where objects are enduring. Here, events are first-class citizens conceptually. Here, events come first :)

I'm just giving a suggestion. You don't have to follow it or the words I use. I can be a tad dramatic in my choice of words, ha!

Given that most CRUD frameworks like Django assume the dominant worldview of objects first, I need to recalibrate my experimentation of the event-sourcing pattern in general.

I think event-first is highly promising. But I run a business with real world constraints. As much as I would love to revamp everything from scratch, I have to plan this. Probably will take Harry's suggestion and use your event-sourcing library in a separate sideproject to play with to learn.

[johnbywater]

@simkimsia

The Python library makes it easy to use correlation and causation IDs (except that I didn't actually document this very well so far - but the usage is covered in the test suite). https://eventsourcing.readthedocs.io/en/stable/topics/features.html#additional-features

You just inherit your model entities from EntityWithECC, and then pass in the "processed event" when triggering a new event, if there is one. https://eventsourcing.readthedocs.io/en/stable/_modules/eventsourcing/domain/model/entity.html#EntityWithECC

But the Python eventsourcing library makes it easy to debug "evented systems" without this, because it has the SingleThreadedRunner which processes everything synchronously in a single thread, so you can easily step through the behaviours in a debugger, and easily stop on breakpoints.

I suppose the advantage of using event IDs, correlation IDs, and causation IDs is being able to trace things back after all the events have been written. That's very useful for auditing, not just debugging. Without recording these relations, it's basically not possible to see what actually caused what, across a distributed system. That's the reason I added it as a feature in the library. It was possible to do it yourself before, but this feature makes it easier for those who need it.

[johnbywater]

@simkimsia

where do you draw the boundaries so that the various downstream events or their notifications are part of a single process event?

There's a diagram on this page which shows where the boundary is drawn for a single process event. https://eventsourcing.readthedocs.io/en/stable/topics/process.html#process-application

To be fair, I should perhaps more strictly follow Alfred North Whitehead's terminology, and call this atomic thing an "actual occasion of processing", and use the term "processing event" for a nexus of such things, where the limiting case is a "processing event that has only one actual occasion of processing". That would allow the processing of a set of events to be considered also as an event, which is more natural.

As you can see, I'm still being quite sloppy in my use of language, and should tighten things up more. :-)

[johnbywater]

@simkimsia

Not saying one way is right or wrong.

Good point. Not dealing directly in terms of events isn't wrong. But it's important to understand that the "enduring objects" are abstractions and the atomic "actual occasions" are the concrete reality. To think it's the other way around is to suffer from the fallacy of misplaced concreteness, to fall into the trap of "presentational immediacy" and fail to recognise the degrees of abstraction involved in perceiving the enduring objects as "things".

Traditional CRUD models are built up from events too, thing happen, but what happens isn't coded for explicitly.

Once you can recognise, the general case, that actual worlds are built up from actual occasions of experience, that these actual occasions (e.g. atomic domain events, and atomic process events) are the actual entities, then you can rest having finally reached solid ground, since there is "no going behind the actual entities to find anything more real." :-)

[johnbywater]

@simkimsia

I recommend that you put in your documentation on first page, and at the very top a giant warning sign. Something like the below.

Haha. I'm definitely wondering how to make this more approachable from the "old world". I'm genuinely struggling to remember what it was like thinking about things without understanding modern process philosophy.

[simkimsia]

@johnbywater

I'm definitely wondering how to make this more approachable from the "old world". I'm genuinely struggling to remember what it was like thinking about things without understanding modern process philosophy.

Take it from me a total beginner to the whole eventsourcing pattern and ur eventsourcing library. And now less so of a beginner but still remember what a beginner feels like.

Make the warning Loud and clear and repetitive and appeal to their self interests and paradoxically catered to their worldview

What are the people trying to achieve when they go to your docs?

They are trying to understand your library. That’s their self interests. They want to do it as quickly as possible and painless as possible.

Make it loud and clear that their existing world view of objects first is the very thing in their way of achieving their goal.

Tell them the story of the fish in the water.

Tell them they will somehow use the lens of objects first to understand your library but it will be very difficult. Bob YouTube in event sourcing 101 help when he uses the example of not knowing whether a user chose socks in the shopping cart. Helps to illustrate the weakness of objects first.

But most developers after getting that lesson will then assume okay then I will do a shopping cart + events where events are the secondary.

This is where your warning need to go further than bobs excellent example. You need to tell the readers of your docs in no uncertain terms

Instead of trying to bending backwards like mad trying to make objects first approach work for evented systems, try to realize the truth

THERE IS NO OBJECT

then show them an example of how complicated and ugly it is with the objects + events example

Then with the same use case show a pure events-first approach.

In other words

1. Do a pure objects no events approach like what Bob did. To show the inadequacy of objects only
2. Do the same with objects plus events approach to show how ugly or painful it is
3. Do the same with events first or events only approach.
4. Keep emphasising non stop as u go through example 1-3 that the readers object first worldview is deep and subconscious bias just like how water is invisible to fish despite being everywhere around the fish

I might write it up as a blog post. Just to make the lesson concrete for myself and as a ritual call it as the end of my total beginner phase.

Might need help with the examples code

[Bryant-Yang]

I thought I could take a break from reading，but it's just the beginning 😃. Great discussion.

[johnbywater]

@simkimsia

THERE IS NO OBJECT

This isn't really accurate. Objects are the things that subjects apprehend. A subject is an object for another subject. In Whitehead's scheme, the actual occasions of experience are the subjects. The content of the experience is the many "feelings" of which they are comprised. The unity of each actual occasions is also a feeling, a new feeling that is created by the process of its becoming, each one being a unique and novel advance in the universe. The feelings are processes of appropriation, bringing other actual occasions ("data") into the subjective objectifications ("datum"). This may involve negation of feeling ("negative prehension") where an otherwise "positive feeling" ("positive prehension") is held inactive.

Whitehead distinguishes four stages: the datum, the process, the satisfaction, and the decision.

Discussing the scientific method, Whitehead wrote: "The true method of discovery is like the flight of an aeroplane. It starts from the ground of particular observation; it makes a flight in the thin air of imaginative generalization; and it again lands for renewed observation rendered acute by rational interpretation.")

As Ward Cunningham wrote in EPISODES: "We are particularly interested in the sequence of mental states that lead to important decisions. We call the sequence an episode. An episode builds toward a climax where the decision is made. Before the decision, we find facts, share opinions, build concentration and generally prepare for an event that cannot be known in advance. After the climax, the decision is known, but the episode continues. In the tail of an episode we act on our decision, promulgate it, follow it through to its consequences. We also leave a trace of the episode behind in its products. It is from this trace that we must often pick up the pieces of thought in some future episode."

These are the "actual entities". They become, and then perish into "stubborn fact", but do not change. Each actual entity, once it exists, is a "fact" with a "date". Each existing actual entity provides potential for the process of creating other actual entities. That's the relation between "being" and "becoming". And the actual entity is a "decision" or a "cutting off" of the process of becoming, which happens when something is fully defined, when a settlement is reached, when there is no longer any incoherence within the selection of feelings in the process, and so when there is a harmony and is also a unity. The resulting unity of feeling is what makes the actual occasion what it is.

There are other kinds of objects, in particular the "eternal objects" which, as "pure potentials", also "tell no tales of actual occasions". There are consequently no novel eternal objects.

There are also "enduring objects" which are "social nexus" of "actual entities" with "personal order". A nexus has "social order" when the actual entities share a "fact of togetherness". Such a "social nexus" has "personal order" when the actual entities are ordered serially (like the domain events of an aggregate in a domain model).

A non-social nexus has no such order and "answers to the notion of chaos" (which brings us back to the title of this "Cosmic Python" book). It is a "pure many" and so cannot be said to correspond to an object.

But in discussing "objects", it should be remembered that what is fundamental are the "feelings of feelings that have already been felt". The "objects" and "subjects" are a layer of order that kind of sits over this relational and processual reality.

"The ontological principle declares that every decision is referable to one or more actual entities, because in separation from actual entities there is nothing, merely nonentity— 'The rest is silence'"

https://archive.org/stream/AlfredNorthWhiteheadProcessAndReality/Alfred%20North%20Whitehead%20-%20Process%20and%20Reality_djvu.txt