Journal migration tool

[ktoso]

Just an idea for now, but perhaps a rather useful one.

The following question comes up sometimes on akka-user: "I need to change my journal, because X-DB sucks and I need to use Y-DB now". Using the new read side we could simply "read all events" and store them to the other journal.

This depends on two features:

• [ ] the new query side - so we're able to stream "all events" ( link for anyone who's not aware yet http://letitcrash.com/post/96687159512/akka-persistence-on-the-query-side-the-conclusion ),
• [ ] allowing multiple journals in one actor system (while this is not really required, it could be using 2 actor systems, but would be optimal to keep this in one) so we can read from one, and store in the other one. (basically this task: https://github.com/akka/akka/issues/15587)

With the above two things in place, this should be rather simple to implement - or document.

[patriknw]

Some additional (overlapping) input.

Rewriting the persistent event history is not encouraged, but sometimes needed. E.g.

• event compaction (remove redundant intermediate events not needed for reaching the same final state) migration of old event representations to new representations (to be able to remove old code handling old representation)
• migrating from one journal implementation to another
• It should be able to stream events and save new events (possibly in another store).

This probably a tool that is run offline with real system shutdown.

[devluencer-089]

Such a tool would be great. Here are some thoughts how and why I would use such a tool in my current project.

• filter and cleanse current event streams that are 'polluted' with Confirm-Messages introduced by a workaround explained here. (Option b: Join streams using AtLeastOnceDelivery in individual persistent actors to a "topic" aggregator.)
• move framework-events (e.g. ShardCoordinator-Events and -Snapshots) to a dedicated journal
• migration of old event representations to new representations as this is sometimes less of a pain than writing long upcasting chains or using event adapters.
• split coarse grained events that "accidentally" made their way into the journal into fine grained events (rewriting history)
• use alternative serialization for events
• migrate event history to a new plugin for evaluation and/or migration

Hope this helps to tailor the implementation if you decide to support such a tool.

[devluencer-089]

almost forgot:

• retroactively adding labels/tags/categories to events once Akka 2.5 is released. :-)

[patriknw]

@michaelom thanks for great feedback

[scullxbones]

The first custom query I implemented was the all events (aka event firehose), this makes a lot of sense.

Something else to consider is possibly exposing the write side of journals as a sink, mapping the event envelopes into atomic writes.

[patriknw]

I agree that a stream based PersistentWriter would be nice. Also mentioned in #15667

[patriknw]

I think the building blocks are in place to implement this, and can also be done by end user (or external library). A PersistentActor can work as the writer, and integrate with streams using mapAsync and ask.

[ktoso]

Technically may be revisited once we revisit "projections" tooling so keeping this open even though inactive since a while. Feel free to close if that would be better tho