Batch processing

[woodpk]

Does Akkatecture provide for the ability to do batch data import processing? For example, if my application was importing a bulk export from another platform, instead of processing these as 'one command at a time' does Akkatecture allow for creating the domain event journal events in a faster batch process ?

[malclear]

I think the question you're asking is "is it possible to mutate multiple entities with a single command?". Yes, you can do that. There is nothing preventing you from raising several events based on the handling of a single command. Akkatecture has an 'EmitAll' function that ties together a collection of events into a single publish/transaction.

[woodpk]

Thanks for the input. I already read that info in your documentation.

I meant the original question. At work, I download failing medical claims information from our enterprise medical records system. They come in batches of 100,000 - 1,000,000 records per report.

I want to be able to batch import these records into a system that transforms each set of records into several events and persists those events (and then, of course, sends messages to a projector system to load projections w/ the event data as appropriate).

The transformation would convert the downloaded data into: ClaimAddedEvent(s), ClaimFailedEvent(s), ClaimApprovedEvent(s), ClientAddedEvent(s), StaffAddedEvent(s).

I'm working through the petabridge course (finally) now and have read the info about akka systems being designed to injest large volumes of data. But regarding your particular CQRS/ ES project, does it have existing methods to handle instances of heavy data flow or is the underlying akka system designed such that it handles this well even when the base CQRS/ES system is designed to push 1 command at a time and 1 event at a time ?

This may be a conversation you would rather have outside of a Github Issue ? Let me know if you'd rather close the issue and communicate another way ?

Thanks for getting back to me, patrick

---

From: Malcolm Learner @.> Sent: Saturday, July 3, 2021 05:16 PM To: Lutando/Akkatecture @.> Cc: Patrick Wood @.>; Author @.> Subject: Re: [Lutando/Akkatecture] Batch processing (#444)

I think the question you're asking is "is it possible to mutate multiple entities with a single command?". Yes, you can do that. There is nothing preventing you from raising several events based on the handling of a single command. Akkatecture has an 'EmitAll' function that ties together a collection of events into a single publish/transaction.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Lutando/Akkatecture/issues/444#issuecomment-873471515, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALS6BUFF55YIUMM2SD3BE7DTV544LANCNFSM47YJVMAQ.

[malclear]

At the end of the day, you'll have to loop through your report handling each record one at a time. This can happen either outside the domain where each iteration creates one command sent into the domain, or inside the domain where the iterating occurs in a single command handler. Given the latter approach, your command would need to contain the entirety of your data. This may be impractical, given its size. I would suggest a read and iteration through the records from client code, with each record producing its own command. I do something similar myself and it is extremely fast.

I am only an occasional contributor to this project. The author, Lutando, can be reached at times by way of the Discord channel: https://akkatecture.net/community.

[MarkusGeigerDev]

😞 https://github.com/Lutando/Akkatecture/issues/446