search

rebus-org / Rebus

:bus: Simple and lean service bus implementation for .NET
https://mookid.dk/category/rebus
Other
2.26k stars 355 forks source link

Cosmos DB saga storage #923

Open gideonkorir opened 3 years ago

gideonkorir commented 3 years ago

Similar to this previous issue. I'd like to run rebus on Azure with Cosmos DB rather than use SQL Server since we are using currently using cosmos for all our data storage.

I've been looking into it and have an implementation here.

So far this is what I've come up with:

  1. Partitioning by some correlation property (e.g. OrderId) means that looking up the saga data by any other correlation property will lead to a cross partition query. My current implementation only supports a single correlation property and throws if multiple properties are specified. This means all tests in Rebus.Tests.Contracts that have multiple correlation properties will fail.
  2. Partitioning by saga data type (or some value derived from it) allows multiple correlation properties since we always know the partition key but will use up more RUs since we won't be doing a point query unless we are using the id property.

I've checked the MassTransit saga persistence and Nservicebus saga persistence and they all seem to be using (1) above.

Using (2) might be ok if one is ok with limits of a single partition

mookid8000 commented 3 years ago

I'd definitely go for (1).

Regarding the tests, maybe it would make sense to simply not use the contract tests for this persister.

Or maybe it would make sense to split the contract tests out into separate fixtures for single-property and multi-property correlation scenarios, which would then make it possible for single-property persisters to include only the tests they can work with....?

gideonkorir commented 3 years ago

I have implemented both options (I was thinking that (2) might be better for someone migrating from other stores and they are ok with the limitations + it was quite simple to implement).

On the tests for (1) I copied over the contract tests and modified the correlation to get them to work. It was pretty straightforward.

I've pushed all the code + tests here.

TimterLaare commented 2 years ago

I would love this feature in a future release! @mookid8000 What is the status of this issue?

mookid8000 commented 2 years ago

Not sure 🙂 @gideonkorir does it work for you?

gideonkorir commented 2 years ago

@mookid8000 I didn't get to use it in production :( but it was working for all initial tests that I had though I'd be happy to collaborate to get it in a good state

TimterLaare commented 2 years ago

@gideonkorir @mookid8000 Which steps need to be taken to get this implemented?

mookid8000 commented 2 years ago

Hi @gideonkorir , did your CosmosDB saga persister work as expected? Would you be interested in contributing it to rebus-org?

gideonkorir commented 2 years ago

@mookid8000 The persister worked as expected for my local testing but as I discussed before I never got the chance to use it in prod. I am happy to contribute it to rebus-org let me know what I need to do :)

@TimterLaare the testing bit is what maybe needs improving I am not sure what other test cases I need to cover in order for them to be considered complete. I am also not generating a nuget package right now