Is support for Azure DocumentDB planned?

[mcquiggd]

Firstly, very interesting project..my compliments.

I would like to know if support is planned for Azure DocumentDB; if not I will try to find time to contribute towards this,

On a similar subject, is there a roadmap available? Would love to see what the team is planning..

[mookid8000]

@hdrachmann did actually experiment a little with Azure DocumentDB when it was first announced, mostly because he was interested in seeing if it could be used as an event store... he concluded that it was not suited for that, as it cannot enforce the two unique constraints that Cirqus requires on events: 1) global sequence number, and 2) aggregate root ID + aggregate root sequence number.

I don't see any problems in using it as a view manager though - it's only a matter of implementing IViewManager

[mcquiggd]

@mookid8000

Thanks for that information...

And yes, my initial thoughts were to look at DocumentDB primarily for my Views.. I will look at creating a view manager that follows a similar structure as other implementations, and try some performance metrics etc.

Although I would be comfortable using Mongo, I have a particular requirement that the system can be deployed within Azure as a fully managed PAAS (and my client has ruled out MongoLab etc for their own reasons); hence the interest in DocumentDB, which seemed a logical choice...

Slightly off the original topic, but have you found any particular backend to show advantages over the others currently supported, as either an event store or view datastore while using Cirqus in Production?

[mookid8000]

We are currently building a solution that relies only on SQL Server - it seems to work really well with SQL Azure, both as an event store, and as a view manager (via the HybridDB package, which I can really recommend!). This way, you can have PAAS and rich LINQ-enabled views at the same time ;)

We also have a couple of solutions that use MongoDB both as event store and for the views, which also works well - you just have to remember to configure the client's connection recycling to be 60 seconds instead of the default 4 minutes, because Azure (for some reason) seems to cut TCP connections that last longer than that when there's no activity.

[mcquiggd]

@mookid8000

Again, thanks for the insights - I am going through the code at the moment, having read a Google translation of the Danish blog posts introducing Cirqus.

I will do some testing of the HybridDB approach, before embarking on a serious attempt to build a DocumentDB View Manager... but I have not abandoned that idea just yet ;)

When deploying, do you separate your systems into, for example:

Azure App Service WebSite (Build Command)

Service Bus 1

Azure Worker Role (Command Handler) - load Aggregate, apply Command, accumulate Events

Azure SQL (Event Store) - Events persisted and sent to Service Bus in same transaction

Service Bus 2

Azure Worker Role (Event Handler) - Projections

View Store(s)

[mookid8000]

Not really :) we have several projects running fine with just a single web application instance that builds and processes commands, processing events in the views as well - no service bus or anything.

I could see myself delegating view processing to other nodes to speed up that part if it became a bottleneck, but I would do a lot to avoid having truly asynchronous commands, because it's just so nice and easy to work with when command processing is synchronous.

[mcquiggd]

Yep...I can understand... ;-)

My current 'challenge' would work happily with eventual consistency... the system load could be very high, and variable. I need to be able to support scaling up of individual aspects of the system, especially, as you point out, the view processing, as there are a number of read models, and a Data Warehouse.

Plus, potentially I want to be able to add / update / remove subscribers to Events, without a service interruption, as a Microservice architecture.

[mookid8000]

But yes, in that case I would create commands in the website and push them via service bus to an endpoint that processes commands.

The view managers will catch up by themselves because they periodically check the event store to see if there's work to you.

You could minimize latency by introducing some kind of low-latency delivery of the emitted events by implementing IEventDispatcher and hooking it up to the command processor, publishing the emitted events to all view managers.

The subscribers in this case would then be one or more ViewManagerEventDispatcher instances, to which you would dispatch the published events.

The ViewManagerEventDispatcher primary mode of operation is simply by periodically checking the event store to see if it should catch up and dispatch some events to the views that it is hosting - but it will also accept an event batch dispatched to it if it happens to fit just right with the sequence numbers as the next event batch, allowing you to minimize latency with a kind of "fast-track" mechanism.

I hope that made sense :) I'll be happy to help you some more if you need it, don't hesitate to ask :)

[mcquiggd]

Thanks... I have a lot to get on with - will keep in touch :+1:

[mcquiggd]

Typical - I had allocated my weekend to polishing up my DocumentDB implementation for Cirqus, and Microsoft go and announce MongoDB protocol support for DocumentDB ;)

http://techcrunch.com/2016/03/31/microsofts-documentdb-now-lets-you-use-your-mad-mongodb-skills/

But seriously, it seems unless they are making some pretty major changes to the DocumentDB feature set it will not be a case of 'just switch your connection string'.

The good news is that there are pricing changes coming, which along with elastic scale, and now global replication groups, might well make DocumentDB a very sound choice...

I hope to put my DocumentDB provider for Cirqus up on Github this weekend.

[mookid8000]

😃 DocumentDB is definitely interesting, so it would be pretty cool to have a view manager for it. How did it go? Please let me know if you need any pointers to get going.

[mcquiggd]

@mookid8000

Well, actually it was reasonably straightforward, initially I started following the template of your MongoDB implementation, and as I progressed I gained a greater understanding of the framework, and made a couple of changes in approach to suit DocumentDB.

I have not been able to publish my current code to Github this week due to family commitments (and importantly, looking for contracts to finance my startup software company here in Brasil), but it will be soon.

I am at the stage where I have implemented the following:

EventStore (tested, working, but need to perform performance tests / stress tests) View Manager (tested, working, but need to perform performance tests / stress tests) AutomatedDistributionState (untested - need some feedback from you regarding that, and how it is triggered) Logger (tested, working) Snapshots (untested - need some feedback from you regarding that, and how it is triggered)

At the moment I have been developing with a test harness based on the SQLSug sample, with DocumentDB as the EventStore, Logger, and both DocumentDB and MongoDB implementing the three Views, and the data is correctly replicated across the Views.

Now, there are several areas where I need some feedback / opinions; and especially where I have taken a different approach, due to DocumentDB being designed to use Collections as both a Scale Unit, and a Pricing Unit, with the recommendation to store multiple types in a single Collection, and many methods being intended for async operation in order to scale. It also has throttling issues to consider, being PAAS, and the current implementation is a bit chatty with the database.

I see an opportunity to make a more performant / efficient implementation, and I am in the middle of coding another version that takes more advantage of DocumentDBs strengths in terms of scale; again I would like to get your input on that.

When I am able to publish my code so far, I am going to open a new 'issue', to keep information more easy to find...

David