Closed andreasohlund closed 10 years ago
Based on our discussion: We'll implement this now and postpone implementing command-line tool (https://github.com/Particular/ServiceControl/issues/114) until more user requirements justify it.
@andreasohlund - can we set this to RC / v1.0 release ?
Yes
Sent from my iPhone
On 18 nov 2013, at 12:17, Danny Cohen notifications@github.com wrote:
@andreasohlund - can we set this to RC / v1.0 release ?
— Reply to this email directly or view it on GitHub.
Milestone set to RC
adding some discussion points with @johnsimons:
@johnsimons - please add, if I have missed out anything else based on our discussion.
SC is about providing real time data to SP + SI.
Storing customers audit data is not really the primary focus of SC. If SC had to store 7 years of audit data, I am quite sure we wouldn't be able to run an efficient and super fast monitoring backend.
We need to consider offloading this historical data somewhere else.
@andreasohlund @dannycohen Thoughts?
There are several requirements on issues of audit / data retention / online monitoring etc.
Having said that, I would say that this issue, of supporting audit data expiration, is a nice-to-have: It goes half way in the direction of allowing users to periodically clean up some of the redundant stuff on their SC database (compared to the command-line functionality described in #114, setting expiration only supports future clean up policy and is a lot less of a powerful clean up tool for Opie or Archie)
We need more customer feedback on these issues in order define requirements we can be certain about.
Gentlemen, for what it is worth, I tend to fall in the same camp as Danny. We desparately need the real time insight (no pun intended) that these tools offer. Indu has some great points as well and while it is only my opinion, I think separating the two and providing two separate and distinct ways of managing the processed messed and errored messages is the way to go. Even it if it requires creating a another config setting that pushes processed messages to the audit queue for SC purposes and another one that pushes it to a Archive queue.
Regarding the automatic expiration of messages (i.e. Audit), my organization desparately would like to see this implemented. As we have killed our Dev server twice. We are pumping millions of records through and it is chewing up disk. We tried deleting the collection the other day and it was deleting about 1000 messages every 3 seconds which was unacceptable. I tried finding a way to delete the messages in the Silverlight IDE using a query that looks for the message before a specific sent date, but I was unsuccessful. I am sure this can be done in the Silverlight app, but regretabbly, I did not land on the right link to show me the way. Hence, last friday I started adding a page to our web portal do exactly that as this script seems to be all over the place on the web. But again this is a problem also because we currently cannot access SC from another server (which this bug is being addressed int he next release). Anyway, long story short, not all of your users are RavenDB experts, far from it, and having an automated solution for managing the number of messages in the database based on a given date will go a long way to easing the concerns of nervous managers and developers.
Guys, a simple solution can be the RavenDB "expiration bundle" that can be configured to automatically delete documents older than a certain date. just an idea.
Hi @JeffreyAllenMiller We already doing the "archiving queue", see #183
Regarding auto expiration and the RavenDB "expiration bundle", it sounds like we need both auto expire and also prune based on quota. The later is harder to do because we need to calculate used space and prune older data. @synhershko do you know if Raven supports this?
@johnsimons yes through the quotas bundle https://github.com/ayende/ravendb/tree/master/Raven.Database/Bundles/Quotas
Depending on the actual requirements, I'd recommend combining both to work based on the required logic. I wouldn't be surprised if having both running in parallel will cause bugs.
I wouldn't be surprised if having both running in parallel will cause bugs
Hmmm?
Strike that, the quotas bundle isn't expiring, it just stops you from writing past the quota. You (or me) will have to roll your own based on expiration & quotas.
@johnsimons there was a consensus to include this feature into 1.0.0
The expiration module is about to be ready to be merged in. I need you guys to decide on which messages exactly to expire and delete.
This is what I have currently:
ProcessedAt
attribute with a Status of Successful
only (e.g. not expiring ResolvedSuccessfully
which is the status of a message that previously failed).There may be subtleties like message statuses, message types, IsSystemMessage etc that need to be accounted for. /cc @johnsimons @andreasohlund @dannycohen
@synhershko - thanks!
ResolvedSuccessfully
that would be nice, but optional.@synhershko - thanks!
We now support expiration. Writing down the configurations stuff now.
Use the expiration bundle to have raven auto expire audit messages.
Note: We're only expiring audit messages, not errors, heartbeats etc
Replaces https://github.com/Particular/ServicePulse/issues/64