search

rebus-org / Rebus.SqlServer

:bus: Microsoft SQL Server transport and persistence for Rebus
https://mookid.dk/category/rebus
Other
43 stars 42 forks source link

DataBus Attachments removal strategy ? #10

Closed IsaacSee closed 6 years ago

IsaacSee commented 6 years ago

Hello, first thanks for the good job on the Rebus project. I am using the Sql.DataBus and discovered that my table dbo.tDataBus grew up to 50 Go. After some researches I can not find how attachments should be cleaned/deleted. What am I missing ?

mookid8000 commented 6 years ago

There is no was out of the box for Rebus to handle this because Rebus has no way of knowing when to remove an attachment.

Therefore it is made such that all attachment storages are capable of somehow indicating when an attachment was last read, allowing you to manually set up something that removes old attachments.

With the SQL storage, the [LastReadTime] column reveals when the attachment was last read – maybe you can use that to come up with a strategy for deleting old attachments?

I have thought about adding an API to Rebus that helps with managing attachments, but I haven't come up with anything yet. Do you have any ideas on how that could be done?

IsaacSee commented 6 years ago

Thanks @mookid8000. I understand the complexity of a generic strategy for attachments deletion.

I think first and quick approach would be to defer the deletion decision to the lib user with a simple API to manually delete an attachment when we know it has been consumed. I naturally searched it on the DataBusAttachment class and on the IDataBus interface.

Concerning the SQL storage, if I am not mistaken, my problem for an external Delete all attachments witch have not been read since 2 days routine was that before any read the [LastReadTime] remains null and we do not have a simple way to know when the attachment was created (to avoid deleting a freshly created which has not been read yet and/or one that will never be read). It might be nice to add a [CreationDate] or always setup the [LastReadTime] on creation (the creation would then be considered as a read).

mookid8000 commented 6 years ago

Concerning the SQL storage, if I am not mistaken, my problem for an external Delete all attachments witch have not been read since 2 days routine was that before any read the [LastReadTime] remains null and we do not have a simple way to know when the attachment was created (to avoid deleting a freshly created which has not been read yet and/or one that will never be read). It might be nice to add a [CreationDate] or always setup the [LastReadTime] on creation (the creation would then be considered as a read).

Damn – I never considered the case where an attachment would never be read.

The SQL storage could easily set another timestamp at the time of creation. I created an issue for it here: #11

IsaacSee commented 6 years ago

That should do it ! I close the question as I have my answer, thank you.