Recovering from a loss of data integrity / support for transactions

[isaac-peka]

I'm running into some problems keeping data integrity between getstream and our database with transactions enabled, although the bigger problem is recovering from any loss of integrity. Problem is that at any point in the transaction the model could fail to save/delete and rollback, but the activity still remains in getstream.

At the moment I've had to settle on a hacky solution which just filters out any activity that fails to be linked to an object during the enriching process, but this feels a little bit inadequate. Ideally there would be a management command that I could run during maintenance to remove any deleted models from getstream, something like haystack's rebuild/update index commands, but looking at the api I'm not sure if that's possible.

Do you have any general advice / is there a way to build in better support for transactions?

[tbarbugli]

unfortunately Django does not expose transactions callbacks (only 1.9 does it) so it is fairly hard for the integration library to detect rollbacks / execute only after the transaction is commited out of the box. One way to do this is to disable the default behavior and manually perform activity inserts/deletes in your own app. Feel free to add some code from your app o add more description of your use case.

[isaac-peka]

Not sure manually inserting/deleting is really the best way to go since it adds a lot of complication and editing code gets a lot more involved.

Actually, what you mentioned about the transaction callbacks being exposed in 1.9 helped me stumble across a library that provides transaction commit callbacks for Django 1.6+, so thanks for that :) https://django-transaction-hooks.readthedocs.org/en/latest/.

[isaac-peka]

Here's one example of what I mean (in regards to cleaning up). I've set up a manual signal handler that creates notifications only. Something like this:

@receiver(post_save, sender=ConversationMessage)
def send_message_notification(instance, created, **kwargs):
    if created:
        recipients = instance.conversation.users.exclude(pk=instance.author_id)
        feeds = ["notification:%d" % u.id for u in recipients]
        stream_client.add_to_many({
            'actor': create_reference(instance.author),
            'verb': 'message',
            'object': create_reference(instance.conversation),
            'message': create_reference(instance),
            'time': instance.created_at,
            'foreign_id': create_reference(instance),
        }, feeds)

How would I write the other side of this handler, i.e. the post_delete signal? Because the notifications are not bound to a source feed.