massadmin + django-denorm leads to performance problems... make them async?

[kennknowles]

Hi there,

I am using this package with initcrash/django-denorm and I have a problem with their interaction. It can be fixed by either package, and both fixes are useful.

Denormalization is asynchronous except when a model is saved directly. Massadmin saves the model directly, causing all denormalization to be synchronous.

Solutions:

• Do not denormalize synchronously even when using save()
• Turn off denormalizations for a bit during massadmin

These are sort of the same solution - it will involve disconnecting a signal and then reconnecting it, or something like that. I have not figured out the best way.

• Perform the massadmin edits asynchronously, such as via Django-RQ

I like this, because (a) massadmin seems to be a more active project with easier to understand code and (b) I already use django-rq for lots of asynchronous things such as some manual denormalization that is too hard to express with django-denorm.

So I am already working on this solution and it is mostly done except for troubling pickling various things. I wonder if you have opinions about this problem before I proceed further.

Cheers,

Kenn

[bufke]

Async is a bit scary because an error would not notify the end user. I'd be open to a settings.py option that is disabled by default. We also can't have a queuing system as a required dependency.

"Turn off denormalizations for a bit during massadmin" I'm not familiar with the project so I can't really comment here.

Unrelated do you like django-rq? I'm a bit annoyed with Celery but it seems the go to option.

[kennknowles]

Yea, I agree with you. This really should be the responsibility of the denorm library, or else the responsibility of my code to override the behavior. It is complex and quirky enough I'll probably go with a lightweight clone or rewrite though, which is a bummer.

As for django-rq, yes I like it. Very lightweight and painless. I'm not sure what its scaling limitations are. Surely there must be a point at which a dedicated broker like RabbitMQ is superior to Redis. It has been a while since I used Celery, and I have only used it as a "client" - it was someone else's job to keep the whole thing running smoohtly - so I can't say how much simpler django-rq might be. If you are already using redis as the backing store for celery, then it will be very painless to try it out.