Closed Sovetnikov closed 4 years ago
At the moment, messages are just bytes
that form part of the Task
model. Parsing out queue_name
and actor_name
from this byte stream as new fields on the Task
model duplicates the data and probably breaks the neat encapsulation that we currently enjoy a little.
@Bogdanp, is there a reason why messages are retained in encoded format as part of a BinaryField
? If not then creating a foreign key to a fully parsed and indexed new Message
model feels like a cleaner approach?
@jcass77 actor name and queue name stored in model just speeds up the search in these fields. We have more than 100 000 messages in two weeks and search becomes unusable. Binary message representation have lots of information that not in model fields, like traceback (I have doubts whether this works, will check this out on failed messages), eta, arguments, pipelines and callbacks data, that very handy in admin interface. Storing all message attributes in model fields too complex and needs regular maintenance if changes will be made in dramatiq, so full message storage is fine here.
Storage of tasks state and history is temporary by it's nature i think, and performance is more important than normalization and storage size.
Thanks! This looks good and I'll take a deeper look at it next weekend. I am fine with denormalizing part of the message if it helps in use cases like these, though I don't know if I'd really recommend using the admin middleware in a performance-sensitive environment in general.
Thanks! I've rebased and merged your changes. I'll cut a release next weekend. Sorry this took so long, but I've been extremely busy recently.
Just having very poor performance without this changes.