mookid8000
commented
5 years ago Small clarification: The current ordering is (Priority DESC, Id ASC), meaning that higher properties take precedence, and other than that messages are ordered by their insertion time.
So.... just to be entirely clear about this (my brain is struggling to work correctly these days, probably because of the heat 🔥): What you're proposing is this: Change the ordering to (Visible ASC, Priority DESC, Id ASC)... but wouldn't that screw up the purpose of the priority column?
What if the ordering was (Priority DESC, Visible ASC, Id ASC) – wouldn't that mean that a higher priority always takes precedence, and then next comes the times at which the messages became visible – and only then then comes the ID (which will then only become relevant when messages are inserted within the same 100 ns interval, because that is the default precision of the datetimeoffset SQL type).
I think that would make the most sense – what do you think?
MrMDavidson
We noticed recently that the queries used, both for the leased and regular, SQL transport both filter out not-yet-visible messages (good) but then ignore the visibility value when performing ordering and instead only use the insertion order (See: https://github.com/rebus-org/Rebus.SqlServer/blob/master/Rebus.SqlServer/SqlServer/Transport/SqlServerTransport.cs#L308 ). If you imagine a snapshot of the queue that looks like (note:
SentAtisn't a real column... it's just for illustration purposes. But imagine it's the time at which the message was inserted into the queue)If we imagine the system has been under heavy load and is now, at 10:05, only just getting to this point in the queue the messages will be processed as 1000, 1500, and finally 2000. However the
Visibleproperty implies that the messages will be processed as (2000, 1500, 1000). This is because the current ordering is(Id, Priority). However I think it should actually be(Visible, Priority, Id).Any thoughts here? We're not expecting, or relying on, strict queue ordering however we are seeing issues under heavy load where long-deferred messages are jumping ahead of messages, with the same priority, that under low load levels would have been processed long ago. Eg. If we were consuming messages faster than they would be produced this queue would have been processed as (2000, 1500, 1000).