Message retry improvements.

[hackaugusto]

Abstract

• Remove the dependency of among RetryQueue and the MatrixTransport. To do this the RetryQueue should be polled for changes instead of a pushing text. The current design is forcing the use of mocks for testing, which makes it harder to understand the intention of a test.
  • Make RetryQueue a plain old data object, instead of a separate thread.
• Have only one exponential backoff for message retries. The queue exists because the messages have to be processed in order, retrying a new message will have no side effect if the older messages have not been processed yet. Having a retry generator per message is just confusing and wasteful.
  • Remove the expiration generator, this won't be necessary once only a single backoff is used.
• Only wake up the RetryQueue if there are messages to be retried. IOW, the retry_interval has to be dependent on the current expiration.
• Do not add unordered messages to the queue, namely Ping, Pong, and Delivered. These messages by definition are not retriable, so they don't have to go to the queue and have special handling there.

[hackaugusto]

~As an additional side note: Since Matrix is a chat room, the messages do persist for some time, retrying is totally necessary under most circumstances, with the exception of when an operator clear the messages to free up space. Therefore the retry timeout can be significantly large. Ideally we would be able to detect if a message was removed from the room, and only retry then.~

Edit: I forgot one scenario, retries are necessary to fix race conditions against the blockchain, so the retry interval cannot be large, it has to be around the expect time it takes for a block to propagate through the network.

[ulope]

Regarding your last point, there is #4110 which is somewhat related.

[andrevmatos]

About 2nd, the reason it was done like that is performance: having a per-message generator allows us to batch all messages that need to be sent in the same time window (I think rn the loop is ~5s). Specially on bursts of messages to the same room, this can save us several requests. Just a headsup though, go ahead if you think it's not worth it.

[hackaugusto]

@andrevmatos did you measure the performance impact? Do you have a reference to the results?

[andrevmatos]

No, we didn't, but everytime you see N messages batched together, you can assume the http roundtrip time for sending these messages was cut ~N times