jmalloc / ax

A message-driven application toolkit for Go. [EXPERIMENTAL]
MIT License
9 stars 3 forks source link

Temporarily increase RMQ pre-fetch while holding onto delayed messages before retry. #82

Closed jmalloc closed 6 years ago

jmalloc commented 6 years ago

This does address #31, but still doesn't help that much without knowing how many times the message has been delivered.

@danilvpetrov can you think of a way (probably with DLX) that we can get a redelivery count from RMQ without introducing a delay between retries at the transport layer itself?

jmalloc commented 6 years ago

This issue also kind of implements the opposite of #48, so I should probably change that.

codecov-io commented 6 years ago

Codecov Report

Merging #82 into master will decrease coverage by 1.5%. The diff coverage is 0.92%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #82      +/-   ##
==========================================
- Coverage   34.14%   32.64%   -1.51%     
==========================================
  Files          56       56              
  Lines        1883     1973      +90     
==========================================
+ Hits          643      644       +1     
- Misses       1230     1318      +88     
- Partials       10       11       +1
Impacted Files Coverage Δ
src/ax/endpoint/transport.go 100% <ø> (ø) :arrow_up:
src/axrmq/topology.go 0% <0%> (ø) :arrow_up:
src/axrmq/transport.go 0% <0%> (ø) :arrow_up:
src/axrmq/consumer.go 0% <0%> (ø) :arrow_up:
src/axrmq/acknowledger.go 0% <0%> (ø) :arrow_up:
src/axrmq/marshaling.go 0% <0%> (ø) :arrow_up:
src/ax/endpoint/receiver.go 0% <0%> (ø) :arrow_up:
src/axrmq/publisher.go 0% <0%> (ø) :arrow_up:
src/ax/endpoint/retry.go 7.14% <7.14%> (+7.14%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 476c7a8...e272315. Read the comment docs.

jmalloc commented 6 years ago

rabbitmq/rabbitmq-server#502 would help immensely here.

jmalloc commented 6 years ago

This implementation now has a fairly reliable delivery count which is obtained by using the DLX features to return a message to its original queue when we reject it.

This means we can't use the usual rejection feature for actually rejecting the message. Instead it is explicitly re-published to the error queue, and then the original message is acknowledged.

In the face of hard-crashes this may result in duplicate messages on the error queue, or messages both in the pending and error queues, however the outbox ultimately deduplicates all of these when they are redelivered.

jmalloc commented 6 years ago

BTW, I don't know if I really want to merge this, but it does seem to work.