ruby-shoryuken / shoryuken

A super efficient Amazon SQS thread based message processor for Ruby
Other
2.05k stars 279 forks source link

Stop the dispatching of new messages when a SIGTERM signal has been received #750

Closed hoffi closed 11 months ago

hoffi commented 11 months ago

Fixes #749

phstc commented 11 months ago

@hoffi thanks for the submission. That's greatly appreciated.

Some specs are failing but don't seem related to your change. I don't recall those being flaky before. But it seems they are also failing here https://github.com/ruby-shoryuken/shoryuken/pull/747

Can you have a look? 🙏 👀

hoffi commented 11 months ago

The enqueued_at timestamp in job.serialize differs slightly from the value in hash here: https://github.com/ruby-shoryuken/shoryuken/blob/main/spec/shared_examples_for_active_job.rb#L47

I have looked into the #serialize method from ActiveJob and it always returns a new timestamp: https://github.com/rails/rails/blob/7-1-stable/activejob/lib/active_job/core.rb#L122

But the correct enqueued_at timestamp is not stored in the job object, so i'm not sure where to get it. Should we use something like ActiveSupport::Testing::TimeHelpers#freeze_time to ensure the same timestamp is used?

Or just use the hash argument instead of job.serialize and generate the hash without the job_id from that?

phstc commented 11 months ago

@hoffi freeze_time seems like a good idea, WDYT? It seems the hash requires more changes.

I don't know why it's only happening now 🤷

hoffi commented 11 months ago

@phstc i have added the freeze_time helper which fixed the test on my machine

phstc commented 11 months ago

@phstc i have added the freeze_time helper which fixed the test on my machine

@hoffi That's awesome! Thank you

https://github.com/ruby-shoryuken/shoryuken/actions/runs/6644142158/job/18052725694?pr=750#step:4:34

undefined method `freeze_time' for #<RSpec::ExampleGroups::ActiveJobQueueAdaptersShoryukenAdapter::Enqueue::WhenFifo:0x00555b2f3f[34](https://github.com/ruby-shoryuken/shoryuken/actions/runs/6644142158/job/18052725694?pr=750#step:4:35)48>
     Shared Example Group: "active_job_adapters" called from ./spec/shoryuken/extensions/active_job_adapter_spec.rb:6
     # ./spec/shared_examples_for_active_job.rb:49:in `block (4 levels) in <top (required)>'

Unfortunately, it's failing for Rails 4. Is there any alternative we could use? Maybe https://github.com/travisjeffery/timecop?

hoffi commented 11 months ago

Yes, i have seen now it is only available since rails 5.

However i just realized that it might be problematic to include the enqueued_at timestamp in the deduplication id?

isn‘t it the same as with #457?

phstc commented 11 months ago

@hoffi hm for standard works, we use only the body https://github.com/ruby-shoryuken/shoryuken/blob/f02d28b4bf79258233e977736d1fd878e203a687/lib/shoryuken/queue.rb#L133

It's probably OK to remove enqueued_at. The uniqueness should be based on what the user sent, not on what was generated.

It makes me a bit uncomfortable to change something like that after so long (especially given that it was working before), but rationally thinking makes it feel safe.

I'm up for either change: Timecop or removing enqueued_at.

If you prefer the enqueued_at removal, can you test it locally to check if the deduplication works?

hoffi commented 11 months ago

It makes me a bit uncomfortable to change something like that after so long (especially given that it was working before), but rationally thinking makes it feel safe.

Yes however it was changed in rails 6 that the enqueued_at timestamp is returned in the message body:

If you prefer the enqueued_at removal, can you test it locally to check if the deduplication works?

I will try to see if i can test it

phstc commented 11 months ago

Yes however it was changed in rails 6 that the enqueued_at timestamp is returned in the message body

@hoffi if that's the case, let's remove it then. It feels way safer given that ^ the deduplication was added prior to Rails 6.

Thanks for thoroughly reviewing this.

hoffi commented 11 months ago

@phstc i have removed the timestamp. Can sou take a look at it?

phstc commented 11 months ago

@hoffi looks great! Awesome work. Thank you

hoffi commented 11 months ago

Nice! Thanks :)