toptal / chewy

High-level Elasticsearch Ruby framework based on the official elasticsearch-ruby client
MIT License
1.88k stars 366 forks source link

Add helper to cleanup delayed sidekiq timechunks #894

Closed Drowze closed 9 months ago

Drowze commented 1 year ago

For context:

https://github.com/toptal/chewy/pull/869 added the delayed_sidekiq strategy, which by design will only create a sidekiq worker to update an index every N seconds (10 seconds by default). The implementation details for that involves redis sorted sets: it creates a sorted set for each index that uses the strategy and adds a new member to it everytime we call reindex (with the weight being calculated from the current timestamp, generating a unique number every N seconds).

The problem:

Since we're writing directly to Redis, if we use this strategy in tests we're subject to state leaking and creating flaky tests. For example:

On the above scenario, test B will fail as no worker will be enqueued since import! was called within 10 seconds in between the tests (also note that Sidekiq runs in memory by default when running tests, so workers enqueued by test A are not present in test B).

The fix:

This PR adds a way to workaround the issue by cleaning up the saved timechunks. It can easily be added to e.g. rspec hooks via:

RSpec.configure do |config|
  config.after { Chewy::Strategy::DelayedSidekiq.clear_timechunks! }
end

Before submitting the PR make sure the following are checked:

konalegi commented 9 months ago

Could you please rebase PR?

konalegi commented 9 months ago

hey @Drowze , could you please rebase your PR?

Drowze commented 9 months ago

Hey @konalegi, sorry for the delay! I didn't see the notification for the first message ๐Ÿ˜…

Just rebased this PR! ๐Ÿ‘

konalegi commented 9 months ago

Hey, unfortunately rubocop is failing :( I suppose it's because we recently switched from ruby 2.x to 3.x syntax.

Drowze commented 9 months ago

I see! Just fixed the rubocop issues and updated the PR ๐Ÿ˜„