[fix] delayed_sidekiq race condition

[skcc321]

In the PR I'm addressing two issues with delayed_sidekiq strategy.

1. There is a chance for race conditions exactly in this place.

           unless redis.zrank(timechunks_key, timechunk_key) # read
             redis.zadd(timechunks_key, at, timechunk_key) # write

   I eliminated it using Lua script and making the combination of operations atomic.

2. Also, there was a possibility to bloat Redis if there is an issue with reindexing (ES connection timeout for instance) Before submitting the PR make sure the following are checked:

• [x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
• [ ] Wrote good commit messages.
• [ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
• [x] Feature branch is up-to-date with master (if not - rebase it).
• [x] Squashed related commits together.
• [ ] Added tests.
• [x] Added an entry to the changelog if the new code introduces user-observable changes. See changelog entry format for details.

[konalegi]

please ping me once you are ready to merge. I'll release it

[konalegi]

Please fix rubocop issues, and something is odd with our specs and CI in general :/

[konalegi]

Please take a look at your PR to see what's wrong. I created a separate PR to check CI and it seems to be working. https://github.com/toptal/chewy/pull/942

[ebeagusamuel]

@skcc321 Please try rebasing on toptal/chewy master branch.

[skcc321]

Sorry for the delay, guys. Yep, I rebased it on upstream/master today. So, we've been testing it on prod 2 weeks. Looks like it completely resolved the race condition issue. @konalegi, @ebeagusamuel should be ready to merge now. Could you guys restart the CI to verify the PR?

[konalegi]

Since you've started using real redis in specs, you have to redis service, smth like and provide proper env variables for sidekiq/redis depending how you want to test it.

jobs:
  ruby-3:
    services:    
      redis:
        # Docker Hub image
        image: redis
        # Set health checks to wait until redis has started
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

[skcc321]

Since you've started using real redis in specs, you have to redis service, smth like and provide proper env variables for sidekiq/redis depending how you want to test it.

jobs:
  ruby-3:
    services:    
      redis:
        # Docker Hub image
        image: redis
        # Set health checks to wait until redis has started
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

done. let's try again the CI run.

[konalegi]

@skcc321 still failing :/

[skcc321]

@skcc321 still failing :/

OK, I'll take a look tonight.

[skcc321]

@konalegi one more attempt please

[skcc321]

@konalegi looks like two failed checks are not related to the changes I made. could you restart them?

[skcc321]

@konalegi one of them has passed. Could you restart the last one?

[konalegi]

Great, merging!