until_and_while_executing with separate server and client lock_args broken since 8.0.8

[Roguelazer]

Describe the bug

After upgrading from 8.0.7 to 8.0.10, jobs with until_and_while_executing that pass a custom lock_args_method which behaves differently on the server and the client (as shown in the README) no longer work. Based on bisection, this appears to have been broken by 63e943109479dcf610f8bd5b19652c87d7086187 (the parent commit, 7d5e40a06cbd927f928f3ebaadff8010f07b869d, still works).

Expected behavior Given a worker as shown below, run the following client code:

1.upto(2)
  UniqueBugWorker.perform_async(id)
  UniqueBugWorker.perform_async(id)
  UniqueBugWorker.perform_async(id)
  UniqueBugWorker.perform_async(id, "foo")
  UniqueBugWorker.perform_async(id, "foo", "bar")
end

We expect to see exactly six invocations.

Current behavior Since 63e943109479dcf610f8bd5b19652c87d7086187, we end up seeing at most two invocations (and sometimes only 1, which is even more worrying). It appears that the lock digest is being reused between the until and while executing phases.

For some reason, the conflicted jobs also aren't getting rescheduled; they just get dropped on the floor.

worse yet, we end up creating and leaking uniquejobs: locks with the correct digest, so after downgrading to 8.0.7, the jobs are locked out until the lock TTL expires.

Worker class

class UniqueBugWorker
  include Sidekiq::Worker
  sidekiq_options queue: "repro",
    lock: :until_and_while_executing,
    on_conflict: {server: :reschedule, client: :log},
    lock_args_method: ->(args) do
      if Sidekiq.server?
        [args.first]
      else
        args
      end
    end

  def perform(id, foo = nil, bar = nil)
    Rails.logger.info { "UniqueBugWorker: performing #{id} with #{foo} #{bar}" }
    sleep 1
  end
end

Additional context Add any other context about the problem here.

[reneklacan]

We seem to have been running into same issues since upgrade but with until_executed (possibly with other locks as well but I only have confirmation about issues with until_executed)

[mhenrixon]

@reneklacan do you also use custom lock args?

[reneklacan]

@mhenrixon no