Me again! We have been digging at this issue and although the method causing the issue itself:
def hostname_registered?(hostname)
SidekiqAlive.registered_instances.any? do |ri|
/#{hostname}/ =~ ri
end
end
was removed here we were still facing a similar issue. After some digging it's still the same problem we found here.
We still need to use the registered_instances method and that ultimately uses a SCAN that's O(N) and we have so many jobs that our sidekiq could have potentially millions of keys at the time so SCAN is not an option.
And since the goal of that method is just to return the registered keys I don't think SCAN is necessary. In this PR I'm just creating a set of keys (registered instances) and the score of each key is their TTL. This way we can retrieve them without having to go through everything on redis.
Hi @arturictus !
Me again! We have been digging at this issue and although the method causing the issue itself:
was removed here we were still facing a similar issue. After some digging it's still the same problem we found here.
We still need to use the
registered_instances
method and that ultimately uses a SCAN that's O(N) and we have so many jobs that our sidekiq could have potentially millions of keys at the time so SCAN is not an option.And since the goal of that method is just to return the registered keys I don't think SCAN is necessary. In this PR I'm just creating a set of keys (registered instances) and the score of each key is their TTL. This way we can retrieve them without having to go through everything on redis.
What do you think?