[Question] Hesitating between sidecloq and cron + sidekiq-unique-jobs

[allaire]

Hi @mattyr !

Our team is looking to move away from a single app server to a distributed infrastructure. Our plan is to have multiple app servers (Rails w/ Sidekiq on each), and a DB/Redis server. The current issue is where the CRON (currently using whenever, executing Sidekiq tasks) should go.

That's why we started looking for gems doing something similar. Right now we're hesitating between using sidecloq all the way, or use CRON + sidekiq-unique-jobs on each server.

Also found sidekiq-cron, but they have a lot of open issues, and a big one about missing half jobs.

Do you have an opinion on the matter?

Thanks for your work!

[mattyr]

I actually created sidecloq with the primary goal being to replace similar deployment setups. In the past I've done the cron + workers on same machine, cron and workers on separate machines, or running other schedulers in concert with workers. In my opinion, those were overly complex and/or prone to create SPOF's, scheduling race conditions, or basically became monitoring/maintenance headaches. I wanted something where:

• my configuration for workers and any scheduled work tasks was deployed in the same way I deploy just the workers.
• the scheduler was neither a SPOF, but also did not take excessive resources (granted, cron/most schedulers are and should be incredibly light-weight) if run on every machine
• didn't rely on something like sidekiq-unique-jobs to make sure that multiple identical scheduled tasks were not scheduled concurrently (i believe that scaling massively using the approach of pushing from everywhere and just letting one job "stick" could become problematic, but i have not tested this theory)

Did this help? Sidecloq aims to please... and be simple. I'm super curious if there are specific features that you feel are missing. I don't want to recreate another existing tool, but the way you describe your setup sounds really similar to 2 or 3 I've replaced with sidecloq in the past.

[mattyr]

... also, Sidecloq should be much less resource intensive on the Redis side when run at scale than sidekiq-cron -- the resources it consumes per-worker are drastically less.

[allaire]

@mattyr Excellent, thanks for the thorough replies!

Curious, are you using Sidecloq in production? It seems relatively new, but built on popular gems.

Also, is there any issue when deploying the app (rolling restarts of sidekiq/puma) or if I kill PID the leader worker?

Thanks!

[mattyr]

I am using it in several projects -- it actually evolved into a gem from one project that has been running for a couple of years. That project actually continually runs multiple workers, and there are bursts of deploys to it (including rolling restarts) that can peak at several deploys per hour. I have never had any kind of issue with sidecloq itself (the simple leader election ensures there isn't ever a conflict of different code versions pushing jobs simultaneously).

[mattyr]

if you kill the leader, another leader will take over automatically within a short period of time. in a 1-worker deployment, killing the leader does kill the scheduler, so that's total scheduling system failure.

[mattyr]

going to close this for now, but feel free to re-open with additional questions. feedback is also much appreciated!

[allaire]

@mattyr sure thing! We're going to move forward with sidecloq. I can report in after we implement it 👍