Why picking a cron-based implementation over a database-based solution

[olivierobert]

The current implementation relies on a crontab-based scheduler to pick up jobs at a fixed rate of 1 minute. It works well so no issues.

But did you consider using the built-in scheduler of Rails called ActiveJob which is backed with Redis?

[nattaponaie]

Well, I have considered about this built-in scheduler before. According to the requirements, I only needs a simple scheduler to run a job with specific time.

In my opinion, Rufus-scheduler library is answer the question. This is because, it's easy to setup and use.

However, if the task itself takes a lot of times, memory when execute. Then, I would reconsider to move on Sidekiq as it uses memory to improve.

[olivierobert]

Noted.

The downside of the current implementation is that the 1-minute frequency is kind of arbitrary (why 1 minute and not 30 seconds) and would result in less prompt response time. If a job is created upon the upload then the scraping will start almost immediately if there are no other jobs in the queue. Using a queue system also allows parallelizing jobs too.