assimbly / runtime

Java & REST API's for creating and running integrations
Apache License 2.0
17 stars 3 forks source link

[Scheduler]Cron seems inactive after a while #238

Open Hooghof opened 3 months ago

Hooghof commented 3 months ago

On the insights instance, after running well for about 14 days, the flows starting with a scheduler and set to cron selector no longer appeared to start at the set time. Restarting the flows did not appear to solve the problem. It seems that the problem lies at a higher level, quartz perhaps? It turned out that the same flow in the test backend was still working, but not in production anymore.

After a short discussion with Raymond I found this, perhaps related? https://github.com/Talend/apache-camel/blob/master/components/camel-quartz/src/main/docs/quartz-component.adoc

skin27 commented 3 months ago

The link is the same as the official documentation:

https://camel.apache.org/components/next/quartz-component.html

Based on the documentaiton it may happen because of a "misfire". A "misfire" means they are not executed as expected at the scheduled time. A misfire can have many cause, but in our case I think it's either running out of resources (CPU/Memory) or a concurrency issue. Other less likely causes are time syncrhonization problem, misconfiguration or network latency.

The idea is to have the following steps:

  1. Check if this also happens in Camel 4.
  2. If so change to configuration (in the frontend):

Add interruptJobsOnShutdown=true Add recoverableJob=true Add &trigger.misfireInstruction=2 (of 3 of 4)

  1. If still causes problem than it also possible to create a custom scheduler so we may add more logging to it. By default this should be avoided.

As alternative we may test if we can use the cron component as this doesn't use quartz. For the repeater we can use the timer (or scheduler) component.

skin27 commented 3 months ago

Move this to active. IF we run into same problem then reopen this issue with all remarks.