hyrise / Cockpit

A Cockpit for Self-Driving Databases.
15 stars 3 forks source link

Add custom job scheduler #754

Closed Alexander-Dubrawski closed 3 years ago

Alexander-Dubrawski commented 4 years ago

This PR is a draft for a custom job scheduler. At the moment we are using APScheduler as a job scheduler. For our purpose, this scheduler is quite complex. Moreover, we are getting some undefined behavior if we want to stop the execution of the jobs (sometimes the jobs are stop and sometimes not). The custom job scheduler works as follows:

  1. Register a job via a function.
  2. create a job handler thread (this handler will execute the job every x seconds. If a job is still running it skip the execution).
  3. The job executes the function from 1 every interval in a new thread (task).
  4. The task thread sends an event if the job was executed so that the handler can know if it needs to skip the execution or not.

You can reproduce the strange behavior of the APSScheduler with

Code ```python from apscheduler.schedulers.background import BackgroundScheduler import time def background_job(): print("run") if __name__ == '__main__': scheduler = BackgroundScheduler() start_worker_job = scheduler.add_job(func=background_job, trigger="interval", seconds=0.01,) scheduler.start() print("Scheduler started") time.sleep(0.1) scheduler.shutdown(wait=True) print("Scheduler paused") time.sleep(2) scheduler = BackgroundScheduler() start_worker_job = scheduler.add_job(func=background_job, trigger="interval", seconds=0.01,) scheduler.start() print("Scheduler resumed") time.sleep(0.1) scheduler.shutdown(wait=True) ```

Some times it working and sometimes we get the error message RuntimeError: cannot schedule new futures after shutdown

Bouncner commented 3 years ago

Close?

Alexander-Dubrawski commented 3 years ago

At the moment the APScheduler is just working fine. If we want to replace it in the future with a custom scheduler, we can come back to this PR.