meetings / gearsloth

Gearman job persistence and delayed execution system
MIT License
2 stars 0 forks source link

initial demo of new runner-adapter logic with fs adapter #74

Closed amv closed 8 years ago

amv commented 10 years ago

THIS IS NOT READY TO MERGE.

It is meant to showcase the proposal for new runner-adapter logic and the new ejector logic - and acts as a place to discuss and make them better. The code is not tested through, but should give a good idea of the execution path.

The idea with the new logic would be the following:

  1. Runner's listenTask call provides adapter a callback( error, task, domain, db_state ) -parameter.
  2. Adapter fetches tasks, locks them for processing and calls callback(null, task_object, db_domain, { lock_or_transaction }) for each task individually.
  3. Runner guarantees that if it does not receive the first error parameter, it will always do it's best to end the callback execution to a call to one of the following adapter functions: disableListenedTask( task, db_state, callback ) OR updateListenedTask( task, db_state, callback ).
  4. Each of these adapter functions use the callback to signal that "disable" or "update" has been processed, and passes an Error as the first parameter if it did not.
  5. If runner receives a callback call without Error as the first parameter, it submits the task to the controller job.

A succesfull execution path could look like this:

Runner.startPolling()
  Runner --> Adapter.listenTask( function task_callback( error, task, domain, db_state ) {} )
    Adapter.start_poll_loop();

Adapter poll loop found something:
  LOCK = Adapter.get_lock( { TASK } )
  Adapter --> Runner.task_callback( null, { TASK }, 'default_db', { lock : LOCK }
    Runner.add_retry_timeout_to_task( { TASK } )
    Runner --> Adapter.updateListenedTask( { TASK }, { lock : LOCK }, function submit_task_callback( error ) {} )

Adapter database update succeeds:
  Adapter.release_lock( LOCK )
  Adapter --> Runner.submit_task_callback( null )
    Runner: TASK.eject_function = 'gearsloth_eject-' + domain
    Runner --> Client.submitControllerJobBg( { TASK } )

Controller gets and executes tasks:
  Controller.eject_task( TASK )
    Client.submitJob( TASK.eject_function, { TASK } )
amv commented 10 years ago

Changed the interface of the adapter a bit to keep the eject task name generation out of the actual adapter implementations.

Adapters got a new getDomains( callback( error, db_name_list ) ) -call, which should resolve names of the of the separate databases this current adapter is connected to.

The database name of each task is also passed to the runner as part of the listenTask callback (as the parameter "domain") so that the eject_function name can be properly added to the task before it is passed on to the controller. This way runner does not need to parse the name from the id (which is good as id implementation can be highly adapter dependent).

None of the adapter specific tasks work yet but unit and e2e test should be updated to match the new spec. I'll go and update the original pull request description if I can to match the small changes made.

amv commented 10 years ago

The current version does some initial work for #68 and implements #65.

DB initialization was also moved inside the constructor which should ease development on #66 and #25.

Make test runs now with a limited test coverage BUT if you run the unit tests before the integration tests (and not the other way around like we do now) inside the same mocha runtime, stuff starts to break... So there is still a lot to work with in the current tests.

amv commented 8 years ago

This pull request has gone stale. The sqlite branch has been built on top of this branch, and there should be a pull request to merge that instead.