CityofToronto / bdit_flashcrow

Working repository for MOVE, a project to modernize transportation data systems at the City of Toronto.
MIT License
9 stars 2 forks source link

Multi-location: scaffold for job execution framework #556

Closed candu closed 4 years ago

candu commented 4 years ago

Description

554 and #555 form the basis for implementing scheduler on top of pg-boss (or, if needed, other tools). This task starts the actual implementation by creating a hierarchy of job-type-specific executor classes, and by integrating pg-boss within JobController.

Acceptance Criteria

Additional Notes Note that pg-boss initialization will need to be integrated with test:db-startup scripts at this point - totally doable, and should build off similar parts of #555 .

candu commented 4 years ago

Hit a couple of midsize challenges here.

First was authentication: to relate jobs to users, scheduler needed to share user sessions with web, which meant moving that out of its in-memory store in web and into database.

Second was timestamp handling: pg-boss uses TIMESTAMP WITH TIME ZONE columns, whereas MOVE uses TIMESTAMP WITHOUT TIME ZONE with the convention that all timestamps are jurisdiction-local. This meant adding a new set of types and validators to handle pg-boss timestamps.

Total impact here was roughly a day of work - though the tasks here did take into account a bit of unexpected additional time, so this isn't problematic by any means.