architecture plan

[jamesturk]

The current version of bobsled was sort of thrown together as-needed. I've been thinking through what we actually need. Here's an outline of a plan for a future-proof bobsled.

Needed endpoints:

• [ ] using git as a primary data store for jobs. scheduling/etc. can be updated in CloudWatch w/ a post-commit hook (alternatively, could switch to using a beat & checking GitHub)
• [x] / should have an overview of each task, and colored boxes indicating success/failure
• [ ] /run?id=, - details on specific run(s)

Lambdas:

• [x] ecs task that starts a job running and stores a new run record to a DB
• [x] task that runs every ~10 minutes and updates status of all running jobs
• [ ] autoscaling of cluster (can be manually defined in a setting, but will save $)

Job States:

• RUNNING (initial)
• FROZEN (still running, no updates in 1 hour)
• ERROR (found Traceback)
• SUCCESS (stopped running, no traceback)

Example Job Schema:

name: NC
entrypoint: ./pupa-scrape.sh nc
memory_soft: 128
cron: 0 4 * * ? *
environment: openstates-pupa
image: openstates/openstates-pupa
enabled: true

Example Run Schema:

job: NC
entrypoint: ./pupa-scrape.sh nc
start: 2017-03-18
end: null
status: RUNNING
started_by: automatic
traceback: ...

[jamesturk]

Pacific states are now running new lambda as a test, should be ready for status check task