Add spec testing framework into Jobson mainline

adamkewley commented 6 years ago

External to the open-source project, I've written a small test framework that tests Jobson specs end-to-end via the API. This has been very useful for finding production / integration bugs.

Current implementation:

Each spec dir may contain a tests.yml file. The tests.yml file contains job inputs and expectations (example)
A python script (here) scans through the specs for each test.yml file
The script submits requests via the Jobson API and checks that the job runs (or doesn't), meets expectations, etc.

This is just a hacky implementation but has smoked out a bunch of integration bugs over the months I've been using it. This issue is for designing a cleaner test framework directly into Jobson so any consumers can quickly test specs, rather than having to do what I did.

adamkewley commented 6 years ago

Made a repo for the hacky python impl:

https://github.com/adamkewley/jobson_systemtests

adamkewley commented 5 years ago

Added to wishlist, closed because this ticket is stale.

adamkewley / jobson

Add spec testing framework into Jobson mainline #19