Open SeanTAllen opened 6 years ago
@jemc @Theodus any thoughts on what automated tests for stable would look like?
Maybe it could be an integration test that sets up an example bundle.json
that clones its own repo (pony-stable
), and imports some types to prove that it worked?
@jemc i was just thinking the same thing. and then compile the program with stable env ponyc
.
we can verify
add
(this could be expanded for each type eventually)eventually i think we'd want to test other features like
tag
local-git
github
@rawkode any interest in this one? it's a little more advanced but probably manageable as a start. there's a bit of "how much can we use ponytest for" (a lot see https://github.com/ponylang/changelog-tool/blob/master/tests/main.pony tests)? and some would have to be some kind of automated integration test(s), so there's a bit of complication but its pretty straightforward. if you are interested, i'd say drop idea here for tests and what the runner would be and @jemc and I could give feedback.
@SeanTAllen Yeah, I'll look at what I can do and come back with a plan :+1:
@rawkode assume. i just sent you an invite to be part of our "contributor" team. it doesn't give you much in the way of privileges but does allow for you to appear in the "Assignee" list and be assigned to issues you are interested in working on. Baby steps towards committer!
So this is pretty tough.
My initial thought is that we need a JSON Schema that will evolve with the tool and provide a means of documenting the format for other integrations; as well a confidence that stable conforms with said schema.
From what I can tell, we could probably use ponytest for arbitary comparison tests, but it wouldn't be pretty and we'd need to provide the json
for comparison. Which means, in order for the test not to be flaky, we need to add branching/tag/commit sha support to stable add
assert_eq("json", loaded_bundle.json)
Hopefully any assumptions I've made are correct :smile:
I'll try pushing this forward. #70 gets some scaffolding in place, leaving open the following test TODOs:
stable fetch
stable add
stable env
: #73
At the moment, the best we can do when checking a PR is to verify that stable builds. We have no automated tests of basic functionality. I'm opening this issue to collect and track thoughts on what tests we should add.