Closed migurski closed 8 years ago
Updated the sample JSON per @wpietri’s suggestion.
This looks good to me. I think the OUTPUT_FILE code more properly belongs in the superclass in that we want it to happen for all acceptance tests rather than just this one, but since we currently have only one, it's a distinction without a difference. However, I'd rather not merge the Travis bit until the nightly job is up, running, and producing a report that people will actually see on a daily basis.
Latest version looks fine to me too. There are a few things that could be changed to make the file more human-readable and -useful if that's desired – like readable dates and links to browserstack jobs (though from what I hear that's not really posssible).
Also, will we have a hook into Slack so we'll see when these tests have been run?
Oh, and saving archived test results maybe?
None of my suggestions are intended to delay merging this PR though.
I’m putting the finishing touches on the nightly job now!
It looks like if the tests are run n times per os/browser, we will see every test represented in the JSON. Might be cool to have one entry per os/browser combination, a pass/fail ratio, and a list of exceptions.
Ah, I was thinking that the summarization would be taken care of by whatever script was doing the user-visible reporting. E.g., that there would be an HTML page that would load this stuff with JS. Or that some script would grab the data after a run and send us email.
I'm fine with that. I wasn't sure whether this was supposed to be user-visible or consumed by another service (like your dashboard?).
I believe this is read to merge.
git pull
, with 25 test reps and no retries.
Acceptance tests now upload data to S3 in a publicly-accessible JSON file.
Sample data: