jekyll / Utterson

CI benchmarking suite for Jekyll
MIT License
13 stars 8 forks source link

Have a record of all checks in a PR #62

Open ashmaroli opened 6 years ago

ashmaroli commented 6 years ago

This is to allow atomic comparison of the impact of the proposed changes of a PR in [WIP] mode.

Scenario: Examine impact of all proposed commits on performance
  Given that I have a WIP pull request with multiple individually pushed commits
  And that those commits encompass meaningful and significant changes
  When I examine the "report page" url for a commit
  Then I should be able to inspect the report page of any "previous commit" from the current page
  And I should be able to inspect the report page of any "following commit" from the current page

The benefit from this is that contributors will have easier insight on the course of their final proposal. For example, if a WIP-PR has 5 individually pushed commits with 3 commits that improve performance and the following 2 that slightly lower performance, the author would have to currently either:


Related requirements:

pathawks commented 6 years ago

You are saying the generated HTML report for every commit should include a link to the parent commit and and child commits?

ashmaroli commented 6 years ago

You are saying the generated HTML report for every commit should include a link to the parent commit and and child commits?

More or less, yes. But mainly just a tab for all checks run in given PR..

pathawks commented 6 years ago

This is currently impossible in the generated HTML. Since each checkrun is atomic, there is no knowledge of past or future commits in the PR. What we could maybe do is have GitHub keep a check run for each commit in the Checks tab, instead of overwriting the last Performance Check when a new one is added.