Ci performance - Githubissues

tum-ei-eda / etiss

Extendable Translating Instruction Set Simulator

https://tum-ei-eda.github.io/etiss/

Other

29 stars 36 forks source link

Ci performance #77

Closed Samanti-Das closed 2 years ago

Samanti-Das commented 3 years ago

This analyzes the performance regarding the MIPS value for the three JIT engines TCC, GCC and LLVM.

The entire MIPS performance history, along with the respective github commit hash is stored in a json file _statsfile.json in a new branch _benchmarkresults.

For the current workflow run, the performance of the JIT engines are displayed as a comment under an Issue.

For the past workflow runs, the history is showed graphically (_MIPS_value vs commithash) in the github Wiki under the folder _PerformanceMetrics.

wysiwyng commented 3 years ago

i assume the failing ci is because there is nothing to compare to. this should be fixed on any commit after this has been merged?

Correct, as this comparison relies on data tracked in the git repo on a separate branch - which does not exist yet - it does not find any data to compare against.

rafzi commented 2 years ago

it seems this logic is still missing a piece. https://github.com/tum-ei-eda/etiss/runs/3983628320?check_suite_focus=true

how about:

try to get results and compare. if there are no previous results, remember failed status, but return success in this ci step
store results regardless of failed status
if failed before, return failure in ci

wysiwyng commented 2 years ago

this works now, the code for this PR was missing its data storage branch. we should discuss about this data branch in #83, maybe we can move this data storage e.g. to the wiki repository to not clutter the main repo.

For now, performance will be reported to the wiki: https://github.com/tum-ei-eda/etiss/wiki/Performance_Metrics. i have disabled reporting to a permanent issue, if we want such functionality it can be enabled again. new pull requests should also get some performance metrics commented on them.