amazon-ion / ion-java-benchmark-cli

Apache License 2.0
7 stars 9 forks source link

Adds the ability to compare benchmark results from different ion-java commits. #17

Closed linlin-s closed 3 years ago

linlin-s commented 3 years ago

Description of changes: This PR adds the ability of comparing benchmark results from different ion-java commits and adds the ability to generate Ion Struct in a nested format. There's a unit test added for this comparison API.

Motivation:

There's a GitHub actions workflow which will be triggered when a new commit is pushed to the master branch of Ion java to help developers detect the performance regression. One way to detect the performance regression is to compare the benchmark result from the new commit with the one from the existing commit. The comparison process in the workflow will be finished by this API. After comparing process, the workflow will keep working on regression detecting process.

Review guide:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

linlin-s commented 3 years ago

The result from benchmark results comparison process: {input:"testIonData.10n",params:{format:"ION_BINARY",type:"FILE",api:"DOM"},relative_difference_score:{'·gc.alloc.rate':-0.002851051607559,'Heap usage':0.0219662830324952,'Serialized size':0.000000,speed:0.017441311589927}}

Examples of generated random nested Ion Struct: corporate::{firstName:"𪍝া釖ᒰ蒨𝃠鏴𝝍잴🅎⠕쉱곃⩢𦈣ﲯ⾻𥢛羏𠫹",last_updated:3428T,addresses:["𪁵ᦴ阍ﴃ𨷔𩡖𧓴씯쎥G皎讕𢉫煈滑𠶀ܧ𡝭","𢴫𣲣𥘘𧒞殼笅톺騑𒋻𥆡ᢚ𥾗쥨𖧋쥈𫆃𦴘𩅼쐂魧","𦈇𠝠𢛞⩴𧿩䈆阷糊𩊻𐰑籑ﮩ𪕤𧇨挄𧾎刽𦠹𢦂𢴝","𪑍𖠭欳墡𪇁ᥨ𠛺𡋩嵠朣ⴈ给♂飧𢻤𨁐䮓𦄦𦭗𢼵","K竤匎俏𖥺མ嚖𝟙𤡒𢺂𣗁𡲾京𦖃⒩𨰇","㘑𩹆ⷃ譤獘착𣓽𦢱ꋩ鴥椾𤼼ⰺ𫈄𠇆墆𦘥𪡋𢛰䱇","𦩁𢙙𪉤䜤䋑𩍖䖾𑚮𥝧퉪랺𧰲윖𠝯⑃噽𥴳徇𫘿ᴫ"]} corporate::club_member::{firstName:"🜆ൄ㼖䝧𫍪𠀗ϵ瓔𩞈𢖆𝍂뭀巉𦴅𧮴𝙂혖𥱛㳜𠹳",last_updated:1774T,addresses:["𤑝𨚾눃䃶𦲠棥㯰𥡦𩙿嚧𒇙栄欣𣥁𡌽𦛝︕𤸉黌㟜","徹𪇆𦞑𣚋赒𦄚ᴏ㡄ꉺ𡶝㈵⊟𨰗𨷂쐝嫞𢾍乷碌𣸸","𪙢ꃎ黆꾣𥮦뺲𠹓恂𡀉䢕れ𢯖洜𦮰𡑱ꉝ𤅪𤨤从𧧀","ⶮ𠡿𥩋𒆈𒐱ꘂ撇䘸⎩䄝즮蠚𒅫𤧚놶삏ꔾ코繹","暅𝟤巡𤻼햊矟ᕞ⦰篶ᆰ𠤀孥ꨍ༜꺤♬䭯㧻ₛ𠘼"]}