search

callstack / reassure

Performance testing companion for React and React Native
https://callstack.github.io/reassure/
MIT License
1.26k stars 28 forks source link

feat: pass test runner options #511

Closed mdjastrzebski closed 1 month ago

mdjastrzebski commented 3 months ago

Summary

Resolves #294 Resolves #510

By providing easy option to forward flags to test runner (Jest by default): yarn reassure -- --coverage=false.

Test plan

Manual testing.

changeset-bot[bot] commented 3 months ago

🦋 Changeset detected

Latest commit: bc0385f9c27348e5d429b869f7540369a8b1308e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages | Name | Type | | --------------------------- | ----- | | @callstack/reassure-cli | Minor | | reassure | Minor | | @callstack/reassure-compare | Minor | | @callstack/reassure-danger | Minor | | @callstack/reassure-logger | Minor | | @callstack/reassure-measure | Minor |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions[bot] commented 3 months ago

Performance Comparison Report

Significant Changes To Duration

There are no entries

Meaningless Changes To Duration

Show entries | Name | Type | Duration | Count | | ---------------------------------- | -------- | ------------------------------------- | ------ | | Async Component | render | 102.9 ms → 100.2 ms (-2.7 ms, -2.6%) | 7 → 7 | | fib 30 | function | 38.7 ms → 38.9 ms (+0.1 ms, ±0.0%) | 1 → 1 | | fib 31 | function | 63.1 ms → 61.6 ms (-1.5 ms, -2.3%) | 1 → 1 | | fib 32 | function | 101.0 ms → 101.0 ms (-0.0 ms, ±0.0%) | 1 → 1 | | InitialRenders 1 | render | 1.1 ms → 0.9 ms (-0.2 ms, -18.2%) 🟢 | 2 → 2 | | InitialRenders 3 | render | 1.4 ms → 1.4 ms | 4 → 4 | | ManyRenderIssues | render | 1.8 ms → 2.1 ms (+0.3 ms, +16.7%) 🔴 | 5 → 5 | | Other Component 10 | render | 65.6 ms → 64.8 ms (-0.8 ms, -1.2%) | 4 → 4 | | Other Component 10 legacy scenario | render | 62.3 ms → 63.0 ms (+0.7 ms, +1.1%) | 4 → 4 | | Other Component 20 | render | 61.9 ms → 62.5 ms (+0.6 ms, +1.1%) | 4 → 4 | | RedundantUpdates | render | 0.6 ms → 0.7 ms (+0.1 ms, +16.7%) 🔴 | 2 → 2 |
Show details | Name | Type | Duration | Count | | ---------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Async Component | render | **Baseline**
Mean: 102.9 ms
Stdev: 8.9 ms (8.6%)
Runs: 101 110 101 101 112 101 109 112 82 100
Warmup runs: 109

**Current**
Mean: 100.2 ms
Stdev: 3.4 ms (3.4%)
Runs: 106 104 100 95 98 100 101 100 102 96
Warmup runs: 104 | **Baseline**
Mean: 7
Stdev: 0 (0.0%)
Runs: 7 7 7 7 7 7 7 7 7 7
Render issues:
- Initial updates: 1

**Current**
Mean: 7
Stdev: 0 (0.0%)
Runs: 7 7 7 7 7 7 7 7 7 7
Render issues:
- Initial updates: 1 | | fib 30 | function | **Baseline**
Mean: 38.7 ms
Stdev: 0.4 ms (1.0%)
Runs: 38.8 38.8 38.8 38.8 38.9 39.0 38.9 37.7 38.9 38.9
Warmup runs: 39.6

**Current**
Mean: 38.9 ms
Stdev: 0.1 ms (0.2%)
Runs: 38.9 39.0 38.9 38.9 38.9 39.0 38.8 38.7 38.8 38.8
Warmup runs: 39.3 | **Baseline**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1

**Current**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1 | | fib 31 | function | **Baseline**
Mean: 63.1 ms
Stdev: 1.6 ms (2.6%)
Runs: 62.8 62.7 67.2 60.4 62.9 63.1 63.0 63.0 63.0 62.9
Warmup runs: 62.3

**Current**
Mean: 61.6 ms
Stdev: 1.7 ms (2.8%)
Runs: 62.2 62.2 62.1 62.1 62.8 62.8 62.8 58.6 58.2 62.8
Warmup runs: 62.2 | **Baseline**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1

**Current**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1 | | fib 32 | function | **Baseline**
Mean: 101.0 ms
Stdev: 1.6 ms (1.6%)
Runs: 100.8 100.8 101.9 101.8 100.9 96.6 101.9 102.0 101.9 101.9
Warmup runs: 100.7

**Current**
Mean: 101.0 ms
Stdev: 1.3 ms (1.3%)
Runs: 100.6 100.4 97.7 101.4 101.8 101.9 101.9 100.7 102.0 101.9
Warmup runs: 100.7 | **Baseline**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1

**Current**
Mean: 1
Stdev: 0 (0.0%)
Runs: 1 1 1 1 1 1 1 1 1 1 | | InitialRenders 1 | render | **Baseline**
Mean: 1.1 ms
Stdev: 0.6 ms (51.6%)
Runs: 1 2 0 1 1 1 2 1 1 1
Warmup runs: 2

**Current**
Mean: 0.9 ms
Stdev: 0.3 ms (35.1%)
Runs: 1 1 1 1 0 1 1 1 1 1
Warmup runs: 2 | **Baseline**
Mean: 2
Stdev: 0 (0.0%)
Runs: 2 2 2 2 2 2 2 2 2 2
Render issues:
- Initial updates: 1

**Current**
Mean: 2
Stdev: 0 (0.0%)
Runs: 2 2 2 2 2 2 2 2 2 2
Render issues:
- Initial updates: 1 | | InitialRenders 3 | render | **Baseline**
Mean: 1.4 ms
Stdev: 0.7 ms (49.9%)
Runs: 2 0 2 1 1 2 1 2 2 1
Warmup runs: 1

**Current**
Mean: 1.4 ms
Stdev: 0.7 ms (49.9%)
Runs: 0 1 1 1 2 2 2 2 1 2
Warmup runs: 2 | **Baseline**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 3

**Current**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 3 | | ManyRenderIssues | render | **Baseline**
Mean: 1.8 ms
Stdev: 0.6 ms (35.1%)
Runs: 2 2 3 2 2 2 1 1 1 2
Warmup runs: 1

**Current**
Mean: 2.1 ms
Stdev: 0.6 ms (27.0%)
Runs: 3 2 2 3 2 2 2 2 2 1
Warmup runs: 2 | **Baseline**
Mean: 5
Stdev: 0 (0.0%)
Runs: 5 5 5 5 5 5 5 5 5 5
Render issues:
- Initial updates: 2
- Redundant updates: 2 (3, 4)

**Current**
Mean: 5
Stdev: 0 (0.0%)
Runs: 5 5 5 5 5 5 5 5 5 5
Render issues:
- Initial updates: 2
- Redundant updates: 2 (3, 4) | | Other Component 10 | render | **Baseline**
Mean: 65.6 ms
Stdev: 5.0 ms (7.6%)
Runs: 75 71 65 70 60 64 65 61 60 65
Warmup runs: 263

**Current**
Mean: 64.8 ms
Stdev: 3.6 ms (5.6%)
Runs: 64 62 65 66 66 69 70 67 60 59
Warmup runs: 70 | **Baseline**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1

**Current**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1 | | Other Component 10 legacy scenario | render | **Baseline**
Mean: 62.3 ms
Stdev: 3.8 ms (6.1%)
Runs: 61 61 64 61 65 63 59 57 61 71
Warmup runs: 63

**Current**
Mean: 63.0 ms
Stdev: 3.4 ms (5.4%)
Runs: 60 61 67 65 57 61 62 67 63 67
Warmup runs: 66 | **Baseline**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1

**Current**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1 | | Other Component 20 | render | **Baseline**
Mean: 61.9 ms
Stdev: 5.9 ms (9.5%)
Runs: 59 57 65 58 57 67 54 61 61 70 60 61 65 58 79 53 62 60 65 65
Warmup runs: 61

**Current**
Mean: 62.5 ms
Stdev: 5.0 ms (7.9%)
Runs: 57 55 63 63 60 68 62 66 55 69 59 56 61 67 74 66 64 64 61 60
Warmup runs: 64 | **Baseline**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1

**Current**
Mean: 4
Stdev: 0 (0.0%)
Runs: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
Render issues:
- Initial updates: 1 | | RedundantUpdates | render | **Baseline**
Mean: 0.6 ms
Stdev: 0.8 ms (140.5%)
Runs: 0 2 1 0 0 0 2 0 1 0
Warmup runs: 1

**Current**
Mean: 0.7 ms
Stdev: 0.8 ms (117.6%)
Runs: 0 2 0 1 1 0 0 1 0 2
Warmup runs: 1 | **Baseline**
Mean: 2
Stdev: 0 (0.0%)
Runs: 2 2 2 2 2 2 2 2 2 2
Render issues:
- Redundant updates: 1 (1)

**Current**
Mean: 2
Stdev: 0 (0.0%)
Runs: 2 2 2 2 2 2 2 2 2 2
Render issues:
- Redundant updates: 1 (1) |

Render Count Changes

There are no entries

Render Issues

Name Initial Updates Redundant Updates
Other Component 10 1 🔴 -
Other Component 10 legacy scenario 1 🔴 -
Other Component 20 1 🔴 -
InitialRenders 1 1 🔴 -
InitialRenders 3 3 🔴 -
RedundantUpdates - 1 (1) 🔴
ManyRenderIssues 2 🔴 2 (3, 4) 🔴
Async Component 1 🔴 -

Added Scenarios

There are no entries

Removed Scenarios

There are no entries

Generated by :no_entry_sign: dangerJS against bc0385f9c27348e5d429b869f7540369a8b1308e

mdjastrzebski commented 1 month ago

This PR has been released in Reassure v1.2.0 🎉