search

razorpay / blade

Design System that powers Razorpay
https://blade.razorpay.com
MIT License
495 stars 129 forks source link

Improving blade coverage methodology #2337

Open parichay28 opened 3 weeks ago

parichay28 commented 3 weeks ago

Currently blade coverage reported is runtime based code ref. The coverage is collected as & when the user triggers newer flows. This poses a challenge as it can introduce outliers (sample route SS below) and affect overall coverage reports.

Internal dashboard link

image

While runtime based approach is good to understand an overall upward / downward trend of coverage. It has following downsides:

  1. The coverage numbers are dictated by user flows triggered.
  2. The current delay is 7s which might be large & any page transitions / user actions in between might be missed. Further the coverage is collected once a week (perhaps this could be allowed to be configured by the consumers).
  3. Currently we rely on navigate event to collect coverage, but CSR apps might not always have a route transition involved. (ex: state based triggering of certain UIs)
  4. At days where a particular route is not visited, the coverage might drop to zero (we can filter this out in our dashboards).
  5. During outages / incidents leading to a downtime, this might affect the overall coverage reports (ex: we recently had to disable Bulk Payouts due to internal issues for ~2 days)
  6. For pages with A/B testing enabled we might not get the complete picture (depending on the split).

We currently rely on this approach to report monthly blade coverage for FY25 Quality Scorecard. To get more consistent coverage reports, static analysis might be more suited (though will have it's own challenges). This can remove dependencies on users / playwright E2Es & get actual coverage data.

Wanted to discuss this with larger forum & understand what the team thinks about this.

AnshulNautiyal commented 3 weeks ago

@saurabhdaware Regarding the points raised by @parichay28, I believe the Blade team might have considered similar points when finalising their approach for code coverage. Do you have any documentation on this? Specifically, I’m interested in understanding how and why the runtime-based approach was chosen.