Open davehunt opened 1 year ago
This error is coming out of the 'web-vitals' package used by 'sentry' used by the 'matrix-react-sdk' we use for the benchmark. I'm not sure why this doesn't cause trouble for matrix chat itself in safari?
last week I found that it consistently timed out on running the cpuBenchmark script
e.g. happens just a bit earlier in the verbose error log
19:18:45 INFO - raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:45 INFO - raptor-browsertime Info: // simple CPU benchmarking
19:18:45 INFO - raptor-browsertime Info: // following https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/blob/master/modules/ext.navigationTiming.js
19:18:45 INFO - raptor-browsertime Info: // except not in worker to avoid CSP issues.
19:18:45 INFO - raptor-browsertime Info: const amount = 100000000;
19:18:45 INFO - raptor-browsertime Info: const startTime = performance.now();
19:18:45 INFO - raptor-browsertime Info: for ( let i = amount; i > 0; i-- ) {
19:18:45 INFO - raptor-browsertime Info: // empty
19:18:45 INFO - raptor-browsertime Info: }
19:18:45 INFO - raptor-browsertime Info: return Math.round( performance.now() - startTime );
19:18:45 INFO - raptor-browsertime Info: })();
19:18:45 INFO - raptor-browsertime Info: [browsertime] Couldn't execute script named cpuBenchmark error:TimeoutError
19:18:45 ERROR - raptor-browsertime Error: Browsertime failed to run
19:18:45 INFO - raptor-browsertime Info: [browsertime] Failed to execute user script: TimeoutError
and then what follows immediately after is the same message in the first comment from Dave
19:18:45 ERROR - raptor-browsertime Error: Browsertime failed to run
19:18:46 INFO - raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:46 INFO - raptor-browsertime Info: const supported = PerformanceObserver.supportedEntryTypes;
19:18:46 INFO - raptor-browsertime Info: if (!supported || supported.indexOf('layout-shift') === -1) {
19:18:46 INFO - raptor-browsertime Info: return;
19:18:46 INFO - raptor-browsertime Info: }
19:18:46 INFO - raptor-browsertime Info: // See https://web.dev/layout-instability-api
19:18:46 INFO - raptor-browsertime Info: // https://github.com/mmocny/web-vitals/wiki/Snippets-for-LSN-using-PerformanceObserver#max-session-gap1s-limit5s
19:18:46 INFO - raptor-browsertime Info: let max = 0;
19:18:46 INFO - raptor-browsertime Info: let curr = 0;
19:18:46 INFO - raptor-browsertime Info: let firstTs = Number.NEGATIVE_INFINITY;
19:18:46 INFO - raptor-browsertime Info: let prevTs = Number.NEGATIVE_INFINITY;
19:18:46 INFO - raptor-browsertime Info: const observer = new PerformanceObserver(list => {});
19:18:46 INFO - raptor-browsertime Info: observer.observe({ type: 'layout-shift', buffered: true });
19:18:46 INFO - raptor-browsertime Info: const list = observer.takeRecords();
19:18:46 INFO - raptor-browsertime Info: for (let entry of list) {
19:18:46 INFO - raptor-browsertime Info: if (entry.hadRecentInput) {
19:18:46 INFO - raptor-browsertime Info: continue;
19:18:46 INFO - raptor-browsertime Info: }
19:18:46 INFO - raptor-browsertime Info: if (entry.startTime - firstTs > 5000 || entry.startTime - prevTs > 1000) {
19:18:46 INFO - raptor-browsertime Info: firstTs = entry.startTime;
19:18:46 INFO - raptor-browsertime Info: curr = 0;
19:18:46 INFO - raptor-browsertime Info: }
19:18:46 INFO - raptor-browsertime Info: prevTs = entry.startTime;
19:18:46 INFO - raptor-browsertime Info: curr += entry.value;
19:18:46 INFO - raptor-browsertime Info: max = Math.max(max, curr);
19:18:46 INFO - raptor-browsertime Info: }
19:18:46 INFO - raptor-browsertime Info: return max;
19:18:46 INFO - raptor-browsertime Info: })();
19:18:46 INFO - raptor-browsertime Info:
19:18:46 CRITICAL - raptor-browsertime Critical: [browsertime] Failed to execute user script: TimeoutError
so I tried both: 1) removing cpuBenchmark from the commands during run time 2) keeping it, but lowering the amount variable
And in both cases it seemed to be passing locally, but not on CI unfortunately (it seems to hang indefinitely waiting for data). Not entirely sure why even cpuBenchmark is failing on matrix-react-bench but not other benchmarks running on Safari. Also the cpuBenchmark error might likely be misleading altogether.
anyhow, @moztcampbell, would you know where I could find more information regarding the sentry/web-vitals/mrb-sdk interactions to look into this more?
Whilst testing to see which of our performance tests are able to run in Safari on macOS I discovered the following error for this benchmark: