search

mozilla-spidermonkey / matrix-react-bench

Experimental browser performance micro-benchmark using matrix-react-sdk
Other
3 stars 3 forks source link

Benchmark fails to run in Safari #4

Open davehunt opened 1 year ago

davehunt commented 1 year ago

Whilst testing to see which of our performance tests are able to run in Safari on macOS I discovered the following error for this benchmark:

15:04:51 ERROR - raptor-browsertime Error: Browsertime failed to run 15:04:51 INFO - raptor-browsertime Info: [browsertime] Executing script return (function() { 15:04:51 INFO - raptor-browsertime Info: const supported = PerformanceObserver.supportedEntryTypes; 15:04:51 INFO - raptor-browsertime Info: if (!supported || supported.indexOf('layout-shift') === -1) { 15:04:51 INFO - raptor-browsertime Info: return; 15:04:51 INFO - raptor-browsertime Info: } 15:04:51 INFO - raptor-browsertime Info: // See https://web.dev/layout-instability-api 15:04:51 INFO - raptor-browsertime Info: // https://github.com/mmocny/web-vitals/wiki/Snippets-for-LSN-using-PerformanceObserver#max-session-gap1s-limit5s 15:04:51 INFO - raptor-browsertime Info: let max = 0; 15:04:51 INFO - raptor-browsertime Info: let curr = 0; 15:04:51 INFO - raptor-browsertime Info: let firstTs = Number.NEGATIVE_INFINITY; 15:04:51 INFO - raptor-browsertime Info: let prevTs = Number.NEGATIVE_INFINITY; 15:04:51 INFO - raptor-browsertime Info: const observer = new PerformanceObserver(list => {}); 15:04:51 INFO - raptor-browsertime Info: observer.observe({ type: 'layout-shift', buffered: true }); 15:04:51 INFO - raptor-browsertime Info: const list = observer.takeRecords(); 15:04:51 INFO - raptor-browsertime Info: for (let entry of list) { 15:04:51 INFO - raptor-browsertime Info: if (entry.hadRecentInput) { 15:04:51 INFO - raptor-browsertime Info: continue; 15:04:51 INFO - raptor-browsertime Info: } 15:04:51 INFO - raptor-browsertime Info: if (entry.startTime - firstTs > 5000 || entry.startTime - prevTs > 1000) { 15:04:51 INFO - raptor-browsertime Info: firstTs = entry.startTime; 15:04:51 INFO - raptor-browsertime Info: curr = 0; 15:04:51 INFO - raptor-browsertime Info: } 15:04:51 INFO - raptor-browsertime Info: prevTs = entry.startTime; 15:04:51 INFO - raptor-browsertime Info: curr += entry.value; 15:04:51 INFO - raptor-browsertime Info: max = Math.max(max, curr); 15:04:51 INFO - raptor-browsertime Info: } 15:04:51 INFO - raptor-browsertime Info: return max; 15:04:51 INFO - raptor-browsertime Info: })(); 15:04:51 INFO - raptor-browsertime Info: 15:04:51 CRITICAL - raptor-browsertime Critical: [browsertime] Failed to execute user script: TimeoutError 15:04:51 ERROR - Traceback (most recent call last): 15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/raptor.py", line 206, in 15:04:51 INFO - main() 15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/raptor.py", line 152, in main 15:04:51 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names) 15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/perftest.py", line 469, in run_tests 15:04:51 INFO - self.run_test(test, timeout=int(test.get("page_timeout"))) 15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/browsertime/base.py", line 803, in run_test 15:04:51 INFO - raise Exception(self.browsertime_failure) 15:04:51 INFO - Exception: [browsertime] Failed to execute user script: TimeoutError 15:04:51 ERROR - Return code: 1

moztcampbell commented 1 year ago

This error is coming out of the 'web-vitals' package used by 'sentry' used by the 'matrix-react-sdk' we use for the benchmark. I'm not sure why this doesn't cause trouble for matrix chat itself in safari?

92kns commented 1 year ago

last week I found that it consistently timed out on running the cpuBenchmark script

e.g. happens just a bit earlier in the verbose error log 

19:18:45     INFO -  raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:45     INFO -  raptor-browsertime Info:     // simple CPU benchmarking
19:18:45     INFO -  raptor-browsertime Info:     // following https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/blob/master/modules/ext.navigationTiming.js
19:18:45     INFO -  raptor-browsertime Info:     // except not in worker to avoid CSP issues.
19:18:45     INFO -  raptor-browsertime Info:     const amount = 100000000;
19:18:45     INFO -  raptor-browsertime Info:     const startTime = performance.now();
19:18:45     INFO -  raptor-browsertime Info:     for ( let i = amount; i > 0; i-- ) {
19:18:45     INFO -  raptor-browsertime Info:         // empty
19:18:45     INFO -  raptor-browsertime Info:     }
19:18:45     INFO -  raptor-browsertime Info:     return Math.round( performance.now() - startTime );
19:18:45     INFO -  raptor-browsertime Info:  })();
19:18:45     INFO -  raptor-browsertime Info: [browsertime] Couldn't execute script named cpuBenchmark error:TimeoutError
19:18:45    ERROR -  raptor-browsertime Error: Browsertime failed to run
19:18:45     INFO -  raptor-browsertime Info: [browsertime] Failed to execute user script: TimeoutError

and then what follows immediately after is the same message in the first comment from Dave

19:18:45    ERROR -  raptor-browsertime Error: Browsertime failed to run
19:18:46     INFO -  raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:46     INFO -  raptor-browsertime Info:   const supported = PerformanceObserver.supportedEntryTypes;
19:18:46     INFO -  raptor-browsertime Info:   if (!supported || supported.indexOf('layout-shift') === -1) {
19:18:46     INFO -  raptor-browsertime Info:     return;
19:18:46     INFO -  raptor-browsertime Info:   }
19:18:46     INFO -  raptor-browsertime Info:   // See https://web.dev/layout-instability-api
19:18:46     INFO -  raptor-browsertime Info:   // https://github.com/mmocny/web-vitals/wiki/Snippets-for-LSN-using-PerformanceObserver#max-session-gap1s-limit5s
19:18:46     INFO -  raptor-browsertime Info:   let max = 0;
19:18:46     INFO -  raptor-browsertime Info:   let curr = 0;
19:18:46     INFO -  raptor-browsertime Info:   let firstTs = Number.NEGATIVE_INFINITY;
19:18:46     INFO -  raptor-browsertime Info:   let  prevTs = Number.NEGATIVE_INFINITY;
19:18:46     INFO -  raptor-browsertime Info:   const observer = new PerformanceObserver(list => {});
19:18:46     INFO -  raptor-browsertime Info:   observer.observe({ type: 'layout-shift', buffered: true });
19:18:46     INFO -  raptor-browsertime Info:   const list = observer.takeRecords();
19:18:46     INFO -  raptor-browsertime Info:   for (let entry of list) {
19:18:46     INFO -  raptor-browsertime Info:     if (entry.hadRecentInput) {
19:18:46     INFO -  raptor-browsertime Info:       continue;
19:18:46     INFO -  raptor-browsertime Info:     }
19:18:46     INFO -  raptor-browsertime Info:     if (entry.startTime - firstTs > 5000 || entry.startTime - prevTs > 1000) {
19:18:46     INFO -  raptor-browsertime Info:       firstTs = entry.startTime;
19:18:46     INFO -  raptor-browsertime Info:       curr = 0;
19:18:46     INFO -  raptor-browsertime Info:     }
19:18:46     INFO -  raptor-browsertime Info:     prevTs = entry.startTime;
19:18:46     INFO -  raptor-browsertime Info:     curr += entry.value;
19:18:46     INFO -  raptor-browsertime Info:     max = Math.max(max, curr);
19:18:46     INFO -  raptor-browsertime Info:   }
19:18:46     INFO -  raptor-browsertime Info:   return max;
19:18:46     INFO -  raptor-browsertime Info: })();
19:18:46     INFO -  raptor-browsertime Info:
19:18:46 CRITICAL -  raptor-browsertime Critical: [browsertime] Failed to execute user script: TimeoutError

so I tried both: 1) removing cpuBenchmark from the commands during run time 2) keeping it, but lowering the amount variable

And in both cases it seemed to be passing locally, but not on CI unfortunately (it seems to hang indefinitely waiting for data). Not entirely sure why even cpuBenchmark is failing on matrix-react-bench but not other benchmarks running on Safari. Also the cpuBenchmark error might likely be misleading altogether.

anyhow, @moztcampbell, would you know where I could find more information regarding the sentry/web-vitals/mrb-sdk interactions to look into this more?