Speedometer 2.1 score dropped significantly since v1.61

[valynor]

1.61.101 Chromium: 120.0.6099.71 (Official Build) (arm64) on Mac Studio M1 Max, macOS 14.1.2

Since updating to Brave 1.61 the Speedometer 2.1 score has dropped significantly. In a guest window I got 420-430 before and now it is down to 350-360 points across multiple tests. Interestingly the deprecated Octane Benchmark shows better scores than ever before around 80700.

Safari 17.2 still gets ~405 on https://browserbench.org/Speedometer2.1/

Please investigate, ~80 points less is quite a huge drop.

[rebron]

cc: @atuchin-m

[atuchin-m]

@valynor thanks for the report.

1. We have a CI platform to track the performance, it's included speedometer2 on m1 mac. According to it the results, Brave on CR120 is event better than on CR119.

   

2. A local MBP M1 pro mac gives me: ~400 on Brave 1.60.125 ~422 on Brave 1.61.101 ~419 on Chrome 119.0.6045.199 ~441 on Chrome 120.0.6099.71

So I can't confirm the issue neither for my MBP or for CI bots (mac mini-m1). I admit than the result could degrade to some specific configuration, but we need more data to continue the investigation.

cc @iefremov

[atuchin-m]

I suggest to check it againt:

• reboot the machine, stop the other process, check no background activity in Activity Monitor
• check the stabilty of the internet connection(yes, it can affect the results)
• take a fresh clean temporary browser profile (no extensions, no cache, etc): "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1
• recheck the old release https://github.com/brave/brave-browser/releases/v1.60.125 (please don't use it without --user-data-dir, because the old versions could affect your data).
• check Chrome or Chromium 120, i.e. https://googlechromelabs.github.io/chrome-for-testing/

If you confirm the issue, we will try to find folks with a similar machine to get a reproduction.

[atuchin-m]

@valynor Another thing to check: does it work well for a usual session (not in a guest window)?

[stephendonner]

Here are my results on

Brave | 1.61.100 Chromium: 120.0.6099.62 (Official Build) (arm64)
-- | --
Revision | 63d361b4e5fd163e032192aa8e033cf85f846575
OS | macOS Version 13.6.1 (Build 22G313)

395	example	404	example	401	example

And on

Brave | 1.61.101 Chromium: 120.0.6099.71 (Official Build) (arm64)
-- | --
Revision | a2125180c183a2b5775b45c2138fda37d240b773
OS | macOS Version 13.6.1 (Build 22G313)

example	example	example	example	example	example

https://browserbench.org/Speedometer2.1/?iterationCount=100

example	example

[valynor]

@atuchin-m Sorry for the late reply, I was trying to get to the bottom of this issue all day long but to no avail:

• reboot the machine, stop the other process, check no background activity in Activity Monitor

Rebooted several times, no other apps running

• check the stabilty of the internet connection(yes, it can affect the results)

I know my InterNet connection is fine but I checked anyway:

This is exactly the maximum my connection can deliver.

• take a fresh clean temporary browser profile (no extensions, no cache, etc): "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1

When I do that I instantly get 426-446 Speedometer 2.1 scores - every time. Going back to my default profile (with extensions, I know uBlock Origin slows down this benchmark a bit) I get ~320-330 and in a guest window it's ~350.

Some other things I tried:

• creating a new user profile: no change
• moving the default profile out of the Brave directory and letting Brave create a fresh one: still ~350 score
• restoring the settings to default via brave://settings/reset. no change either

At this point I decided to remove Brave completely via App Cleaner, rebooted and did a fresh install. And this is where it gets really spooky, because I have no idea what's going on or how this is even possible:

I launched the freshly installed Brave, went straight to a guest window and: ­~350 score.

• Went back to the default profile window: ~350
• Restarted Brave and tested directly in the fresh default window: 430!
• Switching to a guest window: 350
• Switching back to default: always 350ish results until I restart the FRESH browser and then I get ~430 scores again. I have no idea what is going on.

So I synced back my usual profile via sync chain ... still 350 score but I decided to test some more, this time in the default profile:

• I ran Speedometer 2.1 back to back about 5-10 times without reloading the tab (just clicking Test Again immediately) and noticed each score gets progressively lower, from 330 down to ~220.
• On the later runs you can even see the benchmark struggling and stopping for a few seconds. If I do 1-2 more runs after this the tab crashes (Oh snap!).
• this is repeatable

I can guess what you are thinking now ... it's the hardware.

• CPU temp under load stays <46°C on all cores.

• RAM usage is <30%.

• There is no network activity, the whole test runs from memory at this point.

• Rebooted into Apple Diagnostics: No Errors.

• Rebooted the Mac and did 10 iterations of Speedometer 2.1 in Safari 17.2: All scores 390-410, no slowing down, no crashes.

• And just now I did another "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1 and all scores were 426-446, no slowing down, no crashes over 15 tests.

• going back to Brave default profile after this: consecutive runs are slowing down

IMHO the tests on Safari and the Brave tmp profile strongly suggest that the issue is not my hardware but that somehow Brave is responsible for this. Any suggestions how to proceed from here would be very welcome. I had no issues with Brave at all before the v1.61 update, the browser didn't even crash once in well over a year.

I'd also like to point out my other recent bug report #34783 which might or might not be related.

Edit: Did a couple more consecutive Speedometer runs in Safari and this is perfectly reproducible: Safari score remains stable around 400, Brave default profile score quickly goes down and crashes the tab after ~­8 runs. Here's the crash report from the last tab crash that I provoked this way:

Crash from Saturday, 9 December 2023 at 00:46:52 Status:	Uploaded
Uploaded crash report ID:	91b20500-8bde-260b-0000-000000000000
Upload time:	Saturday, 9 December 2023 at 00:46:54
Local crash context:	7496d1ac-f966-496d-a5f7-2cc3d3995952

[atuchin-m]

Thanks for the details report @valynor.

At this point I decided to remove Brave completely via App Cleaner, rebooted and did a fresh install.

Was at a fresh install with preserving your data (= the default user-data-dir)?

creating a new user profile: no change

do you using Add New profile via menu in the default user-data-dir?

If I do 1-2 more runs after this the tab crashes

thanks for the crash report. I've checked https://github.com/brave/brave-browser/issues/34783, it can't affect speedometer results. Reported to the team. As for 91b20500-8bde-260b-0000-000000000000 it's a renderer out-of-memory crash. The crash dump also shows that you have some extensions installed. Could you please double check you have nothing at brave://extensions ?

It's definitely not the hardware, because you have the stable good results on a clean profile. I suspect it's related to the browser settings (they affect to all the profiles in a user-data-dir). In theory, it also could be some an OS level feature or an application.

I suggest the following to found what affect the results:

1. check the number of javascript context on speedometer page (see the screenshot)
2. check brave://flags/, make sure anything in the default state.
3. disabled any extra lists on brave://settings/shields/filters
4. try to disable any non default flags in brave://settings. You can compare brave://settings on/tmp/clean_tmp_profile_1 with your default installation.
5. share brave://version/?show-variations-cmd

Also check 'Details' tabs when you look at speedometer2 results. In case of network level delays the first result of the row could be extremely low, that could major affect the final score.

You could also use https://browserbench.org/Speedometer2.1/?iterationCount=100 to get 10x more iterations (so you do need to repeat the test multiple times). It also reduces the first result "lag".

[valynor]

@atuchin-m

Was at a fresh install with preserving your data (= the default user-data-dir)?

No, I used App Cleaner ( https://freemacsoft.net/appcleaner/ ) which gets rid of almost everything from a macOS install but I went to ~/Library/Application Support/BraveSoftware and there were some files left so I deleted the entire directory manually.

I also did a Finder search in ~/Library/ for "Brave" after that and found nothing so I'm confident to say it was a completely fresh install with no data from the previous install left.

"Brave-Browser.dmg" was a fresh download from https://brave.com/download/, too.

do you using Add New profile via menu in the default user-data-dir?

That was just one of the things I tried yesterday and it did not help to get the ~430 Speedometer results.

thanks for the crash report. I've checked #34783, it can't affect speedometer results. Reported to the team. As for 91b20500-8bde-260b-0000-000000000000 it's a renderer out-of-memory crash. The crash dump also shows that you have some extensions installed. Could you please double check you have nothing at brave://extensions ?

That crash happened while using my usual default profile, i.e. with extensions:

Of these only uBO got updated recently.

These are my Brave Shields settings:

I switched off "Trackers and Ads blocking" in Brave Shields to not conflict with uBlock Origin.

3. disabled any extra lists on brave://settings/shields/filters

Confirmed, there are no extra lists enabled (I removed them earlier because I don't use the Brave blocking anyway).

You could also use https://browserbench.org/Speedometer2.1/?iterationCount=100 to get 10x more iterations (so you do need to repeat the test multiple times). It also reduces the first result "lag".

I just did that in my standard user profile (with extensions) and it crashed the tab at 1635/4800 iterations. Then I added "browserbench.org" to the trusted sites in uBlock Origin 1.54 (effectively turning off uBO for this domain) and it ran to 4800/4800 without issue (score 343). I then repeated this in a guest window and again 4800/4800, score 347.

So it seems that uBlock Origin is causing the tab crashes and probably the slowdowns too when doing long Speedometer runs.

But I'm still not getting the expected ~430 score in a guest window, only when using "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1

It's definitely not the hardware, because you have the stable good results on a clean profile. I suspect it's related to the browser settings (they affect to all the profiles in a user-data-dir). In theory, it also could be some an OS level feature or an application.

Just for reference, I only have these two 3rd party apps running in the background, everything else is stock Sonoma:

• https://github.com/exelban/stats an open source system monitor
• https://github.com/p0deje/Maccy an open source clipboard manager

1. check the number of javascript context on speedometer page (see the screenshot)

2. check brave://flags/, make sure anything in the default state.

I have not changed anything in flags and it's all on default.

5. share brave://version/?show-variations-cmd

` Brave Software Inc Copyright © 2023 The Brave Authors. All rights reserved. Brave	1.61.101 Chromium: 120.0.6099.71 (Official Build) (arm64)
Revision	a2125180c183a2b5775b45c2138fda37d240b773
OS	macOS Version 14.1.2 (Build 23B92)
JavaScript	V8 12.0.267.8
User agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Command Line	/Applications/Brave Browser.app/Contents/MacOS/Brave Browser --disable-domain-reliability --enable-dom-distiller --enable-distillability-service --origin-trial-public-key=bYUKPJoPnCxeNvu72j4EmPuK7tr1PAC7SHh8ld9Mw3E=,fMS4mpO6buLQ/QMd+zJmxzty/VQ6B1EUZqoCU04zoRU= --sync-url=https://sync-v2.brave.com/v2 --lso-url=https://no-thanks.invalid --variations-server-url=https://variations.brave.com/seed --variations-insecure-server-url=https://variations.brave.com/seed --flag-switches-begin --flag-switches-end --component-updater=url-source=https://go-updater.brave.com/extensions
Executable Path	/Applications/Brave Browser.app/Contents/MacOS/Brave Browser
Profile Path	/Users/valynor/Library/Application Support/BraveSoftware/Brave-Browser/Default
Linker	lld
Active variations	AllowCertainClientHintsStudy:EnabledBraveAIChatEnabledStudy:EnabledBraveAdblockMobileNotificationsListDefault:EnabledBraveAds.AdServingStudy:DefaultAdNotificationsPerHour=10/MaximumAdNotificationsPerDay=100/MaximumInlineContentAdsPerHour=6/MaximumInlineContentAdsPerDay=20/AdServingVersion=2BraveAutoTranslateStudy:AutoTranslateDisabledBraveCleanupSessionCookiesOnSessionRestore:DisabledBraveDebounceStudy:EnabledBraveForgetFirstPartyStorage:EnabledBraveGoogleSignInPermissionStudy:DisabledBraveHttpsByDefaultRolloutStudy:EnabledBraveP3AConstellationReleaseRollout:EnabledBraveRewardsWebUiPanelStudy:EnabledBraveScreenFingerprintingBlockerStudy:EnabledBraveSearchAdStudy:EnabledInProcessDesktopClampPlatformVersionClientHint:DisabledCookieListDefaultStudy:EnabledCrossPlatformVPNStudy:EnabledDefault1pBlockingStudy:DisabledDisableReduceLanguage:EnabledEphemeralStorageStudy:EnabledFirstPartyEphemeralDomainBlockStudy:EnabledForceDisableBlinkFeatures:EnabledMacCoreLocationBackendStudy:EnabledPartitionBlinkMemoryCacheStudy:EnabledPartitionConnectionsByNetworkIsolationKeyStudy:EnabledPermissionLifetimeReleaseStudy:EnabledSmilAutoSuspendOnLagKillSwitch:Disabled_EmergencyKillSwitchUserActivityStudy:TriggersWhatsNewStudy:Enabled
Command-line variations	--force-fieldtrials="AllowCertainClientHintsStudy/Enabled/BraveAIChatEnabledStudy/Enabled/BraveAdblockMobileNotificationsListDefault/Enabled/BraveAds.AdServingStudy/DefaultAdNotificationsPerHour=10/MaximumAdNotificationsPerDay=100/MaximumInlineContentAdsPerHour=6/MaximumInlineContentAdsPerDay=20/AdServingVersion=2/BraveAutoTranslateStudy/AutoTranslateDisabled/BraveCleanupSessionCookiesOnSessionRestore/Disabled/BraveDebounceStudy/Enabled/BraveForgetFirstPartyStorage/Enabled/BraveGoogleSignInPermissionStudy/Disabled/BraveHttpsByDefaultRolloutStudy/Enabled/BraveP3AConstellationReleaseRollout/Enabled/BraveRewardsWebUiPanelStudy/Enabled/BraveScreenFingerprintingBlockerStudy/Enabled/BraveSearchAdStudy/EnabledInProcessDesktop/ClampPlatformVersionClientHint/Disabled/CookieListDefaultStudy/Enabled/CrossPlatformVPNStudy/Enabled/Default1pBlockingStudy/Disabled/DisableReduceLanguage/Enabled/EphemeralStorageStudy/Enabled/FirstPartyEphemeralDomainBlockStudy/Enabled/ForceDisableBlinkFeatures/Enabled/MacCoreLocationBackendStudy/Enabled/PartitionBlinkMemoryCacheStudy/Enabled/PartitionConnectionsByNetworkIsolationKeyStudy/Enabled/PermissionLifetimeReleaseStudy/Enabled/SmilAutoSuspendOnLagKillSwitch/Disabled_EmergencyKillSwitch/UserActivityStudy/Triggers/*WhatsNewStudy/Enabled/" --force-fieldtrial-params="BraveAds%2EAdServingStudy.DefaultAdNotificationsPerHour%3D10%2FMaximumAdNotificationsPerDay%3D100%2FMaximumInlineContentAdsPerHour%3D6%2FMaximumInlineContentAdsPerDay%3D20%2FAdServingVersion%3D2:ad_serving_version/2/default_ad_notifications_per_hour/10/maximum_ad_notifications_per_day/100/maximum_inline_content_ads_per_day/20/maximum_inline_content_ads_per_hour/6,UserActivityStudy.Triggers:idle_time_threshold/5s/threshold/0%2E5/time_window/1h/triggers/0D0B14110D0B14110D0B14110D0B1411%3D-1%2E0%3B0D0B1411070707%3D-1%2E0%3B07070707%3D-1%2E0%3B0B1512%3D0%2E5%3B0B1513%3D0%2E5%3B0B1514%3D0%2E5%3B06%3D1%2E0%3B04%3D0%2E5,WhatsNewStudy.Enabled:target_major_version_stable/1%2E60" --enable-features="AIChat<BraveAIChatEnabledStudy,AdServing<BraveAds.AdServingStudy,AllowCertainClientHints<AllowCertainClientHintsStudy,BraveAdblockCookieListDefault<CookieListDefaultStudy,BraveAdblockMobileNotificationsListDefault<BraveAdblockMobileNotificationsListDefault,BraveBlockScreenFingerprinting<BraveScreenFingerprintingBlockerStudy,BraveDebounce<BraveDebounceStudy,BraveDomainBlock1PES<FirstPartyEphemeralDomainBlockStudy,BraveFirstPartyEphemeralStorage<FirstPartyEphemeralDomainBlockStudy,BraveForgetFirstPartyStorage<BraveForgetFirstPartyStorage,BraveP3AConstellation<BraveP3AConstellationReleaseRollout,BraveReduceLanguage<DisableReduceLanguage,BraveRewardsWebUIPanel<BraveRewardsWebUiPanelStudy,BraveVPN<CrossPlatformVPNStudy,BraveVPNLinkSubscriptionAndroidUI<CrossPlatformVPNStudy,EphemeralStorage<EphemeralStorageStudy,HttpsByDefault<BraveHttpsByDefaultRolloutStudy,MacCoreLocationBackend<MacCoreLocationBackendStudy,PartitionBlinkMemoryCache<PartitionBlinkMemoryCacheStudy,PartitionConnectionsByNetworkIsolationKey<PartitionConnectionsByNetworkIsolationKeyStudy,PartitionExpectCTStateByNetworkIsolationKey<PartitionConnectionsByNetworkIsolationKeyStudy,PartitionHttpServerPropertiesByNetworkIsolationKey<PartitionConnectionsByNetworkIsolationKeyStudy,PartitionSSLSessionsByNetworkIsolationKey<PartitionConnectionsByNetworkIsolationKeyStudy,PermissionLifetime<PermissionLifetimeReleaseStudy,ShouldAlwaysRunBraveAdsService<BraveSearchAdStudy,ShouldAlwaysTriggerBraveSearchResultAdEvents<BraveSearchAdStudy,ShouldLaunchBraveAdsAsInProcessService<BraveSearchAdStudy,ShouldSupportSearchResultAds<BraveSearchAdStudy,SplitHostCacheByNetworkIsolationKey<PartitionConnectionsByNetworkIsolationKeyStudy,UserActivity<UserActivityStudy" --disable-features="BraveAdblockDefault1pBlocking<Default1pBlockingStudy,BraveCleanupSessionCookiesOnSessionRestore<BraveCleanupSessionCookiesOnSessionRestore,BraveEnableAutoTranslate<BraveAutoTranslateStudy,BraveGoogleSignInPermission<BraveGoogleSignInPermissionStudy,ClampPlatformVersionClientHint<ClampPlatformVersionClientHint,ComputePressure<ForceDisableBlinkFeatures,IdleDetection<ForceDisableBlinkFeatures,NavigatorPluginsFixed<ForceDisableBlinkFeatures,SignedExchangeSubresourcePrefetch<ForceDisableBlinkFeatures,SmilAutoSuspendOnLag<SmilAutoSuspendOnLagKillSwitch,TextFragmentAnchor<ForceDisableBlinkFeatures,WebNFC<ForceDisableBlinkFeatures,WebOTP<ForceDisableBlinkFeatures" --disable-field-trial-config

[atuchin-m]

@valynor

I just did that in my standard user profile (with extensions) and it crashed the tab at 1635/4800 iterations. Then I added "browserbench.org" to the trusted sites in uBlock Origin 1.54 (effectively turning off uBO for this domain) and it ran to 4800/4800 without issue (score 343). I then repeated this in a guest window and again 4800/4800, score 347. So it seems that uBlock Origin is causing the tab crashes and probably the slowdowns too when doing long Speedometer runs. The way I see it is uBO or other extensions are the primary reason of your low speedometer results. Please check your main profile with all the extension in disabled state (on brave://extensions). Making an "exception" via the extensions settings is definitely not enough, most extensions inject to a page first, then check the exception lists.

It also explains, why:

So I synced back my usual profile via sync chain ... still 350 score but I decided to test some more, this time in the default profile

Your got your extensions back and the results degraded.

I then repeated this in a guest window and again 4800/4800, score 347.

Do you have uBO enabled in the guest window? (Allow in Incognito)

[valynor]

@atuchin-m

I did some more testing with extensions on and off but it was inconclusive so I went back to creating a fresh temp profile with the command you told me and discovered this inexplicable behaviour:

0. I made sure the profile in the /tmp/ directory was actually deleted
1. "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1
2. maximized the Brave window by option-double clicking on a corner
3. pasting https://browserbench.org/Speedometer2.1/ into the URL bar
4. running Speedometer 2.1 and getting the good score (~430)
5. clicked "Test Again" several times, score stays good

Now this is where it gets really weird:

6. I opened a guest profile from the Menu > More Tools > Open Guest Profile
7. I closed this Guest Window after a few seconds without doing anything in the window
8. I re-ran Speedometer from the still open tab in the default tmp profile: ~350 score and it does not go back up anymore even if I run it repeatedly

• Closing the browser and restarting it with "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" --user-data-dir=/tmp/clean_tmp_profile_1 in the same tmp profile instantly gives me 430ish scores again.
• I open and close a guest profile window and it is slow (350ish) again.

This happens EVERY time, 100% reproducibly. I hope you can reproduce this on your devices, too.

I can create a screen capture if you would like to see it for yourself.

• It also happens when I open & close a new Private Window (Shift+CMD+N) instead of the Guest Profile Window: good scores before, -20% scores after.

I have no idea what Brave is doing when I just open a Private/Guest Window but something is clearly very wrong here with the ARM64 version's performance on macOS Sonoma 14.1.2 with a M1 Max.

The loss of Speedometer performance described above is exactly what I am seeing on my real profile but in a Guest Profile Window with no extensions loaded (uBO is not running in my Guest Profile Window).

[atuchin-m]

@valynor The steps are interesting, but I can't reproduce the bug on my m1 pro MBP. I will ask colleagues tomorrow.

There is a "feature" in the profile subsystem: once you open another profile (including a guest one), it stays in an active state until you close the browser. It explains why a closed guest window could affect the primary profile.

1. Could you also try to repeat the test by adding --disable-brave-extension to the command line?
2. Please share the in-browser Task Manager screenshot at step 8. (menu -> more tools -> task manager) 
3. Let's record the performance traces: go to brave://tracing in another tab, start recording (see the screenshot), reproduce the good speedometer2 run, stop recording, and download the file. Then repeat the same for the "bad" result. Then share both files. Not: You will get a lower score, but it should be enough to catch the difference. Chromium instructions for the references: https://www.chromium.org/developers/how-tos/submitting-a-performance-bug/
4. The screencast will also help.

[valynor]

@atuchin-m

1. Could you also try to repeat the test by adding --disable-brave-extension to the command line?

I tried, no difference. BTW: I updated to Sonoma 14.2.

2. Please share the in-browser Task Manager screenshot at step 8. (menu -> more tools -> task manager)

Fun fact: while doing the task manager screenshot I discovered that simply opening the task manager window and closing it a couple seconds later will also give me the bad scores until I restart Brave.

4. The screencast will also help.

https://drive.google.com/file/d/1jmjCqpQme02BbfPQBze4k5YJnyugxBTW/view?usp=sharing

• note that only terminal and finder are running
• in the menu bar you can see my system monitor app showing: watts/hottest cpu core temp - usage per core - cpu% - gpu% - ram% - ssd i/o - network i/o
• note that I'm deleting the previous tmp profile before running the test, i.e. I'm starting in a completely fresh profile
• I'm just doing one speedometer run before/after to keep the video short

I will upload the requested traces later in a separate post.

1/2

[valynor]

2/2

Good score: https://drive.google.com/file/d/1WI_0LOK8JcFRbuBmySuHDJHiMoKjAODe/view?usp=drive_link

Bad score: https://drive.google.com/file/d/16yUf1MH4eBI_-cHofgrebdGa2anjAkjp/view?usp=drive_link

Really hope this helps you finding out what's going on.