search

brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.54k stars 2.27k forks source link

Test failure: BraveNetworkAuditTest.BasicTests #40190

Open brave-builds opened 1 month ago

brave-builds commented 1 month ago

Greetings human!

Bad news. BraveNetworkAuditTest.BasicTests failed on macos x64 beta v1.69.131.

Stack trace ``` [ RUN ] BraveNetworkAuditTest.BasicTests [ FAILED ] BraveNetworkAuditTest.BasicTests ```
Previous issues * #39415 * #38812 * #38418 * #37557 * #35021 * #34795 * #34277 * #34351 * #34294 * #33737 * #31796 * #31766 * #28812 * #28150 [Find all](https://github.com/brave/brave-browser/issues?q=type%3Aissue+label%3Abot%2Ftype%2Ftest+in%3Atitle+%22Test+failure%3A+BraveNetworkAuditTest.BasicTests%22)
brave-builds commented 1 month ago

BraveNetworkAuditTest.BasicTests failed on macos x64 beta v1.69.132.

Stack trace ``` [ RUN ] BraveNetworkAuditTest.BasicTests [ FAILED ] BraveNetworkAuditTest.BasicTests ```
sangwoo108 commented 1 month ago

I tried running the test on my Mac, and got timeout. After adjusting const int kMaxTimeoutPerLoadedURL = 30, it succeeded. Let me try adjusting the seconds

My bad, I should have used npm run network-audit instead of npm run test -- brave_network_audit_tests

sangwoo108 commented 1 month ago 
[88891:259:0803/120521.801450:WARNING:brave_network_audit_browsertest.cc(171)] NETWORK AUDIT WARNING:http://127.0.0.1:60673/favicon.ico

[88891:259:0803/120521.804476:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjcuMS43MC4wGgwIBRAEIgQgASACKAEaDAgBEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwIBxAEIgQgASACKAEaDAgBEAgiBCABIAIoBBoMCA8QBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken

[88891:259:0803/120521.804665:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjcuMS43MC4wGgwIBRAEIgQgASACKAEaDAgBEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwIBxAEIgQgASACKAEaDAgBEAgiBCABIAIoBBoMCA8QBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken

[88891:259:0803/120521.804846:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjcuMS43MC4wGgwIBRAEIgQgASACKAEaDAgBEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwIBxAEIgQgASACKAEaDAgBEAgiBCABIAIoBBoMCA8QBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken

../../brave/browser/net/brave_network_audit_browsertest.cc:280: Failure
Value of: PerformNetworkAuditProcess(events)
  Actual: false
Expected: true
network-audit FAILED. Import /Users/sangwoo/brave-browser/src/network_log.json in chrome://net-internals for more details.
Stack trace:
0   brave_network_audit_tests           0x0000000102afde2c brave::(anonymous namespace)::BraveNetworkAuditTest::TearDownInProcessBrowserTestFixture() + 1468
1   brave_network_audit_tests           0x0000000104d4fe28 content::BrowserTestBase::SetUp() + 2004
2   brave_network_audit_tests           0x0000000104d4c204 InProcessBrowserTest::SetUp() + 380

[88891:259:0803/120521.823613:INFO:brave_network_audit_browsertest.cc(191)] Network audit results stored in /Users/sangwoo/brave-browser/src/network_audit_results.json

[  FAILED  ] BraveNetworkAuditTest.BasicTests, where TypeParam =  and GetParam() =  (273875 ms)
sangwoo108 commented 1 month ago

It seems we were trying to block safebrowsing.googleapis.com/ but not sure why/where it was requested. Can we get a help from who knows safe browsing well?

diracdeltas commented 1 month ago

@fmarier

fmarier commented 1 month ago

@bridiver was investigating these failures last month and wasn't sure whether or not it was a browser problem: https://bravesoftware.slack.com/archives/C7VLGSR55/p1719517463492349?thread_ts=1719477643.270759&cid=C7VLGSR55

brave-builds commented 1 month ago

BraveNetworkAuditTest.BasicTests failed on macos x64 nightly v1.70.55.

Stack trace ``` [ RUN ] BraveNetworkAuditTest.BasicTests [ FAILED ] BraveNetworkAuditTest.BasicTests ```
fmarier commented 1 month ago

I tried to reproduce locally on Linux using mitmproxy and I only saw requests to safebrowsing.brave.com (as expected).

I also ran the network auditor locally using:

npm run network-audit -- --enable-pixel-output-in-tests

and it was successful.

Maybe it's a macOS-only problem?

fmarier commented 1 month ago

These requests are for the Safe Browsing lists. They take place within 5 minutes of startup.

fmarier commented 1 month ago

@sangwoo108 Were you able to reproduce locally?

sangwoo108 commented 1 month ago

Yes, the callstack above was from my m1 macbook

fmarier commented 1 month ago

@sangwoo108 Hm, okay, I guess there might be something Safe Browsing-related that's only enabled on Mac and not on Windows or Linux. Strange, but possible.

Does it happen both on arm64 and x86_64 builds on Mac?

sangwoo108 commented 1 month ago

Got the same error on my intel mac

[1401:259:0808/162039.115320:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjguMS43MC4wGgwIBRAEIgQgASACKAEaDAgHEAQiBCABIAIoARoMCAEQCCIEIAEgAigEGgwIARAEIgQgASACKAEaDAgPEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken

[1401:259:0808/162039.115846:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjguMS43MC4wGgwIBRAEIgQgASACKAEaDAgHEAQiBCABIAIoARoMCAEQCCIEIAEgAigEGgwIARAEIgQgASACKAEaDAgPEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken

[1401:259:0808/162039.116391:ERROR:brave_network_audit_browsertest.cc(175)] NETWORK AUDIT FAIL:https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?$req=ChYKCGNocm9taXVtEgoxMjguMS43MC4wGgwIBRAEIgQgASACKAEaDAgHEAQiBCABIAIoARoMCAEQCCIEIAEgAigEGgwIARAEIgQgASACKAEaDAgPEAQiBCABIAIoARoMCAMQBCIEIAEgAigBGgwICRAEIgQgASACKAEiAggB&$ct=application/x-protobuf&key=dummytoken
fmarier commented 1 month ago

I don't see anything Mac-specific in the code. That endpoint is defined here and then used here.

We override it here in Brave (prefix defined here), but for some reason on Mac that network delegate isn't redirecting this.

fmarier commented 1 month ago

cc @goodov who might have some ideas for why the network delegate (or the network audit script) is not working on Mac.

fmarier commented 1 month ago

Setting to P2 until we know whether it's the test or the static redirects which are broken.

brave-builds commented 4 days ago

BraveNetworkAuditTest.BasicTests failed on macos x64 release v1.69.168.

Stack trace ``` [ RUN ] BraveNetworkAuditTest.BasicTests [ FAILED ] BraveNetworkAuditTest.BasicTests ```