Comcast Advanced Security blocks Storage by default on some networks (b/149740656)

[dloker]

[REQUIRED] Step 1: Describe your environment

• Xcode version: 11.3.1 (11C504)
• Firebase SDK version: 6.17.0 (fails for multiple versions)
• Firebase Component: Storage
• Component version: 3.5.0
• Installation method: CocoaPods

[REQUIRED] Step 2: Describe the problem

Loading data from Storage using imageView.sd_setImage or using downloadURL causes SSL failure and failure to load data. I've appended the log output at the bottom of this bug report (I anonymized the project URL). The code I have used for loading images for our app has not changed in months, but stopped working yesterday. It seems to work still for the simulator on iOS 11.4, but not 12.2 or 13.3. I was also able to reproduce with a brand new project and one-view app.

I have tried fixing it with App Transport Security Settings - Allow Arbitrary Loads = YES - this does not fix it.

The error looks like this:

2020-02-16 09:52:21.432162-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_context_error_print(1862) boringssl ctx 0x6000039f0390: 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.434125-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_session_handshake_error_print(111) [C2.1:2][0x7fb4f5c08380] 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.434182-0800 FirebaseDo[92100:985470] [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(726) [C2.1:2][0x7fb4f5c08380] handshake failed at state 12288
2020-02-16 09:52:21.434931-0800 FirebaseDo[92100:985470] Connection 2: received failure notification
2020-02-16 09:52:21.435047-0800 FirebaseDo[92100:985470] Connection 2: failed to connect 3:-9858, reason -1
2020-02-16 09:52:21.435118-0800 FirebaseDo[92100:985470] Connection 2: encountered error(3:-9858)
2020-02-16 09:52:21.480842-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_session_handshake_error_print(111) [C4.1:2][0x7fb4f0e05380] 140415105779912:error:10000118:SSL routines:OPENSSL_internal:NO_SUPPORTED_VERSIONS_ENABLED:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/ssl_versions.cc:234:
Error Domain=NSURLErrorDomain Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made."

Full log is at the end.

Steps to reproduce:

To reproduce, simply create a new project that's hooked up to a Firebase project where you've uploaded an image or some other data - I used an image.

Create a one-view app with a button that loads an image from Firebase Storage into an imageview. I've provided our AppDelegate and ViewController swift code below.

I have tried fixing it with App Transport Security Settings - Allow Arbitrary Loads = YES - this does not fix it.

Relevant Code:

AppDelegate.swift

...
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        // Override point for customization after application launch.
        FirebaseApp.configure()
        return true
    }
...

ViewController.swift

import UIKit
import Firebase
import FirebaseUI
class ViewController: UIViewController, FUIAuthDelegate {
    @IBOutlet weak var executeSample: UIButton!
    @IBOutlet weak var imageView: UIImageView!
    @IBAction func execute(_ sender: Any) {
        Auth.auth().signInAnonymously() { (authResult, error) in
            if let error = error {
                print("Error: \(error)")
            }
            guard let user = authResult?.user else { return }
            let isAnonymous = user.isAnonymous  // true
            let uid = user.uid
            let store = Storage.storage()
            let storeRef = store.reference()
            let coverArt = storeRef.child("cover_art.jpg")
            self.imageView.sd_setImage(with: coverArt)
        }
    }
    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view, typically from a nib.
    }
    override func didReceiveMemoryWarning() {
        super.didReceiveMemoryWarning()
        // Dispose of any resources that can be recreated.
    }
}

Full Logs:

2020-02-16 09:52:10.800736-0800 FirebaseDo[92100:985334] <Warning>: Please set a value for FacebookAutoLogAppEventsEnabled. Set the flag to TRUE if you want to collect app install, app launch and in-app purchase events automatically. To request user consent before collecting data, set the flag value to FALSE, then change to TRUE once user consent is received. Learn more: https://developers.facebook.com/docs/app-events/getting-started-app-events-ios#disable-auto-events.
2020-02-16 09:52:10.800884-0800 FirebaseDo[92100:985334] <Warning>: You haven't set a value for FacebookAdvertiserIDCollectionEnabled. Set the flag to TRUE if you want to collect Advertiser ID for better advertising and analytics results. To request user consent before collecting data, set the flag value to FALSE, then change to TRUE once user consent is received. Learn more: https://developers.facebook.com/docs/app-events/getting-started-app-events-ios#disable-auto-events.
2020-02-16 09:52:10.981648-0800 FirebaseDo[92100:985468] 6.17.0 - [Firebase/Analytics][I-ACS023007] Analytics v.60202000 started
2020-02-16 09:52:10.981841-0800 FirebaseDo[92100:985468] 6.17.0 - [Firebase/Analytics][I-ACS023008] To enable debug logging set the following application argument: -FIRAnalyticsDebugEnabled (see http://goo.gl/RfcP7r)
2020-02-16 09:52:11.002604-0800 FirebaseDo[92100:985474] 6.17.0 - [Firebase/Analytics][I-ACS031025] Analytics screen reporting is enabled. Call +[FIRAnalytics setScreenName:setScreenClass:] to set the screen name or override the default screen class name. To disable screen reporting, set the flag FirebaseScreenReportingEnabled to NO (boolean) in the Info.plist
2020-02-16 09:52:11.005015-0800 FirebaseDo[92100:985475] 6.17.0 - [Firebase/Analytics][I-ACS023012] Analytics collection enabled
2020-02-16 09:52:11.030681-0800 FirebaseDo[92100:985334] FBSDKLog: Missing [FBSDKAppEvents appID] for [FBSDKAppEvents publishInstall:]
2020-02-16 09:52:11.031954-0800 FirebaseDo[92100:985334] FBSDKLog: Missing [FBSDKAppEvents appEventsState.appID] for [FBSDKAppEvents flushOnMainQueue:]
2020-02-16 09:52:21.432076-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_context_handle_fatal_alert(1872) [C2.1:2][0x7fb4f5c08380] write alert, level: fatal, description: protocol version
2020-02-16 09:52:21.432162-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_context_error_print(1862) boringssl ctx 0x6000039f0390: 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.434125-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_session_handshake_error_print(111) [C2.1:2][0x7fb4f5c08380] 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.434182-0800 FirebaseDo[92100:985470] [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(726) [C2.1:2][0x7fb4f5c08380] handshake failed at state 12288
2020-02-16 09:52:21.434931-0800 FirebaseDo[92100:985470] Connection 2: received failure notification
2020-02-16 09:52:21.435047-0800 FirebaseDo[92100:985470] Connection 2: failed to connect 3:-9858, reason -1
2020-02-16 09:52:21.435118-0800 FirebaseDo[92100:985470] Connection 2: encountered error(3:-9858)
2020-02-16 09:52:21.458303-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_context_handle_fatal_alert(1872) [C3.1:2][0x7fb4f5e02980] write alert, level: fatal, description: protocol version
2020-02-16 09:52:21.458407-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_context_error_print(1862) boringssl ctx 0x6000039e06e0: 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.458685-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_session_handshake_error_print(111) [C3.1:2][0x7fb4f5e02980] 140415105779912:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/tls_record.cc:242:
2020-02-16 09:52:21.458760-0800 FirebaseDo[92100:985470] [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(726) [C3.1:2][0x7fb4f5e02980] handshake failed at state 12288
2020-02-16 09:52:21.459937-0800 FirebaseDo[92100:985470] Connection 3: received failure notification
2020-02-16 09:52:21.460097-0800 FirebaseDo[92100:985470] Connection 3: failed to connect 3:-9858, reason -1
2020-02-16 09:52:21.460164-0800 FirebaseDo[92100:985470] Connection 3: encountered error(3:-9858)
2020-02-16 09:52:21.480842-0800 FirebaseDo[92100:985470] [BoringSSL] boringssl_session_handshake_error_print(111) [C4.1:2][0x7fb4f0e05380] 140415105779912:error:10000118:SSL routines:OPENSSL_internal:NO_SUPPORTED_VERSIONS_ENABLED:/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl_Sim/boringssl-283.60.3/ssl/ssl_versions.cc:234:
2020-02-16 09:52:21.480986-0800 FirebaseDo[92100:985470] [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(726) [C4.1:2][0x7fb4f0e05380] handshake failed at state 0
2020-02-16 09:52:21.482421-0800 FirebaseDo[92100:985470] Connection 4: received failure notification
2020-02-16 09:52:21.482551-0800 FirebaseDo[92100:985470] Connection 4: failed to connect 3:-9858, reason -1
2020-02-16 09:52:21.482665-0800 FirebaseDo[92100:985470] Connection 4: encountered error(3:-9858)
2020-02-16 09:52:21.485367-0800 FirebaseDo[92100:985470] Task <B7521350-AF90-43FE-AECD-73B4815671E8>.<1> HTTP load failed, 0/0 bytes (error code: -1200 [3:-9858])
2020-02-16 09:52:21.488973-0800 FirebaseDo[92100:985470] Task <B7521350-AF90-43FE-AECD-73B4815671E8>.<1> finished with error [-1200] Error Domain=NSURLErrorDomain Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={NSErrorFailingURLStringKey=https://firebasestorage.googleapis.com/v0/b/sample-project-**_<IDENTIFIER>_**.appspot.com/o/cover_art.jpg?alt=media, NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFStreamErrorDomainKey=3, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <B7521350-AF90-43FE-AECD-73B4815671E8>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <B7521350-AF90-43FE-AECD-73B4815671E8>.<1>"
), NSLocalizedDescription=An SSL error has occurred and a secure connection to the server cannot be made., NSErrorFailingURLKey=https://firebasestorage.googleapis.com/v0/b/sample-project-**_<IDENTIFIER>_**.appspot.com/o/cover_art.jpg?alt=media, NSUnderlyingError=0x600003568930 {Error Domain=kCFErrorDomainCFNetwork Code=-1200 "(null)" UserInfo={_kCFStreamPropertySSLClientCertificateState=0, _kCFNetworkCFStreamSSLErrorOriginalValue=-9858, _kCFStreamErrorDomainKey=3, _kCFStreamErrorCodeKey=-9858}}, _kCFStreamErrorCodeKey=-9858}

[rizafran]

Thanks for reporting the issue, @dloker. Could you provide the following details:

• Is this happening all the time? If so, what location are you in?
• Do you have any restrictions in your network?
• Have you tried connecting to other networks?

[dloker]

I am in the Bay Area, though Firebase is set up in us-central. It is still happening for me every time I try and load something from storage. It also happens when I try to get the metadata info like the downloadUrl.

It's also worth noting that it happens on my iOS device as well, not just the simulator. I have tried it at another location, but it was still Comcast internet. Having just tried it via GoogleFi - it is working. Very interesting. Doesn't appear to be an API issue then, but a comcast issue?

[dloker]

Having checked my Comcast gateway, I noticed Advanced Security was enabled and that it had labeled firebase.googleapis.com as a threat. I disabled the security feature and restarted the gateway and now I can connect and download from Storage.

While this works for me, it is very scary to think that my users who have comcast with their new Advanced Security feature are going to be blocked from downloading from Firebase Storage. I had my friend do the same on his gateway and it is now working for him at the other location.

[morganchen12]

Thanks for raising this, @dloker. @rizafran can you file a bug internally to aggregate any issues/support tickets related to Comcast advanced security?

[Tunscopi]

Yup, I hit the same exact issue. And after banging my head with different ATS configurations to no avail, I stumbled on this issue. I also use comcast, I am in Seattle WA and also have my Firebase set up in us-central.

Thanks @dloker for your investigations, it also worked for me when I tried using my cellular "personal hotspot".

Here's a screenshot from my gateway app:

[rizafran]

Thanks for letting us know about this issue. Internally tracked at b/149740656.

[evancarufel]

I spent hours looking into this as well. Same resolutions as above. Thank you for filling issue.

[morganchen12]

Does this failure also occur on other platforms (web/Android) with Comcast Advanced Security?

[evancarufel]

Web Chrome worked fine

[adelburekovic]

We have the same issue but we couldn't pinpoint what exactly is the problem. So for now the only solution is to tell the customers to unblock firebase?

[jessicay88]

Any updates for this issue?

[morganchen12]

Telling your users to unblock Firebase in Comcast's setting is a viable but not really scalable workaround. We'll try to figure out how to get this globally unblocked on Comcast networks.

No progress updates yet, unfortunately.

[ehmjaysee]

Same problem here. See the screen shot of my Xfinity "threat history". Turning off Wifi immediately solved the problem.

[blazmag]

We've had a few user report that the issue resolved itself on its own. Likely Comcast stopped blacklisting the domain and it took a few days to propagate to users routers.. But can anyone else confirm it's no longer happening? 🤞 🤞

[ehmjaysee]

Problem is still happening for me. Rebooting the router did not help. This router was just installed a month ago, Arris model TG3482G. My location is Minneapolis.

[Update] I found how to disable "Advanced Security" on my router and the problem went away.

BTW if you search for this problem on the xFi helpdesk you will find many people reporting similar problems. See screenshot.

[jessicay88]

Would someone happen to have an update on this? Problem appears to be widespread for many developers that rely on Firebase Storage. For us, this is having a large impact on our app users since we rely on Firebase Storage to store images.

[ehmjaysee]

It might help to register a complaint directly with xfinity. Not sure that anyone at Google/Firebase can fix this. https://www.xfinity.com/support/

[ehmjaysee]

Firebase folks, the current status for Firebase Storage is green which makes sense, but since this is a service impacting problem would this be reported on any Firebase status page? I ask because this gihub page is not generally where I would look to know there was a problem.

[morganchen12]

Good point, I'll ask the support team to update the status.

If any of you have Charles proxy set up, can you run a session of your app and share a proxy log with any relevant Storage requests? If you've redacted all non-Storage info from the log you can post it here, otherwise file it in a support request at Firebase support.

[tennydesign]

Same problem happening to me. Bay Area Comcast as well. Opened another issue but will refer to this one cause this is the right thread.

[tennydesign]

Workaround: If you have Xfinity and enable the Hotspot in their App on "internet devices", the router will propagate another Wifi, aimed for guests. This one is not being routed through the security gate that is blocking Storage access, so it works fine. I think it may be a good idea to keep one blocked, as it is and switch to the open one when needed, just so I keep checking for when the issue is solved.

[Altoar]

Same issue for my web app users with Comcast. The first reports from my users came early this week Monday 17th.

The console log puts this error out when I'm using .getDownloadURL() firebase storage function. Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR breadcrumbs.js:375

Tested on multiple browsers. Hope this issue can get fixed soon

[worldofnick]

Experiencing the same issue here. Taking my device off WiFI and onto LTE solved the issue, but this isn't ideal for or intuitive for users.

[Galkon]

Experiencing a related issue for thousands of users, this is a huge problem for us. We are using the web libraries for uploads in an electron app (on the BrowserWindow render thread). First reports came in around Feb 12th, it took us awhile to get a proper error back:

POST https://firebasestorage.googleapis.com/v0/b/... net::ERR_SSL_PROTOCOL_ERROR

[morganchen12]

Hey all, after talking to the Storage backend team, it looks like this won't realistically be fixable on the Firebase end. If you're running into this issue, please file a support ticket with Firebase or comment here so we can track the extent of the impact, and then also file a ticket to Comcast support. Currently our plan is to try to get the storage domain unblocked by Comcast Advanced Security.

I'll keep this issue updated as it progresses.

[corykrug]

@morganchen12 we're in the same boat. Several of our users, primarily in the Bay Area, but also some in central/Southern California, are unable to use our site. We also filed a Firebase ticket and intend to file one with Comcast as-well.

[tennydesign]

@morganchen12 I get it this is more on the ecosystem than with Storage, but even though this is not fixable completely on your end, to ask us to hit Comcast with tickets is not a great plan. Each one of us have close to zero leverage, many of us are not even Comcast's customers, which gives us even less leverage comparing with what Google has. Please report back if the discussion with Comcast advances, I think this will only be solved by you working together with their team to unblock it.

[morganchen12]

@tennydesign we're still working on it on our end as well, didn't mean to imply that you were all on your own.

[corykrug]

@morganchen12 much appreciated that you’re keeping us in the loop.

I spent quite a long time over the past couple days trying to track down a root cause before finding this issue. Any chance this could be bubbled up on the Firebase system status page? Even though the root cause isn’t with Google itself, I’m sure many would appreciate acknowledgment in a more official and easy to find location.

[corykrug]

Popping my head back in here because I'm baffled at the general quietness on this issue, and I'm getting the impression not a lot of folks are using Firebase seriously. Comcast is one of the larger ISPs here in the US, surely Google has proper contacts.

Any chance we could get acknowledgment that this is being escalated properly within Google?

[morganchen12]

Hi Cory, we're definitely still escalating this issue. This is about as noisy as it gets for issues on this repository, which is nowhere near representative of the scale/impact of the issue.

[evancarufel]

For what it's worth this was not an issue until I updated my pod for my iOS app. I've been waiting for a res for this but figured if it work on an old version of the SDK maybe it is something they could fix. Been on this thread for over a week.

On Wed, Feb 26, 2020, 6:39 PM Morgan Chen notifications@github.com wrote:

Hi Cory, we're definitely still escalating this issue. This is about as noisy as it gets for issues on this repository, which is nowhere near representative of the scale/impact of the issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-ios-sdk/issues/4913?email_source=notifications&email_token=AOS5GHUULZU2YSQG73FGJZ3RE4DVDA5CNFSM4KWFQQ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENCOGZQ#issuecomment-591717222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOS5GHXUZHUYG72BLALJZGLRE4DVDANCNFSM4KWFQQ5A .

[nourabbas96]

Hi all

The bad news here is that this issue is not just happening in the US, it is global with many ISPs blocking Firebase, this is a killing situation for Firebase

We have been trying to fix this for more than 10 days now, multiple ISPs around the world are blocking our firebase application and users can not open our website, we logged a case with Google and waiting for a resolution, the other alternative for us will be to use another Cloud provider

[nourabbas96]

Hey all, after talking to the Storage backend team, it looks like this won't realistically be fixable on the Firebase end. If you're running into this issue, please file a support ticket with Firebase or comment here so we can track the extent of the impact, and then also file a ticket to Comcast support. Currently our plan is to try to get the storage domain unblocked by Comcast Advanced Security.

I'll keep this issue updated as it progresses.

The issue with other ISPs around the world for us: Orange and Vodafone for example

This should be fixed on Google level, not with Comcast because its not only Comcast users encounter this issue

We can not wait for Google to send requests to all ISPs worldwide to unmark this as a security threat

Please advise, we are considering other cloud vendors now as this means that Google Cloud can't be reliable for us

Thanks a lot

[blazmag]

We've ended up migrating away from firebase storage to our main cloud vendor (AWS S3 + CF as CDN) as it looked like it won't be resolved anytime soon. Our case was fairly trivial, but I'm sure that wont be the case with others.

This needs to be elevated and Google needs to put more pressure on those ISPs.

[blazmag]

We've ended up migrating away from firebase storage to our main cloud vendor (AWS S3 + CF as CDN) as it looked like it won't be resolved anytime soon. Our case was fairly trivial, but I'm sure that wont be the case with others.

This needs to be elevated and Google needs to put more pressure on those ISPs.

[george33zmax]

Any news on this. This is really bad :(

[adamhalper2]

Ahhh! I went down so many rabbit holes trying to fix this. So now we wait?

[corykrug]

I posted on the Xfinity/Comcast forums, but I'm sure we all know where that's going to lead: https://forums.xfinity.com/t5/Your-Home-Network/Xfinity-blocking-Google-Firebase-Domains/td-p/3309191

[justkd-google]

The issue with Xfinity/Comcast should be fixed now. Please let us know if you continue to see this problem

[tennydesign]

I'm still getting this: ResponseErrorCode=-1200, NSLocalizedDescription=An unknown error occurred, please check the server response. Which is the exact same error I was getting before.

[tennydesign]

Correction: Rebooted my router and it worked. @justkd-google the correct closure message should include a step to reboot the router. Thanks for hunting this down. Big relief.

[justkd-google]

Glad to hear - thanks @tennydesign for reporting that the solution requires rebooting the router (unfortunate that step is necessary, but it doesn't appear we can get around that).

[dloker]

I can also confirm it works now for myself with XFi Advanced Security enabled. I did not have to reboot the router, though.

[ehmjaysee]

Does this require a Firebase update? Or was it resolved entirely by Xfinity/Comcast?

[morganchen12]

This was resolved by Comcast and does not require any Firebase updates.

[jla]

Barracuda is listing firebasestorage.googleapis.com under the phishing-fraud category.

[morganchen12]

Is it causing access issues for your users?

[jla]

Yes, they are getting a firebase storage error "max retry time for operation exceeded please try again" when they connect through their corporate network. There are a bunch of big companies that use those firewalls.

I already filed a support ticket with Firebase

[morganchen12]

Thanks. We'll track is in the support ticket to avoid spamming this issue that has a bunch of subscribers who may not be using Barracuda.