Android app not respecting user-installed CA certificates

tsndr commented 5 months ago

Hey, today I noticed that my BlueBubbles app can't connect to the server anymore.

My Use Case

Hosted on local Mac mini
Using local DNS that resolves the public domain to the local IP of my Mac mini (same domain and port interally and externally)
Using self signed SSL certificate

Things I've tried

Check if my URL is reachable (internal and external) ✅
Make sure the correct SSL certificate is used by the server (internal and external) ✅
Make sure the CA certificate is installed on my phone ✅
Check if the webapp is working through bluebubbles.app ✅

Observations

The android app can't connect either from within the network nor externally.
The phone app just say's that it can't connect to the Websocket.

Is there any way I can debug this further?

SpaceSaver commented 5 months ago

Does the url shown on the Android client match the url shown on the server?

tsndr commented 5 months ago

Yes it does

SpaceSaver commented 5 months ago

Have you tried restarting the server app?

tsndr commented 5 months ago

Multiple times, also checked for updates (which there weren't any) and even rebooted the Mac mini.

SpaceSaver commented 5 months ago

Try disabling "Detect localhost Address"

tsndr commented 5 months ago

Never had it enabled.

Using local DNS that resolves the public domain to the local IP of my Mac mini (same domain and port interally and externally)

Wasn't necessary, because of my local DNS server, that points my domain to the local address. Everything worked up until recently, and since I didn't make any changes to my configuration, I'd imagine that the recent update to the app could have something to do with it?

jjoelj commented 5 months ago

Make sure the certificate on the server includes the entire certificate chain. Other people have experienced similar issues on 1.13.0. See #2673

tsndr commented 5 months ago

There is no chain, just the CA certificate and the certificate for my domain.

The CA certificate usualy isn't included in the certificate chain since it is installed on all target devices, only intermediate certificates should be (if present).

jjoelj commented 5 months ago

It would be helpful if you could collect logs from the Android app using Settings > Troubleshooting > start logging/end logging.

tsndr commented 5 months ago

Thanks for the hint, was looking for a log but didn't see the option.

Unfortunately the log isn't really helpful:

[BlueBubblesApp][2024-02-27 18:22:05.388529][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:05.433431][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:06.761570][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:06.762308][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:06.762956][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:06.767638][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:22:07.759693][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:07.829685][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:08.575974][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:08.576718][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:08.576964][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:08.582472][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:22:09.483964][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:09.613110][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:10.437295][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:10.438031][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:10.439126][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:10.445130][INFO] Sending method get-server-url to Kotlin

jjoelj commented 5 months ago

Try starting logging, then going to Connection & Server settings and clicking 'fetch latest url'. Then end logging. It should log an error.

tsndr commented 5 months ago

Here we have some errors:

[[BlueBubblesApp][2024-02-27 18:34:27.909874][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:27.958314][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:29.226711][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:29.226945][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:29.227045][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:29.232638][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:30.553920][INFO] (REQUEST[GET]) -> PATH: https://imessage.mydomain.com:54321/api/v1/ping
[BlueBubblesApp][2024-02-27 18:34:30.604116][ERROR] (ERROR[null]) -> PATH: https://imessage.mydomain.com:54321/api/v1/ping
[BlueBubblesApp][2024-02-27 18:34:30.604418][ERROR] (ERROR[null]) -> HandshakeException: Handshake error in client (OS Error: 
    CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
[BlueBubblesApp][2024-02-27 18:34:30.604556][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:30.604595][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:30.688345][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:30.740924][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:32.648183][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:32.717029][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:32.961218][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:32.961789][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:32.961923][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:32.965123][INFO] Sending method get-server-url to Kotlin

....

[BlueBubblesApp][2024-02-27 18:34:40.774652][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:40.818161][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:41.835292][INFO] (REQUEST[GET]) -> PATH: https://imessage.mydomain.com:54321/api/v1/fcm/client
[BlueBubblesApp][2024-02-27 18:34:41.910744][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:41.911118][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:41.911236][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.915888][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.962439][ERROR] (ERROR[null]) -> PATH: https://imessage.mydomain.com:54321/api/v1/fcm/client
[BlueBubblesApp][2024-02-27 18:34:41.962565][ERROR] (ERROR[null]) -> HandshakeException: Handshake error in client (OS Error: 
    CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
[BlueBubblesApp][2024-02-27 18:34:41.962640][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:41.962663][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:41.962821][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:41.962849][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:41.962886][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.964463][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:43.113867][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:43.198279][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:43.844429][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:43.844724][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:43.844943][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:43.848563][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:45.277396][INFO] Connecting to socket...

So it is an SSL error. I do have my CA certificate installed on my device, and double checked through my browser that it works, no certificate issues there.

jjoelj commented 5 months ago

Well, I guess that confirms that it's a certificate issue. And all sources I can find say that it's some issue with the certificate chain. You're right that the CA certificate should not be included in the chain, but double check that your self-signed certificate has all the correct intermediate certificates. Browsers tend to automatically complete the chain to account for misconfigured webservers, which is most likely why there are no issues when going through a browser.

You can view the completed certificate used by your browser, but steps vary based on what browser you're using. I'd recommend exporting whatever certificate the browser completes and comparing it to the one you have added to the server.

tsndr commented 5 months ago

There are no intermediate certificates in my case, my certificate chain looks like this:

CA Certificate
└─ imessage.mydomain.com

This matches the output of openssl s_client and my browsers output.

Only the imessage.mydomain.com certificate has to be send by the server, since the device has the CA certificate installed onto it.

tsndr commented 5 months ago

Could it be possible, that the app isn't respecting the systems CA certificates?

jjoelj commented 5 months ago

https://github.com/flutter/flutter/issues/41781#issuecomment-702948726

I did some more research, and it seems like we have to specifically allow user-installed CA Certs in the network security config on Android. I have no idea why this wasn't an issue before.

tsndr commented 5 months ago

Awesome find, can someone push an update?

Until now, I've just had a red certificate warning on the Connection & Server server page, but everything worked so I didn't really care.

jjoelj commented 5 months ago

Unfortunately, it seems like this is still an open issue with the dart HTTP client which is what gets used in BB https://github.com/dart-lang/sdk/issues/50435

The comment I referenced earlier fixed the issue for a person who wrote their own native code for HTTP, so it won't apply to us.

Again, I don't know why it was working before, but we will investigate what changed.

tsndr commented 5 months ago

Let me know if I can do anything to help.

duronald commented 5 months ago

Also having this issue, rolling back to the last stable version, v1.12.7 temporarily fixes it for now.

duronald commented 5 months ago

Upon further testing, it appears that previous versions of the app did not check the user certificate store at all. I removed my certificate and it still worked fine on previous versions. The new one must autoreject a connection if it can't find the cert and doesn't check the user certificate store.

SpaceSaver commented 5 months ago

Upon further testing, it appears that previous versions of the app did not check the user certificate store at all. I removed my certificate and it still worked fine on previous versions. The new one must autoreject a connection if it can't find the cert and doesn't check the user certificate store.

Make sure that the whole cert chain is installed.

duronald commented 5 months ago

I checked and unless I did something wrong all intermediates and root self signed certs are installed.

BlueBubblesApp / bluebubbles-app