Unable to Connect [webOS 3.0, 3.5, more?]

anthonylavado commented 3 years ago

Issue During certification testing, devices running LG webOS 3.0 or 3.5 were unable to connect to the demo server.

Notes from Testing

When selecting the Connect UI Button after entering the received URL, the message ""Unknown error occurred. Are you connecting to a Jellyfin sever?"" is displayed.

Next Steps It is possible that this was an intermittent issue, but because of the failed result, LG stopped testing on webOS 3/3.5 devices at this point.

For internal (Jellyfin) testing:

Install more webOS emulators and test connectivity with app to internal demo server.
Check equipment list for team members with webOS 3/3.5 equipment and ask them to test against internal demo server.

For community help:

If you have a webOS 3/3.5 device, please let us know if you are able to test this. You can use the public demo server, https://demo.jellyfin.org/stable. The username is demo, with no password. If you need a compiled IPK to install, please let me know. Note that the public server is reset on the hour, every hour. It is recommended to avoid testing at that moment, as it will likely result in a connection error. This does not apply to our internal testing server.

dab2020 commented 3 years ago

I have multiple webOS devices and for me The Jellyfin app has been working on the devices that are running webOS 3.0 and 3.5. but I haven't updated the app in a while so I may be a bit behind the current release. I'll try to recompile it today and test it out.

anthonylavado commented 3 years ago

Connecting to the public demo server from the app may not work just yet - there's a change I have to make to the reverse proxy config.

This change was already made for the internal demo server, so that won't be part of the issue affecting the client.

oddstr13 commented 3 years ago

The error message originates from here https://github.com/jellyfin/jellyfin-webos/blob/25b774482b325c06ef2554c150ae3ea4c60214d3/org.jellyfin.webos/js/index.js#L320-L336 Looks like the isNaN check is inverted, so this would most likely have been caused by a 4xx or 5xx status code returned by the server.

anthonylavado commented 3 years ago

A testing release is available here: https://github.com/jellyfin/jellyfin-webos/issues/50

anthonylavado commented 3 years ago

In #50, a user mentions that they are also unable to connect to the public demo server when using the 3.0 emulator, but it works fine under 5.0 (which is mainly what I tested with). We have a variety of CSP and other headers in place, so I'll have to review these to see what's stopping it.

anthonylavado commented 3 years ago

In #50, @agrisci has figured out the issue. It looks like we need to enable older protocols for SSL for the internal testing server.

https://github.com/jellyfin/jellyfin-webos/issues/50#issuecomment-821836996

dovecode commented 3 years ago

As mentioned in #46, I also cannot connect to the public demo server from my WebOS 6.0.5 TV. Not sure if there's a debug log available for a physical TV I can inspect to see if it is a TLS issue as well,...

anthonylavado commented 3 years ago

So there's been a bunch of work behind the scenes. We've made sure that the internal and public demo servers are using TLS 1.2, as that is supported across all webOS versions.

From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.

We've currently made alternate arrangements to provide LG with an HTTP URL for testing, but obviously we'll still want to figure out how we can make this work for everyone.

caeci11ius commented 3 years ago

From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.

Have run into this specific issue before when using Emby's app with letsencrypt in the past. You're right to note that the tv browser works fine with letsencrypt, but there is a certificate problem that seems specific to apps on webos (this is on an LG C8 running what would've been late v4 or early v5 firmware at the time). I don't imagine it's fixable without cooperation from LG, but would be great if you could draw it to their attention. Maybe they'll do a firmware update!

mono-of-pg commented 3 years ago

I've dealt a lot with SSL/TLS related problems in the past. This specific one could simply be a TLS handshake mismatch.

Your demo server supports the following cipher suites for TLS 1.2:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

https://webostv.developer.lge.com/discover/specifications/web-engine/ states that WebOS 3.x uses a Chromium 38 engine which just might not be capable of using those ciphers.

I'd suggest adding something like this to the TLS 1.2 configuration on your demo server:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

WilcoVertegaal commented 3 years ago

Are we sure that this is related at all to the Jellyfin client? I tried connecting to the demo server through the LG built-in web browser, and it didn't connect. Nothing happens. When I pointed it to http://demo.jellyfin.org/stable it was routed correctly to https://demo.jellyfin.org/stable - so it's not a routing issue of any kind. Seems to line up with @mono-of-pg 's findings.

anthonylavado commented 3 years ago

There is a separate private server that we provide to companies like LG and Roku in order to test clients. We have selectively enabled HTTP without a redirect to HTTPS on that server.

WilcoVertegaal commented 3 years ago

Sorry, that was not the point I was trying to make. I was trying to say that LG televisions seem to have trouble connecting to the demo server via https in general, either through the Jellyfin client or through the built-in web browser. Supporting the observation by others that it is probably a Let's Encrypt and/or TLS version specific issue that can't be solved in the Jellyfin weOS client software, or even in the Jellyfin server software at all.

anthonylavado commented 3 years ago

@WilcoVertegaal Ah yes, I agree. All good then :-)

We'll try @mono-of-pg's findings soon to be sure. If anything, it helps inform our documentation.

mono-of-pg commented 3 years ago

Assuming there's a reverse proxy in front of your demo server this could easily be changed in its configuration.

It even might work with an off-site proxy, so someone with an LG TV could simply set up a local reverse proxy pointing to the demo server without having to change anything on the server at all.

anthonylavado commented 3 years ago

@mono-of-pg There is a reverse proxy, but the item is a bit lower on the list since we're forcing the HTTP option for passing certification at least. I'll pass this along to my fellow Core Team member who set that up.

mono-of-pg commented 3 years ago

I fully understand we're looking for a quick fix here and that there are more important things to spend time on. That's why I'm suggesting to test this because if it really turns out to be that way then it's no jellyfin issue at all.

@WilcoVertegaal can you do some tests with your LG TV if I provide a reverse proxy?

WilcoVertegaal commented 3 years ago

@mono-of-pg I have a personal Jellyfin server behind a reverse proxy myself, Apache, with a Let's Encrypt certificate. Not sure if I know how to reconfigure it properly though :-) If you could either give me some directions, or a correctly configured reverse proxy, I would be happy to test it.

mono-of-pg commented 3 years ago

Very nice @WilcoVertegaal. I assume the connection through your proxy fails right now.

Test your ssl configuration with SSLLabs or testssl.sh if you want to run that from your computer.
Look for "Cipher Suites" and save that for reference.

Edit your Apache reverse proxy configuration to resemble this:

SSLProtocol -all +TLSv1.2 +TLSv1.3
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP
SSLHonorCipherOrder on

This should give you a rating of "A" on SSLLabs and still provide compatibility for older clients/browsers.

Restart Apache and give it a try...

WilcoVertegaal commented 3 years ago

Erm .. As it turns out, connecting to my own server is not failing at all ... SSLLabs cipher suites gives me a whole sh*tload of suites available, then the handshake simulation succeeded for all clients except "Yahoo Slurp Jan 2015".

Then I did the same for https://demo.jellyfin.org - cipher suites are much more limited, handshakes fail for like 40% of the clients.

What I learn from this: it's not Let's Encrypt, but it might very well be a missing SSL cipher suite.

Would you agree @mono-of-pg ?

mono-of-pg commented 3 years ago

Yes, that's exactly what I was talking about...

Now try to point your proxy to the jellyfin demo server and let it terminate SSL for your TV. LG-TV -> Proxy -> Jellyfin-Demo-Server

If that actually works it might be a good indication that this is where the problem really sits.

WilcoVertegaal commented 3 years ago

I can't seem to get it to work, it even fails when I open the website in my browser:

"Proxy Error The proxy server could not handle the request

Reason: Error during SSL Handshake with remote server"

Log file says this:

[Tue May 11 22:27:42.124526 2021] [proxy:error] [pid 4153024:tid 140396758083328] (20014)Internal error (specific information not available): [client 80.112.104.16:55422] AH01084: pass request body failed to [2604:a880:cad:d0::cf3:f001]:443 (demo.jellyfin.org)
[Tue May 11 22:27:42.124623 2021] [proxy:error] [pid 4153024:tid 140396758083328] [client 80.112.104.16:55422] AH00898: Error during SSL Handshake with remote server returned by /
[Tue May 11 22:27:42.124639 2021] [proxy_http:error] [pid 4153024:tid 140396758083328] [client 80.112.104.16:55422] AH01097: pass request body failed to [2604:a880:cad:d0::cf3:f001]:443 (demo.jellyfin.org) from 80.112.104.16 ()

mono-of-pg commented 3 years ago

try this:

SSLProxyVerify none 
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

mono-of-pg commented 3 years ago

@WilcoVertegaal another option would be to simulate the demo servers behavior regarding the ciphers. I'll try to create a configuration tomorrow.

WilcoVertegaal commented 3 years ago

@mono-of-pg I think I'm a little out of my league here. The handshake is correct now, and the initial connection seems fine. But then the opening page forwards itself to ./web/index.html and somehow that ends up as the following URL:

https://jellyfin.wilcovertegaal.nl/stable/web/index.html

So https://jellyfin.wilcovertegaal.nl/ is mapped correctly to https://demo.jellyfin.org/stable/ but forwarding that to https://demo.jellyfin.org/stable/web/index.html ends up as https://jellyfin.wilcovertegaal.nl/stable/web/index.html in the browser address bar.

Tried to solve it with a RewriteRule, but no dice. Sorry.

mono-of-pg commented 3 years ago

@WilcoVertegaal post your config here. Might be an issue with ProxyPass and ProxyPassReverse like not ending with a /

WilcoVertegaal commented 3 years ago

@mono-of-pg this is my config:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    Protocols h2 http/1.1
    ServerName jellyfin.wilcovertegaal.nl
    DocumentRoot /var/www/html/jellyfin/public_html

#    ProxyPreserveHost On
#    ProxyPass "/socket" "ws://127.0.0.1:8096/socket"
#    ProxyPassReverse "/socket" "ws://127.0.0.1:8096/socket"
#    ProxyPass "/" "http://127.0.0.1:8096/"
#    ProxyPassReverse "/" "http://127.0.0.1:8096/"
    SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
    ProxyPass "/socket" "ws://demo.jellyfin.org/stable/socket"
    ProxyPassReverse "/socket" "ws://demo.jellyfin.org/stable/socket"
    ProxyPass "/" "https://demo.jellyfin.org/stable/"
    ProxyPassReverse "/" "https://demo.jellyfin.org/stable/"

SSLCertificateFile /etc/letsencrypt/live/jellyfin.wilcovertegaal.nl/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/jellyfin.wilcovertegaal.nl/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

mono-of-pg commented 3 years ago

Try this @WilcoVertegaal:

        ProxyPass        /stable/socket   ws://demo.jellyfin.org/stable/socket
        ProxyPassReverse /stable/socket   ws://demo.jellyfin.org/stable/socket
        ProxyPassReverse /   https://demo.jellyfin.org/
        ProxyPass        /   https://demo.jellyfin.org/
        ProxyPassReverse /   https://demo.jellyfin.org/

You cannot reach 8096 on the demo server and also it's redirecting to /stable so you have to adapt.

WilcoVertegaal commented 3 years ago

@mono-of-pg nailed it 🥇

With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".

mono-of-pg commented 3 years ago

@mono-of-pg nailed it 🥇

With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".

That's awesome 👍

Now @WilcoVertegaal try this in your config:

SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!AESCCM

It should make your proxy look more like the demo server and offer only the ciphers mentioned above.

If that fails we can be pretty sure we found the root cause of this issue.

WilcoVertegaal commented 3 years ago

That's great - I now have A+ score on SSLLabs :-S

Cipher suites as reported by SSLLabs:

TLS_AES_256_GCM_SHA384 (0x1302)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
-- | --
TLS_CHACHA20_POLY1305_SHA256 (0x1303)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_AES_128_GCM_SHA256 (0x1301)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 128
# TLS 1.2                                                 (suites in server-preferred order)
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 128
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)                                                                                    ECDH x25519 (eq. 3072 bits RSA)   FS | 256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e)                                                                                                                              DH 2048 bits   FS | 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)                                                               ECDH x25519 (eq. 3072 bits RSA)   FS                                                       WEAK | 256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)                                                                                                         DH 2048 bits   FS                                                                   WEAK | 128
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39)                                                                                                         DH 2048 bits   FS                                                                   WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)                                                                                       WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)                                                                                       WEAK | 256

All handshake simulations passed.

Also the demo server still connects via LG TV.

mono-of-pg commented 3 years ago

While an A+ rating looks nice it's not what we're looking for in this case :-D

Your mod_ssl version seems to be a little different from mine so we need to fiddle a bit with the settings to just allow the same ciphers as the demo server does.

WilcoVertegaal commented 3 years ago

I'm on Ubuntu 20.04 with Apache 2.4.41, mod_ssl the same version info and OpenSSL 1.1.1f.

mono-of-pg commented 3 years ago

Apache 2.4.37 and OpenSSL 1.1.1g here. Anyway, we just need to get that cipher string right so we can be sure to compare the right things. openssl should return only ECDHE-RSA-AES256-GCM-SHA384 and DHE-RSA-AES256-GCM-SHA384 like it does on my machine:

[root@pg-dmzweb01 conf.d]# openssl ciphers -v 'ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384' | grep TLSv1.2
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
[root@pg-dmzweb01 conf.d]#

edit: OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

WilcoVertegaal commented 3 years ago

OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

Correct, mine is the same.

mono-of-pg commented 3 years ago

OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked

Correct, mine is the same.

So why does your mod_ssl do something different here?

Try with:

SSLProtocol -all +TLSv1.2
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384

Is this a VirtualHost or your serverwide ssl config?

mono-of-pg commented 3 years ago

Anyway, @anthonylavado as you can read above this issue very likely seems to be connected with the ciphers (not) offered by your reverse proxy. Thanks to @WilcoVertegaal for testing!

WilcoVertegaal commented 3 years ago

Is this a VirtualHost or your serverwide ssl config?

Yes.

Try this:

Still no dice. Sorry @mono-of-pg - I have zero expertise in this area whatsoever.

Thanks to @WilcoVertegaal for testing!

No problem, very happy to be able to contribute! (Next on the list: a VIDAA/Hisense client ... )

kupje commented 3 years ago

The root certificate "DST Root CA X3", which was used for cross-signing the Let's Encrypt R3 CA, expires on September 30 this year: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ From that point, the ISRG Root X1 is the only available trust path for Let's Encrypt certificates. It wasn't fully supported by major platforms until 2018: https://letsencrypt.org/2018/08/06/trusted-by-all-major-root-programs.html

Is this going to be a potential issue with older versions of webOS? I don't know LG's testing requirements, but maybe they dictate that the app must work on a device which has received zero updates (including the CA bundle) in X years? It would be a shame to see a certification attempt go to waste because they happen to test on some ancient device after that date and see a trust issue... Unfortunately I don't have a working dev environment for testing right now, but maybe it's possible to set up an emulator on a firewalled network which can only access the public demo server, install the oldest supported version of webOS, set its system date beyond Sep 30 and attempt to connect?

There's also Buypass, a Norwegian CA whose root certificate has been trusted by major browsers since 2010. They fully support the ACME protocol and can be used with clients such as Certbot: https://www.buypass.com/ssl/products/acme It might be a solution if the new LE trust path turns out to be an issue with legacy devices.

mono-of-pg commented 3 years ago

While this might be an issue after september I think it's pretty obvious that the findings of me and @WilcoVertegaal are the root cause of the problem. Has this been looked at by a dev in the last two months?

anthonylavado commented 3 years ago

As I mentioned before, while I appreciate all of the testing, this in particular isn't the highest for the release issues because we're just using a workaround for the time being.

Basically, for all of our vendor accounts (including Google Play, Apple App Store, etc) we have a private demo server that we provide for them to test with. It's got mostly similar content to the public demo server (just a few items removed for rating reasons). On this private server, we've given the LG Testing IP range access to connect without HTTPS, solely for testing purposes.

We will eventually want to apply these cipher option fixes here and on the publicly accessible server. We'll also use this info to make sure our Reverse Proxy guides have a notice and are up to date once the app releases, so that public users know what to expect. I'll keep this issue open for tracking purposes, but I would deem it no longer "release critical".

oddstr13 commented 3 years ago

The exact error reported is net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH

ares-device-info

modelName : 49UH603V-ZE
sdkVersion : 3.3.0
firmwareVersion : 05.31.00
boardType : K2L_DVB_EU

Console output

Navigating 1...
http://demo.jellyfin.org false
GET https://demo.jellyfin.org/System/Info/Public net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Failure: Object {error: true}
Could not connect to server...
Failure: Object {error: 0}
Could not connect to server...

mono-of-pg commented 3 years ago

Thanks for your update @anthonylavado. As long as it's passing the certification your workaround will do just fine. Very few will be facing this issue anyway, so having it in the documentation will probably be good enough.

kxzaon commented 2 years ago

Guys, the 30th of september is here ! The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable Do you have a workaround beside obtain a buypass certificate ???

oddstr13 commented 2 years ago

Guys, the 30th of september is here ! The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable Do you have a workaround beside obtain a buypass certificate ???

Use http over LAN?

nhanma154 commented 2 years ago

Can I get the ipk file for webos 3.0? I tried compiling but stuck at the connection interface.

trescatorce commented 2 years ago

...

Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system? Thank you!

WilcoVertegaal commented 2 years ago

Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system?

So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.

trescatorce commented 2 years ago

So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.

Thank you, I just send a message to the VIDAA support. Have a nice day!

oddstr13 commented 2 years ago

Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system?

So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.

The correct location for such a request would be https://features.jellyfin.org/, so please create a feature request there. Please include any appropriate information and links.

From a quick look at their website, it looks like the Jellyfin team would have to communicate with VIDAA in an official capacity to get access to documentation and development resources, so there is no reason for you to contact them directly about a Jellyfin app, that might complicate things for us in the future.

We're still working on getting the LG webOS and Samsung Tizen apps into the stores, so it is unlikely VIDAA will get much priority before that is done.

jellyfin / jellyfin-webos

Unable to Connect [webOS 3.0, 3.5, more?] #43

ares-device-info

Console output