Open anthonylavado opened 3 years ago
I have multiple webOS devices and for me The Jellyfin app has been working on the devices that are running webOS 3.0 and 3.5. but I haven't updated the app in a while so I may be a bit behind the current release. I'll try to recompile it today and test it out.
Connecting to the public demo server from the app may not work just yet - there's a change I have to make to the reverse proxy config.
This change was already made for the internal demo server, so that won't be part of the issue affecting the client.
The error message originates from here https://github.com/jellyfin/jellyfin-webos/blob/25b774482b325c06ef2554c150ae3ea4c60214d3/org.jellyfin.webos/js/index.js#L320-L336
Looks like the isNaN
check is inverted, so this would most likely have been caused by a 4xx or 5xx status code returned by the server.
A testing release is available here: https://github.com/jellyfin/jellyfin-webos/issues/50
In #50, a user mentions that they are also unable to connect to the public demo server when using the 3.0 emulator, but it works fine under 5.0 (which is mainly what I tested with). We have a variety of CSP and other headers in place, so I'll have to review these to see what's stopping it.
In #50, @agrisci has figured out the issue. It looks like we need to enable older protocols for SSL for the internal testing server.
https://github.com/jellyfin/jellyfin-webos/issues/50#issuecomment-821836996
As mentioned in #46, I also cannot connect to the public demo server from my WebOS 6.0.5 TV. Not sure if there's a debug log available for a physical TV I can inspect to see if it is a TLS issue as well,...
So there's been a bunch of work behind the scenes. We've made sure that the internal and public demo servers are using TLS 1.2, as that is supported across all webOS versions.
From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.
We've currently made alternate arrangements to provide LG with an HTTP URL for testing, but obviously we'll still want to figure out how we can make this work for everyone.
From some more cursory searching on the internet, it seems that the app interface doesn't like Let's Encrypt certificates for some reason, even though the regular browser is okay with it. Since we use LE for all our sites, this is presenting an issue for us.
Have run into this specific issue before when using Emby's app with letsencrypt in the past. You're right to note that the tv browser works fine with letsencrypt, but there is a certificate problem that seems specific to apps on webos (this is on an LG C8 running what would've been late v4 or early v5 firmware at the time). I don't imagine it's fixable without cooperation from LG, but would be great if you could draw it to their attention. Maybe they'll do a firmware update!
I've dealt a lot with SSL/TLS related problems in the past. This specific one could simply be a TLS handshake mismatch.
Your demo server supports the following cipher suites for TLS 1.2:
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
https://webostv.developer.lge.com/discover/specifications/web-engine/ states that WebOS 3.x uses a Chromium 38 engine which just might not be capable of using those ciphers.
I'd suggest adding something like this to the TLS 1.2 configuration on your demo server:
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Are we sure that this is related at all to the Jellyfin client? I tried connecting to the demo server through the LG built-in web browser, and it didn't connect. Nothing happens. When I pointed it to http://demo.jellyfin.org/stable it was routed correctly to https://demo.jellyfin.org/stable - so it's not a routing issue of any kind. Seems to line up with @mono-of-pg 's findings.
There is a separate private server that we provide to companies like LG and Roku in order to test clients. We have selectively enabled HTTP without a redirect to HTTPS on that server.
Sorry, that was not the point I was trying to make. I was trying to say that LG televisions seem to have trouble connecting to the demo server via https in general, either through the Jellyfin client or through the built-in web browser. Supporting the observation by others that it is probably a Let's Encrypt and/or TLS version specific issue that can't be solved in the Jellyfin weOS client software, or even in the Jellyfin server software at all.
@WilcoVertegaal Ah yes, I agree. All good then :-)
We'll try @mono-of-pg's findings soon to be sure. If anything, it helps inform our documentation.
Assuming there's a reverse proxy in front of your demo server this could easily be changed in its configuration.
It even might work with an off-site proxy, so someone with an LG TV could simply set up a local reverse proxy pointing to the demo server without having to change anything on the server at all.
@mono-of-pg There is a reverse proxy, but the item is a bit lower on the list since we're forcing the HTTP option for passing certification at least. I'll pass this along to my fellow Core Team member who set that up.
I fully understand we're looking for a quick fix here and that there are more important things to spend time on. That's why I'm suggesting to test this because if it really turns out to be that way then it's no jellyfin issue at all.
@WilcoVertegaal can you do some tests with your LG TV if I provide a reverse proxy?
@mono-of-pg I have a personal Jellyfin server behind a reverse proxy myself, Apache, with a Let's Encrypt certificate. Not sure if I know how to reconfigure it properly though :-) If you could either give me some directions, or a correctly configured reverse proxy, I would be happy to test it.
Very nice @WilcoVertegaal. I assume the connection through your proxy fails right now.
Test your ssl configuration with SSLLabs or testssl.sh if you want to run that from your computer.
Look for "Cipher Suites" and save that for reference.
Edit your Apache reverse proxy configuration to resemble this:
SSLProtocol -all +TLSv1.2 +TLSv1.3
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA:EECDH:EDH+AESGCM:EDH:+3DES:ECDH+AESGCM:ECDH+AES:ECDH:AES:HIGH:MEDIUM:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP
SSLHonorCipherOrder on
This should give you a rating of "A" on SSLLabs and still provide compatibility for older clients/browsers.
Restart Apache and give it a try...
Erm .. As it turns out, connecting to my own server is not failing at all ... SSLLabs cipher suites gives me a whole sh*tload of suites available, then the handshake simulation succeeded for all clients except "Yahoo Slurp Jan 2015".
Then I did the same for https://demo.jellyfin.org - cipher suites are much more limited, handshakes fail for like 40% of the clients.
What I learn from this: it's not Let's Encrypt, but it might very well be a missing SSL cipher suite.
Would you agree @mono-of-pg ?
Yes, that's exactly what I was talking about...
Now try to point your proxy to the jellyfin demo server and let it terminate SSL for your TV.
LG-TV -> Proxy -> Jellyfin-Demo-Server
If that actually works it might be a good indication that this is where the problem really sits.
I can't seem to get it to work, it even fails when I open the website in my browser:
"Proxy Error The proxy server could not handle the request
Reason: Error during SSL Handshake with remote server"
Log file says this:
[Tue May 11 22:27:42.124526 2021] [proxy:error] [pid 4153024:tid 140396758083328] (20014)Internal error (specific information not available): [client 80.112.104.16:55422] AH01084: pass request body failed to [2604:a880:cad:d0::cf3:f001]:443 (demo.jellyfin.org)
[Tue May 11 22:27:42.124623 2021] [proxy:error] [pid 4153024:tid 140396758083328] [client 80.112.104.16:55422] AH00898: Error during SSL Handshake with remote server returned by /
[Tue May 11 22:27:42.124639 2021] [proxy_http:error] [pid 4153024:tid 140396758083328] [client 80.112.104.16:55422] AH01097: pass request body failed to [2604:a880:cad:d0::cf3:f001]:443 (demo.jellyfin.org) from 80.112.104.16 ()
try this:
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
@WilcoVertegaal another option would be to simulate the demo servers behavior regarding the ciphers. I'll try to create a configuration tomorrow.
@mono-of-pg I think I'm a little out of my league here. The handshake is correct now, and the initial connection seems fine. But then the opening page forwards itself to ./web/index.html and somehow that ends up as the following URL:
https://jellyfin.wilcovertegaal.nl/stable/web/index.html
So https://jellyfin.wilcovertegaal.nl/ is mapped correctly to https://demo.jellyfin.org/stable/ but forwarding that to https://demo.jellyfin.org/stable/web/index.html ends up as https://jellyfin.wilcovertegaal.nl/stable/web/index.html in the browser address bar.
Tried to solve it with a RewriteRule, but no dice. Sorry.
@WilcoVertegaal post your config here. Might be an issue with ProxyPass and ProxyPassReverse like not ending with a /
@mono-of-pg this is my config:
<IfModule mod_ssl.c>
<VirtualHost *:443>
Protocols h2 http/1.1
ServerName jellyfin.wilcovertegaal.nl
DocumentRoot /var/www/html/jellyfin/public_html
# ProxyPreserveHost On
# ProxyPass "/socket" "ws://127.0.0.1:8096/socket"
# ProxyPassReverse "/socket" "ws://127.0.0.1:8096/socket"
# ProxyPass "/" "http://127.0.0.1:8096/"
# ProxyPassReverse "/" "http://127.0.0.1:8096/"
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
ProxyPass "/socket" "ws://demo.jellyfin.org/stable/socket"
ProxyPassReverse "/socket" "ws://demo.jellyfin.org/stable/socket"
ProxyPass "/" "https://demo.jellyfin.org/stable/"
ProxyPassReverse "/" "https://demo.jellyfin.org/stable/"
SSLCertificateFile /etc/letsencrypt/live/jellyfin.wilcovertegaal.nl/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/jellyfin.wilcovertegaal.nl/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>
Try this @WilcoVertegaal:
ProxyPass /stable/socket ws://demo.jellyfin.org/stable/socket
ProxyPassReverse /stable/socket ws://demo.jellyfin.org/stable/socket
ProxyPassReverse / https://demo.jellyfin.org/
ProxyPass / https://demo.jellyfin.org/
ProxyPassReverse / https://demo.jellyfin.org/
You cannot reach 8096 on the demo server and also it's redirecting to /stable so you have to adapt.
@mono-of-pg nailed it 🥇
With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".
@mono-of-pg nailed it 🥇
With this configuration, my LG TV connects to the demo server and plays content without any problem. Connecting directly to https://demo.jellyfin.org/stable/ says something like "website not found".
That's awesome 👍
Now @WilcoVertegaal try this in your config:
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:!RC4:!CAMELLIA:!SEED:!aNULL:!MD5:!eNULL:!LOW:!EXP:!DSS:!PSK:!SRP:!AESCCM
It should make your proxy look more like the demo server and offer only the ciphers mentioned above.
If that fails we can be pretty sure we found the root cause of this issue.
That's great - I now have A+ score on SSLLabs :-S
Cipher suites as reported by SSLLabs:
TLS_AES_256_GCM_SHA384 (0x1302) ECDH x25519 (eq. 3072 bits RSA) FS | 256
-- | --
TLS_CHACHA20_POLY1305_SHA256 (0x1303) ECDH x25519 (eq. 3072 bits RSA) FS | 256
TLS_AES_128_GCM_SHA256 (0x1301) ECDH x25519 (eq. 3072 bits RSA) FS | 128
# TLS 1.2 (suites in server-preferred order)
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8) ECDH x25519 (eq. 3072 bits RSA) FS | 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ECDH x25519 (eq. 3072 bits RSA) FS | 128
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) ECDH x25519 (eq. 3072 bits RSA) FS | 256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 2048 bits FS | 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) ECDH x25519 (eq. 3072 bits RSA) FS WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) ECDH x25519 (eq. 3072 bits RSA) FS WEAK | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH x25519 (eq. 3072 bits RSA) FS WEAK | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) ECDH x25519 (eq. 3072 bits RSA) FS WEAK | 256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 2048 bits FS WEAK | 128
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39) DH 2048 bits FS WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35) WEAK | 256
All handshake simulations passed.
Also the demo server still connects via LG TV.
While an A+ rating looks nice it's not what we're looking for in this case :-D
Your mod_ssl version seems to be a little different from mine so we need to fiddle a bit with the settings to just allow the same ciphers as the demo server does.
I'm on Ubuntu 20.04 with Apache 2.4.41, mod_ssl the same version info and OpenSSL 1.1.1f.
Apache 2.4.37 and OpenSSL 1.1.1g here. Anyway, we just need to get that cipher string right so we can be sure to compare the right things. openssl should return only ECDHE-RSA-AES256-GCM-SHA384 and DHE-RSA-AES256-GCM-SHA384 like it does on my machine:
[root@pg-dmzweb01 conf.d]# openssl ciphers -v 'ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384' | grep TLSv1.2
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD
[root@pg-dmzweb01 conf.d]#
edit: OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked
OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked
Correct, mine is the same.
OpenSSL 1.1.1f on Ubuntu 20.04.2 LTS gives me the same result - just checked
Correct, mine is the same.
So why does your mod_ssl do something different here?
Try with:
SSLProtocol -all +TLSv1.2
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384
Is this a VirtualHost or your serverwide ssl config?
Anyway, @anthonylavado as you can read above this issue very likely seems to be connected with the ciphers (not) offered by your reverse proxy. Thanks to @WilcoVertegaal for testing!
Is this a VirtualHost or your serverwide ssl config?
Yes.
Try this:
Still no dice. Sorry @mono-of-pg - I have zero expertise in this area whatsoever.
Thanks to @WilcoVertegaal for testing!
No problem, very happy to be able to contribute! (Next on the list: a VIDAA/Hisense client ... )
The root certificate "DST Root CA X3", which was used for cross-signing the Let's Encrypt R3 CA, expires on September 30 this year: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ From that point, the ISRG Root X1 is the only available trust path for Let's Encrypt certificates. It wasn't fully supported by major platforms until 2018: https://letsencrypt.org/2018/08/06/trusted-by-all-major-root-programs.html
Is this going to be a potential issue with older versions of webOS? I don't know LG's testing requirements, but maybe they dictate that the app must work on a device which has received zero updates (including the CA bundle) in X years? It would be a shame to see a certification attempt go to waste because they happen to test on some ancient device after that date and see a trust issue... Unfortunately I don't have a working dev environment for testing right now, but maybe it's possible to set up an emulator on a firewalled network which can only access the public demo server, install the oldest supported version of webOS, set its system date beyond Sep 30 and attempt to connect?
There's also Buypass, a Norwegian CA whose root certificate has been trusted by major browsers since 2010. They fully support the ACME protocol and can be used with clients such as Certbot: https://www.buypass.com/ssl/products/acme It might be a solution if the new LE trust path turns out to be an issue with legacy devices.
While this might be an issue after september I think it's pretty obvious that the findings of me and @WilcoVertegaal are the root cause of the problem. Has this been looked at by a dev in the last two months?
As I mentioned before, while I appreciate all of the testing, this in particular isn't the highest for the release issues because we're just using a workaround for the time being.
Basically, for all of our vendor accounts (including Google Play, Apple App Store, etc) we have a private demo server that we provide for them to test with. It's got mostly similar content to the public demo server (just a few items removed for rating reasons). On this private server, we've given the LG Testing IP range access to connect without HTTPS, solely for testing purposes.
We will eventually want to apply these cipher option fixes here and on the publicly accessible server. We'll also use this info to make sure our Reverse Proxy guides have a notice and are up to date once the app releases, so that public users know what to expect. I'll keep this issue open for tracking purposes, but I would deem it no longer "release critical".
The exact error reported is net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH
modelName : 49UH603V-ZE
sdkVersion : 3.3.0
firmwareVersion : 05.31.00
boardType : K2L_DVB_EU
Navigating 1...
http://demo.jellyfin.org false
GET https://demo.jellyfin.org/System/Info/Public net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Failure: Object {error: true}
Could not connect to server...
Failure: Object {error: 0}
Could not connect to server...
Thanks for your update @anthonylavado. As long as it's passing the certification your workaround will do just fine. Very few will be facing this issue anyway, so having it in the documentation will probably be good enough.
Guys, the 30th of september is here ! The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable Do you have a workaround beside obtain a buypass certificate ???
Guys, the 30th of september is here ! The Jellyfin app on my rooted LG B8 (WebOS : 4.3.0-9604) refuse to connect to my "Letsencrypted" jellyfin server. Same behaviour with https://demo.jellyfin.org/stable Do you have a workaround beside obtain a buypass certificate ???
Use http over LAN?
Can I get the ipk file for webos 3.0? I tried compiling but stuck at the connection interface.
...
Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system? Thank you!
Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system?
So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.
So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.
Thank you, I just send a message to the VIDAA support. Have a nice day!
Hi, sorry if it bothers you, but do you think there is any chance to bring Jellyfin to the VIDAA system?
So far VIDAA have been refusing to answer my requests for documentation, so currently there is no chance. Feel free to request such a feature though. Not sure what the official Github way of submitting such a request would be. Probably not the WebOS repo.
The correct location for such a request would be https://features.jellyfin.org/, so please create a feature request there. Please include any appropriate information and links.
From a quick look at their website, it looks like the Jellyfin team would have to communicate with VIDAA in an official capacity to get access to documentation and development resources, so there is no reason for you to contact them directly about a Jellyfin app, that might complicate things for us in the future.
We're still working on getting the LG webOS and Samsung Tizen apps into the stores, so it is unlikely VIDAA will get much priority before that is done.
Issue During certification testing, devices running LG webOS 3.0 or 3.5 were unable to connect to the demo server.
Notes from Testing
Next Steps It is possible that this was an intermittent issue, but because of the failed result, LG stopped testing on webOS 3/3.5 devices at this point.
For internal (Jellyfin) testing:
For community help:
If you have a webOS 3/3.5 device, please let us know if you are able to test this. You can use the public demo server, https://demo.jellyfin.org/stable. The username is
demo
, with no password. If you need a compiled IPK to install, please let me know. Note that the public server is reset on the hour, every hour. It is recommended to avoid testing at that moment, as it will likely result in a connection error. This does not apply to our internal testing server.