search

runtipi / runtipi-appstore

Official Runtipi AppStore. Apps and updates live here!
https://runtipi.io/docs/apps-available
GNU General Public License v3.0
307 stars 255 forks source link

🐛 [Kasm] 400 Bad Request when exposing via RunTipi.io using Cloudflare tunnel #2394

Closed jasonpearce closed 7 months ago

jasonpearce commented 9 months ago

Store Application

Kasm Workspaces

App version

1.120.20221218

Description

Describe the bug When accessing a new Kasm installation on a RunTipi.io server via a Cloudflare tunnel, the browser response is:

400 Bad Request The plain HTTP request was sent to HTTPS port nginx

Expected behavior I'd like to be able to access https://kasm.example.com, running as a RunTipi.io app, via a Cloudflare tunnel remotely/externally.

Screenshots image

Workspaces Version Latest Ubuntu Server, RunTipi.io, and Kasm app

Workspaces Installation Method Physical Lenovo Tiny PC, Ubuntu Server 22.04, Docker, RunTipi.io, Cloudflare Tunnel, Kasm App via the RunTipi app store

Client Browser (please complete the following information):

Workspace Server Information (please provide the output of the following commands):

Additional context I can use Cloudflare tunnels to externally expose other apps installed from the RunTipi.io app store. I've done so for about eight apps. This issue occurs only within the Kasm app. The Kasm app works fine inside my home network via IP address and port. Thank you.

Steps to reproduce

To Reproduce Steps to reproduce the behavior:

  1. Build a new Ubuntu 22.04 server on a refubished PC at home behind your firewall
  2. Install RunTipi (https://runtipi.io/docs/getting-started/installation)
  3. Install app Hello World as a test baseline (https://runtipi.io/docs/apps-available)
  4. Expose the Hello World app using Cloudflare tunnels (https://runtipi.io/docs/guides/expose-apps-with-cloudflare-tunnels)
  5. Validate that https://helloworld.example.com works externally via the Cloudflare tunnel (it does)
  6. Install app Kasm Workspaces (https://runtipi.io/docs/apps-available)
  7. Validate that https://:port works locally (it does)
  8. Expose the Kasm app using Cloudflare tunnels (https://runtipi.io/docs/guides/expose-apps-with-cloudflare-tunnels)
  9. Validate that https://kasm.example.com works externally via the Cloudflare tunnel (error 400 bad request)

App logs

I'm not finding a way to view logs in the Tipi dashboard.

Browser

Browser Agnostic

Browser logs

No response

User-Config changes

No changes.

Other

Posted this on the Kasm github: https://github.com/kasmtech/workspaces-issues/issues/509

One response was...

The error is indicating that the system is trying to access (presumably the kasm server) over HTTP instead of HTTPS , so somewhere in your stack you need to ensure your reverse proxy style system (Cloudflare Tunnel) is proxying to Kasm via HTTPS and not HTTP. I'm not familiar with Tipi so check there as well

Please confirm the following

steveiliop56 commented 9 months ago

Hello @jasonpearce,

Can you please make sure this is set to https?

image
jasonpearce commented 9 months ago

Thank you for responding @steveiliop56. It is. Here are my Public Hostname settings in Cloudflare (with some information obscured):

Basic Information Public hostname: (https://kasm.example.com/) Path: * Service: https://192.168.xxx.xxx Origin configurations:

steveiliop56 commented 9 months ago

Hello @jasonpearce,

Can you confirm that you can access kasm normally via the ip address and port?

jasonpearce commented 9 months ago

Yes. Locally I can access Kasm via https://:port.

I can also confirm that four other Cloudflare access tunnels to other Runtipi apps work internally and externally. The only difference is that Kasm is the only one using the Cloudflare origin configuration of "http2Origin" in addition to the "noTLSVerify." All others use only "noTLSVerify." If I remove "http2Origin" from the Cloudflare setting for Kasm, I do not observe a difference.

This weekend, I'll attempt to capture and provide ample screen shots if that would be helpful. Again, thank you for assisting.

jasonpearce commented 9 months ago

Here are some screenshots I said I would provide. I hope they are helpful.

Cloudflare settings

cloudflare healthy tunnel

cloudflare public hostnames

cloudflare hello world hostname settings

cloudflare kasm hostname settings

Tipi on local network

tipi login to your account

tipi my apps

Working Hello World Settings for LAN and WAN

hello world open options

hello world via local IP address

hello world via cloudflare public hostname and tor

Working Kasm Settings for LAN

kasm open options

kasm via local IP address but adding https

Not Working Kasm Setting for WAN

kasm via cloudflare public hostname and tor

Closing

Please let me know what additional information I can provide to help you or others identify if this is only an issue on my end or if this is a bug. I did use Tipi to uninstall and reinstall the Kasm app (same results).

meienberger commented 8 months ago

It seems there is an issue on how the tipi reverse-proxy operates and forwards the request to kasm. I cannot find anything useful in the linuxserver environments that could help with it. Probably some tweaking with the nginx headers could help. I will do some testing

jasonpearce commented 8 months ago

This weekend, I upgraded from Tipi v2.5.x to v3.0.3. After doing so, Kasm Workspaces 1.120.20221218 no longer worked via the local IP address. A few other apps also stopped working. I installed some new apps, and some of them would work, others would not.

Worked by local IP before and after upgrade: Cloudflared, Hello World, IT-Tools, Jellyfin, Linkwarden, SearXNG, Stirling-PDF, ViewTube

Worked by local IP before upgrade, but 400 Bad Request after upgrade: Kasm Workspaces

Installed after upgrade and works by local IP: Nextcloud, Uptime Kuma

Installed after upgrade, but 400 Bad Request: Netboot.xyz, Tailscale

My priority is to have Kasm Workspaces working via local IP and via a Cloudflared tunnel. To provide you some more information, I built a new virtual machine to do some testing.

Ubuntu 22.04.04 LTS Desktop: Built a new virtual machine. Fully patched it. Rebooted.

Runtipi v3.0.3: Installed Tipi v 3.0.3. Can always access via local IP. This always worked.

Hello World vLatest: Installed Hello World. This always worked via local IP.

Rebooted.

Kasm Workspaces v1.120.20221218: Installed Kasm Workspaces. In Kasm, installed some virtual browsers (Brave, Chromium, Edge, Firefox, Tor). Tested them all. They all worked via local IP.

Rebooted.

Kasm Workspaces: After reboot, I tested them all again via Local IP. Everything worked.

Cloudflared v2024.2.1: Installed Cloudflared. Went to cloudflare.com and successfully setup a new Connector to my new VM. Both cloudflare.com and Cloudflared looked good (connected, healthy).

Rebooted.

Kasm 400 Bad Request: After installing Cloudflared and rebooting, Kasm stopped working via an IP address. All attempts resulted in error "400 Bad Request". My attempts:

Summary: With Tipi v3.0.3, once Cloudflared is installed I can no longer get Kasm Workspaces to work via a local IP address -- even after Stopping or Uninstalling Cloudflared. Only after reinstalling Kasm Workspaces can I get Kasm to work again, but it will work only until it is stopped or the server reboots. After 1 stop or 1 reboot, Kasm will no longer work by local IP again. Some other apps also appear to have issues via the local IP address if Cloudflared is/was installed (Tailscale, Netboot.xyz); but I did not test them as extensively.

I hope some of this information helps you troubleshoot and is not wasting your time.

jasonpearce commented 8 months ago

I'm not experienced enough to help solve this problem. But in an attempt to help, I asked a LLM to review the code on the Kasm repositories and suggest some changes that might fix the issue. Here's what one AI proposed. I understand this may very well be misleading or incorrect.

LLM AI:

After reviewing the Kasm repositories, I found a potential cause for the issue. The problem might be related to the NGINX reverse proxy configuration in the Kasm Workspaces source code. Specifically, the nginx.conf file in the kasm-workspaces/src/kasm_deployments/nginx/ directory.

To fix the issue, you can try the following steps:

  1. Modify the nginx.conf file to include a new server block that listens on the HTTP protocol and redirects to HTTPS. This will ensure that all requests are using HTTPS, which might be the root cause of the 400 Bad Request error.

Add the following server block within the http block in the nginx.conf file:

server {
    listen 80;
    server_name kasm.example.com;
    return 301 https://$host$request_uri;
}
  1. Make sure that the nginx.conf file includes the http2 directive in the listen directive for the HTTPS server block. This ensures that the connection uses HTTP/2, which is required for Cloudflare tunnels.

Change the existing HTTPS server block to:

server {
    listen 443 ssl http2;
    server_name kasm.example.com;

    # ... (other configurations)
}
  1. After making these changes, rebuild the Docker image and restart the Kasm app to apply the modifications.

Please note that these changes are based on the assumption that the issue is related to HTTP/HTTPS configuration. If the problem persists, further investigation might be required to identify other potential causes.

If you need more assistance or have any questions, please let me know.

jasonpearce commented 7 months ago

After upgrading to...

My results are unchanged:

steveiliop56 commented 7 months ago

I might be able to solve it by fixing some labels. But if that doesn't work I unfortunately will have to disable the expose feature.

jasonpearce commented 7 months ago

Ok. I understand. I agree. If the bug/conflict with Cloudflared cannot be resolved, then removing the expose feature would be the best option to "resolve" this bug. Thank you.

steveiliop56 commented 7 months ago

So I unfortunately cannot solve the traefik issue so the best solution for now is to completely disable traefik on kasm because neither local domains nor exposed work (I get the same issue as you) https://github.com/runtipi/runtipi-appstore/pull/3048