search

nbr23 / youtube-dl-server

Web / REST interface for downloading youtube videos onto a server.
MIT License
241 stars 34 forks source link

[BUG] Invalid HTTP Request #31

Closed angrycuban13 closed 3 years ago

angrycuban13 commented 3 years ago

When trying to access the Web UI from https://domain.com/youtube-dl I get a 502 error and see the following in the logs

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
    event = self.conn.next_event()
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 443, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/usr/local/lib/python3.9/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
    raise self
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 425, in next_event
    event = self._extract_next_receive_event()
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/usr/local/lib/python3.9/site-packages/h11/_readers.py", line 75, in maybe_read_from_IDLE_client
    return Request(
  File "/usr/local/lib/python3.9/site-packages/h11/_events.py", line 71, in __init__
    self._validate()
  File "/usr/local/lib/python3.9/site-packages/h11/_events.py", line 142, in _validate
    raise LocalProtocolError("Found multiple Host: headers")
h11._util.RemoteProtocolError: Found multiple Host: headers

I'm running NGINX with LetsEncrypt and my main domain is behind a CF proxy and this had never been an issue.

This is my nginx.conf

## Version 2020/09/01 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/nginx.conf

user abc;
worker_processes 4;
pid /run/nginx.pid;
include /etc/nginx/modules/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    variables_hash_max_size 2048;
    large_client_header_buffers 4 16k;
    client_max_body_size 0;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    ##
    # Logging Settings
    ##
    access_log /config/log/nginx/access.log;
    error_log /config/log/nginx/error.log;
    ##
    # Gzip Settings
    ##
    gzip on;
    gzip_disable "msie6";
    ##
    # CF Real IP
    ##
    include cf_real-ip.conf;
    real_ip_header X-Forwarded-For;
    ##
    # Theme Park
    ##
    map $host $theme {
    default organizr-dark;
}
    ##
    # WebSocket proxying
    ##
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    ##
    # Virtual Host Configs
    ##
    include /etc/nginx/conf.d/*.conf;
    include /config/nginx/site-confs/*;
    lua_load_resty_core off;
}
daemon off;

This is my proxy.conf

## Version 2020/09/01 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/proxy.conf

client_body_buffer_size 128k;

#Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Advanced Proxy Config
send_timeout 5m;
proxy_read_timeout 240;
proxy_send_timeout 240;
proxy_connect_timeout 240;

# TLS 1.3 early data
proxy_set_header Early-Data $ssl_early_data;

# Basic Proxy Config
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_redirect  http://  $scheme://;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
#proxy_cookie_path / "/; HTTPOnly; Secure"; # enable at your own risk, may break certain apps
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 1024;

## Plex Login Error - API Connection Failed Fix?
proxy_buffer_size 128k;
proxy_busy_buffers_size 256k;
proxy_buffers 4 256k;

This is my ssl.conf

## Version 2020/06/17 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/ssl.conf

### Mozilla Recommendations
# generated 2020-06-17, Mozilla Guideline v5.4, nginx 1.18.0-r0, OpenSSL 1.1.1g-r0, intermediate configuration
# https://ssl-config.mozilla.org/#server=nginx&version=1.18.0-r0&config=intermediate&openssl=1.1.1g-r0&guideline=5.4

ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
ssl_session_tickets off;

# intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;

### Linuxserver.io Defaults

# Certificates
ssl_certificate /config/keys/letsencrypt/fullchain.pem;
ssl_certificate_key /config/keys/letsencrypt/privkey.pem;
# verify chain of trust of OCSP response using Root CA and Intermediate certs
ssl_trusted_certificate /config/keys/letsencrypt/fullchain.pem;

# Diffie-Hellman Parameters
ssl_dhparam /config/nginx/dhparams.pem;

# Resolver
resolver 127.0.0.11 valid=30s; # Docker DNS Server

# Enable TLS 1.3 early data
ssl_early_data on;

# HSTS, remove # from the line below to enable HSTS
add_header Strict-Transport-Security "max-age=15770000; includeSubDomains; preload" always;

# Optional additional headers
add_header Content-Security-Policy "upgrade-insecure-requests";
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-UA-Compatible "IE=Edge" always;
add_header Cache-Control "no-transform" always;
add_header Referrer-Policy "same-origin" always;
add_header Content-Security-Policy "frame-ancestors https://domain.xyz";
angrycuban13 commented 3 years ago

@nbr23 👋

blackerking commented 3 years ago

Same Error for me. I've made the sample.config for https://github.com/linuxserver/docker-swag

blackerking commented 3 years ago

@nbr23 It seems you have to enable proxyheaders and allowed IPs. youtube-dl-server.py -> forwarded-allow-ips='*', proxy-headers='true',

nbr23 commented 3 years ago

Thanks @blackerking ‼ @angrycuban13 sorry for dragging my feet on this one ;) Try https://github.com/nbr23/youtube-dl-server/commit/8cc9d7faefab878b31406a42a94e2eff05e92cdb, editing the config.yml to set the forwarded allow ips to '*' as @blackerking mentioned and let us know!

Thanks!

angrycuban13 commented 3 years ago

@nbr23 hehe I don't blame you, I know I threw a bunch of stuff at you when I opened the issue.

However, still getting errors:

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
    event = self.conn.next_event()
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 443, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/usr/local/lib/python3.9/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
    raise self
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 425, in next_event
    event = self._extract_next_receive_event()
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/usr/local/lib/python3.9/site-packages/h11/_readers.py", line 75, in maybe_read_from_IDLE_client
    return Request(
  File "/usr/local/lib/python3.9/site-packages/h11/_events.py", line 71, in __init__
    self._validate()
  File "/usr/local/lib/python3.9/site-packages/h11/_events.py", line 142, in _validate
    raise LocalProtocolError("Found multiple Host: headers")
h11._util.RemoteProtocolError: Found multiple Host: headers

This is my config

ydl_server:   # youtube-dl-server specific settings
  port: 8080      # Port youtube-dl-server should listen on
  host: 0.0.0.0   # IP youtube-dl-server should bind to
  debug: True    # Enable/Disable debug mode
  metadata_db_path: '/youtube-dl/.ydl-metadata.db' # Path to metadata DB
  output_playlist: '/storage/downloads/youtube-dl/playlists/%(playlist_title)s/%(title)s [%(id)s].%(ext)s' # Playlist output directory
  update_poll_delay_min: 1440 # Automatically check for updates every 24h
  forwarded_allow_ips: '*'
  proxy_headers: 'True'

ydl_options:  # youtube-dl options
  output: '/storage/downloads/youtube-dl/%(title)s [%(id)s].%(ext)s' # output directory
  cache-dir: '/youtube-dl/.cache' # youtube-dl cache directory
  ignore-errors: True # instruct youtube-dl to skip errors
  age-limit: 6 # minimal age requirement / parental control setting
angrycuban13 commented 3 years ago

Okay I deleted my config.yml and let it pull a fresh one and I'm still having issues :/

I tried using double quotes just in case but no dice.

nbr23 commented 3 years ago

:( alright… Do you have a full stack config for your setup (in a docker-compose or something) so I can set the same thing up here to do some testing?

angrycuban13 commented 3 years ago

This is my compose

  youtube-dl:
    container_name: youtube-dl
    environment:
      - PGID=${PGID}
      - PUID=${PUID}
      - TZ=${TZ}
      - YDL_CONFIG_PATH=/youtube-dl/config.yml
    hostname: ${DOCKERHOSTNAME}
    image: nbr23/youtube-dl-server
    logging: *default-logging
    ports:
      - 8083:8080
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${DOCKERCONFDIR}/youtube-dl:/youtube-dl
      - ${DOCKERSTORAGEDIR}:/storage
nbr23 commented 3 years ago

Thx, and where in your nginx config do you serve the youtube-dl content ?

angrycuban13 commented 3 years ago

I run a container called SWAG which has LE, NGINX and F2B bundled in.

This is what my youtube-dl subdirectory config looks like:

# Works with this youtube-dl Fork: https://github.com/nbr23/youtube-dl-server

location /youtube-dl {
    return 301 $scheme://$host/youtube-dl/;
}

location ^~ /youtube-dl/ {
    # enable the next two lines for http auth
    #auth_basic "Restricted";
    #auth_basic_user_file /config/nginx/.htpasswd;

    # enable the next two lines for ldap auth, also customize and enable ldap.conf in the default conf
    auth_request /auth-1;
    #error_page 401 =200 /login;

    include /config/nginx/proxy.conf;
    resolver 127.0.0.11 valid=30s;
    set $upstream_app  youtube-dl;
    set $upstream_port 8080;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    proxy_redirect  off;
    proxy_set_header Referer '';
    proxy_set_header Host $upstream_app:8080;
    rewrite /youtube-dl(.*) $1 break;
}

This is what the proxy.conf which is included in all my reverse proxies:

## Version 2020/09/01 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/proxy.conf

client_body_buffer_size 128k;

#Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Advanced Proxy Config
send_timeout 5m;
proxy_read_timeout 240;
proxy_send_timeout 240;
proxy_connect_timeout 240;

# TLS 1.3 early data
proxy_set_header Early-Data $ssl_early_data;

# Basic Proxy Config
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_redirect  http://  $scheme://;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
#proxy_cookie_path / "/; HTTPOnly; Secure"; # enable at your own risk, may break certain apps
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 1024;

## Plex Login Error - API Connection Failed Fix?
proxy_buffer_size 128k;
proxy_busy_buffers_size 256k;
proxy_buffers 4 256k;

This is my ssl.conf

## Version 2020/06/17 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/ssl.conf

### Mozilla Recommendations
# generated 2020-06-17, Mozilla Guideline v5.4, nginx 1.18.0-r0, OpenSSL 1.1.1g-r0, intermediate configuration
# https://ssl-config.mozilla.org/#server=nginx&version=1.18.0-r0&config=intermediate&openssl=1.1.1g-r0&guideline=5.4

ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
ssl_session_tickets off;

# intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;

### Linuxserver.io Defaults

# Certificates
ssl_certificate /config/keys/letsencrypt/fullchain.pem;
ssl_certificate_key /config/keys/letsencrypt/privkey.pem;
# verify chain of trust of OCSP response using Root CA and Intermediate certs
ssl_trusted_certificate /config/keys/letsencrypt/fullchain.pem;

# Diffie-Hellman Parameters
ssl_dhparam /config/nginx/dhparams.pem;

# Resolver
resolver 127.0.0.11 valid=30s; # Docker DNS Server

# Enable TLS 1.3 early data
ssl_early_data on;

# HSTS, remove # from the line below to enable HSTS
add_header Strict-Transport-Security "max-age=15770000; includeSubDomains; preload" always;

# Optional additional headers
add_header Content-Security-Policy "upgrade-insecure-requests";
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-UA-Compatible "IE=Edge" always;
add_header Cache-Control "no-transform" always;
add_header Referrer-Policy "same-origin" always;
add_header Content-Security-Policy "frame-ancestors https://domain.com";

Let me know if you need anything else!

nbr23 commented 3 years ago

Ok thanks, I'm able to get the error on my end :) I will check and report back, sorry for the wait!

nbr23 commented 3 years ago

Alright, this is weird, but it comes from the double proxy_set_header Host X; (one in your proxy.conf one in your youtube-dl conf).

It doesn't make sense to me why nginx would actually take both into account and pass 2 Host headers, but that is the case. From the doc, I do not understand this as being the behavior: Allows redefining or appending fields to the request header passed to the proxied server. The value can contain text, variables, and their combinations. These directives are inherited from the previous configuration level if and only if there are no proxy_set_header directives defined on the current level. But that is clearly what we are seeing, as this tcpdump capture shows:

image

I will keep looking into this behavior, but in the meantime, a quick fix if that's doable would be to get rid of one of the two instances of the proxy_set_header Host ...

angrycuban13 commented 3 years ago

I'll remove the one within the actual reverse proxy block as it is already defined somewhere else and report back!

angrycuban13 commented 3 years ago

Commenting this line out fixed it!

    proxy_set_header Host $upstream_app:8080;

@blackerking maybe have LSIO update the .conf for youtube-dl?

blackerking commented 3 years ago

I made a pull request: https://github.com/linuxserver/reverse-proxy-confs/pull/287

angrycuban13 commented 3 years ago

Let me see if I can get some traction on it

angrycuban13 commented 3 years ago

I think we can close this now @nbr23

@blackerking thanks for opening that PR.

nbr23 commented 3 years ago

Awesome, thank you both!