NGINX issues since 0.17

auanasgheps commented 3 years ago

Version

Version: 0.17.2

Bug description

Since the update to 0.17, here's what looks like.

If I go to Space settings, it says I have 20 recipes (correct).

Something in the upgrade went wrong. What should I do? I can't access my recipes!

immagine

smilerz commented 3 years ago

@auanasgheps how is Tandoor setup (manual, docker, other)? can you share the logs? also if hit F12 the developer console might provide some additional context.

auanasgheps commented 3 years ago

@smilerz I'm using docker. Anyway I managed to do some troubleshooting and it's about the nginx config. If I go straight to the container works fine. I'll keep you posted

smilerz commented 3 years ago

check out #953 and see if that points you in the right direction.

MaxJa4 commented 3 years ago

@auanasgheps I had the same exact issue since 0.17.0... My nginx reverse proxy was the cause, I had to add the following lines inside the nginx site config (in the location region): proxy_set_header X-Forwarded-Proto $scheme; I also added proxy_set_header Host $Host;, but I don't know if that's necessary. This is partially mentioned in the FAQs

MaxJa4 commented 3 years ago

Seems like 0.17.0 got some new links which are not relative but absolute (just an assumption). proxy_set_header Host $Host; fixed the issue for some kind of worker which had problems with that. proxy_set_header X-Forwarded-Proto $scheme; fixed some 'mixed content' errors. All of that can be seen in the web browser console (F12).

smilerz commented 3 years ago

@vabene1111 does the included nginx config need to be updated?

auanasgheps commented 3 years ago

This is may config before it broke down (pre 0.17).

If I add proxy_set_header Host $Host; I get "Bad Request (400)" and the page doesn't load. In the past I've already fought with Recipes and Nginx. Here we go again.

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /ldaplogin;

        # enable for Authelia
        #include /config/nginx/authelia-location.conf;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app recipes-app;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    proxy_set_header X-Forwarded-Proto $scheme;
    }
}

MaxJa4 commented 3 years ago

Maybe proxy_set_header Host $http_host; works better like mentioned in the issue that @smilerz mentioned

auanasgheps commented 3 years ago

@MaxJa4 tried and I still get 400.

In the meantime here's the browser console immagine

smilerz commented 3 years ago

@auanasgheps after setting the host and scheme as mentioned above clear the cache on the page - some of the URLs are constructed from settings set in LOCAL_STORAGE. Maybe(?) those are not getting overridden correctly? I'd restart the docker as well to make sure that no information is cached anywhere.

MaxJa4 commented 3 years ago

@auanasgheps That's the exact same errors I got. Did you add both "proxy_set_header Host" and "proxy_set_header X-Forwarded-Proto"? If so, then maybe the caching is the issue here like smilerz said. After that, I'm out of ideas (other than restarting nginx once more just to be sure).

auanasgheps commented 3 years ago

I added both "proxy_set_header Host $Host" and "proxy_set_header X-Forwarded-Proto $scheme" Restarted nginx Browed to Recipes CTRL + F5 to clear cache

Bad Request (400)

vabene1111 commented 3 years ago

So first of all the included nginx config and all recommended/documented setups have been tested and were working.

The paths are not absolute but django has several build in ways that manage constructing paths, if a proxy setup in front of django does not pass trough the correct headers than django is not able to determine where it is actually running from and thus cannot tell the frontend components the correct routes to make request to.

From what i can see @auanasgheps you are getting requests blocked because django thinks that its running on http but it actually is served over https. This can happen if the proxy enforces https for the main requests (which it is able to do since it passes trough it) but it cannot rewrite the urls that django determines it needs to use for making api requests.

Please make sure you set proxy_set_header Host $http_host; (http_host not host as stated in the docs and issue #953) the other header proxy_set_header X-Forwarded-Proto $scheme; looks correct. If that does not help please remove all configuration entries from your nginx config that are not clearly mentioned in the installation guide and verify the problem still exists. If it does please post detailed information about your setup (folder locations, OS versions, all relevant config files env/nginx/etc.)

MaxJa4 commented 3 years ago

My nginx config (of the host system, as reverse proxy) looks like this:

server {
    if ($host = SUB.DOMAIN.TLD) {
        return 301 https://$host$request_uri;
    }

    server_name SUB.DOMAIN.TLD;
    listen 80;
    return 404;
}
server {
    server_name SUB.DOMAIN.TLD;
    listen 443 ssl;

    ssl_certificate /etc/letsencrypt/live/SUB.DOMAIN.TLD/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/SUB.DOMAIN.TLD/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

        location / {
                proxy_pass http://127.0.0.1:PORT;
                proxy_redirect http://127.0.0.1:PORT https://SUB.DOMAIN.TLD;
                proxy_set_header Host $Host;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}

Be sure to replace SUB.DOMAIN.TLD (6x) and PORT (2x) in all spots ofc. This is by far not a perfect config but it works for me. If you don't use certbot or letsencrypt, you might need to change some stuff.

auanasgheps commented 3 years ago

So I confirm that if I use proxy_set_header Host $http_host; I get Bad Request 400. Before I was using proxy_set_header Host $Host; just fine.

My setup is: OS Debian 10, Docker 20.10.9

Network flow is: nginx (swag lets encrypt) > recipes front-end. No dedicated nginx recipes container since I already have the main one that does the job for other containers. nginx/swag has access to the /media/recipes folder to serve media.

Here's the proxy config (that doesn't work)

server {
    listen 8080 ssl;
    listen [::]:8080 ssl;

    server_name recipes.*;

    include /config/nginx/ssl.conf;

    #redirect http to https on same port
    error_page 497 301 =307 https://$host:$server_port$request_uri;

    client_max_body_size 0;

    # serve media files
    location /media/recipes {
        alias /recipes-media/;
    }

    location / {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app recipes-app;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Host $http_host;
    #proxy_set_header Host $Host;
    }
}

Here's the relevant part of my .env file

# hosts the application can run under e.g. recipes.mydomain.com,cooking.mydomain.com,...
ALLOWED_HOSTS=recipes.mydomain.tld

# If mediafiles are stored at a different location uncomment and change accordingly
MEDIA_URL=https://recipes.mydomain.tld:8080/media/

CSRF_TRUSTED_ORIGINS=recipes.mydomain.tld

vabene1111 commented 3 years ago

Hmm this is very weird. Can you also post the logs of the nginx/swag and django ? The main thing i want to understand if the requests actually get trough to django or not.

I know that quite a few people had problems with swag before and it was usually related to headers but yours look fine.

auanasgheps commented 3 years ago

nginx log doesn't throw any error, here's the access.log

192.168.0.100 - - [19/Oct/2021:00:35:05 +0200] "GET /favicon.ico HTTP/2.0" 400 143 "https://recipes.mydomain.tld:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0"
192.168.0.100 - - [19/Oct/2021:00:35:06 +0200] "GET /service-worker.js HTTP/2.0" 400 143 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0"
192.168.0.100 - - [19/Oct/2021:00:35:35 +0200] "GET / HTTP/2.0" 400 143 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0"
192.168.0.100 - - [19/Oct/2021:00:35:35 +0200] "GET /favicon.ico HTTP/2.0" 400 143 "https://recipes.mydomain.tld:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0"

If I enable debug mode, here's what I get.

WTH? Looks like nginx is passing my domain twice (first hostname then hostname and port), therefore causing this error.

immagine

vabene1111 commented 3 years ago

ok so the fix for django would be to simple use ALLOWED_HOSTS=* which is not perfect but acceptable for a proxied server.

Other than that i am not sure which line causes this as i am not familiar with swag but might it be that the $upstream_app in proxy_pass $upstream_proto://$upstream_app:$upstream_port; contains the sub domain twice?

auanasgheps commented 3 years ago

ok so the fix for django would be to simple use ALLOWED_HOSTS=* which is not perfect but acceptable for a proxied server.

Yeah. I mean I should only access the frontend via the proxy. I've kept the frontend port bind in case something goes wrong, but it's just for LAN.

I'll test the change and report back.

Other than that i am not sure which line causes this as i am not familiar with swag but might it be that the $upstream_app in proxy_pass $upstream_proto://$upstream_app:$upstream_port; contains the sub domain twice?

It can't be otherwise I would not be able to connect at all. That's the part that actually does the proxy job.

auanasgheps commented 3 years ago

So I used the option ALLOWED_HOSTS=* but it still doesn't work. The error is always the same The domain name provided is not valid according to RFC 1034/1035. Possibly because the header can't be parsed and is something like recipes.mydomain.tld,recipes.mydomain.tld:8080

I'll try to understand why this is happening.

EDIT Okay, there's a /config/nginx/proxy.conf which by default sets the following parameters, so it's not required to specify again proxy_set_header Host $http_host;

# Proxy Header Settings
proxy_set_header Connection $connection_upgrade;
proxy_set_header Early-Data $ssl_early_data;                
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto https;       
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Real-IP $remote_addr;

So definitely needs to be removed from the recipes config. If I remove it I can "access" Recipes, but recipes don't load.

Is any of these parameters interfering with Django?

auanasgheps commented 3 years ago

With the "correct" config, my browser in the developer console says

Mixed Content: The page at 'https://recipes.mydomain.tld:8080/search/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://recipes.mydomain.tld/api/recipe/?query=&keywords_or=true&foods_or=true&books_or=true&internal=false&random=false&new=true&page=1&page_size=25&last_viewed=5'. This request has been blocked; the content must be served over HTTPS.

What the heck is this? The proxy is sending all sort of HTTPS requests/upgrades, there's no request from the client done in HTTP.

EDIT: I've forced several page updates and this error has disappeared, now I get a generic network error.

Here's django debug logs for a page load

172.18.0.5 - - [19/Oct/2021:11:47:30 +0200] "GET /search/ HTTP/1.1" 200 16711 "https://recipes.mydomain.tld:8080/accounts/login/?next=/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/themes/flatly.min.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/css/app.min.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/jquery-3.5.1.min.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/popper.min.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/bootstrap.min.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/css/select2.min.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/select2.min.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/css/select2-bootstrap.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/themes/select2-bootstrap-theme.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/fontawesome/fontawesome_all.min.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/django_js_reverse/reverse.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/vue/css/chunk-vendors.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/vue/js/chunk-vendors.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/vue/css/recipe_search_view.css HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/vue/js/recipe_search_view.js HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/popper.min.js.map HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/js/bootstrap.min.js.map HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:31 +0200] "GET /static/webfonts/fa-solid-900.woff2 HTTP/1.1" 304 0 "https://recipes.mydomain.tld:8080/static/fontawesome/fontawesome_all.min.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
172.18.0.5 - - [19/Oct/2021:11:47:32 +0200] "GET /manifest.json HTTP/1.1" 200 1117 "https://recipes.mydomain.tld:8080/search/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"

vabene1111 commented 3 years ago

Ok to from djangos side of things everything looks correct but in the requests you send i cannot see any of the frontend javascript files being requested.

The mixed content warning, even tough it is gone now, is understandable because if django thinks that it is running under http (which it might if some header breaks) it will render the html containing http links. Even if your proxy would upgrade them when they are executed the browser would think that its mixed content and thus block the requests / show errors.

One thing the django log shows for sure is that the connection refused response that you are seeing in your console comes somewhere from your proxy stack as the requests dont actually make its way to django but get canceled somewhere in front. You could try to visit the /api/xyz url manually and see if the connection refused happens there as well and try to debug which part of your proxy kills that request.

What i see is that your default config uses proxy_set_header Host $host; and not proxy_set_header Host $http_host;. I am not sure if this is a problem but i remember that we added this parameter for a reason, just not which reason.

auanasgheps commented 3 years ago

First of all, I see that proxy_set_header X-Forwarded-Proto $scheme; doesn't work properly alone.

It doesn't make much sense but if I use $scheme or https alone I can't login in a new session with a stupid CSRF error. . If I set both, it works.

So I've ignored the proxy.conf, using this config

        #include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app recipes-app;
        set $upstream_port 8080;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
    }
}

I can login but obviously nothing loads. The browser shows an error about mixed content, identical to this error. I can reach apis like api/recipe/?query=&keywords_or=true&foods_or=true&books_or=true&internal=false&random=false&new=true&page=1&page_size=25&last_viewed=5 or /api/recipe-book/

immagine

vabene1111 commented 3 years ago

Ok i am kind of lost. setting the X-Forwarded-Proto header twice does, as you stated, make no sense and its super weird that this even works.

The fact that the api call alone works but not inside the view makes sense, django thinks its not secure and thus renders http links instead of https but i have no idea why the request isnt getting trough with the correct headers.

Now i dont think we are getting anywhere with this issue, i have some other user with setup issues with whom i chatted on discord today and i will do some remote debugging via anydesk/teamviewer/whatever tomorrow afternoon (european time). If you want i can take a look at your setup as well to determine what is wrong with it, other than that i can just say that everything indicates that your error is caused by django not receiving the correct headers and that something in your request chain must be breaking it but i have no idea which part it is.

auanasgheps commented 3 years ago

I could open an issue to the linuxserver/swag guys, they might know more about Nginx... but not about Django. The problem is we still don't know why it's failing.

To be extra safe I'll create a new linuxserver/swag container with absolutely clean configs. If still doesn't work, I'll open an issue to them referencing this one. If that's the case means you could also easily replicate the issue.

Thanks for offering your help, I'm in EU too, but I want to dig a bit deeper and try to find more. I'll keep you posted!

vabene1111 commented 3 years ago

ok, since headers have been a problem for a few users i will try to add some kind of debug section to the system page that maybe just prints out all the headers django receives, that would ease debugging a little

auanasgheps commented 3 years ago

I can replicate the issue with a clean linuxserver/swag istance.

I've copied my current nginx/swag stack and adjusted the configuration to bind the test istance to another folder
Made some basic changes to run the proxy (SSL generation)
Rename recipes.subdomain.conf.sample to recipes.subdomain.conf and make the changes to work (port and container name)

1st error: CSFR login failure 2nd error (if I'm already logged in): nothing loads, but I might have spotted where the problem is: the error below does not list the port. My reverse proxy is not running on 443 but 8080! Might this be related to Django? The service is exposed on 8080 from both the proxy and the recipes front end.

Additionally, are you willing to try this setup? You should be able to replicate. In the meantime I'll open an issue with the linuxserver guys.

vabene1111 commented 3 years ago

i can take a look at this at some point but i am pretty busy right now, since this is most likely an issue with the reverse proxy configuration i am not sure if my time is best spend debugging this or just recommending people to use the default setup as that, at least for now, enables me to work on things that actually help all users and not just those who run a linuxserver/swag setup.

But it bothers me that your setup does not work so i might find some time to test this, not sure yet when

Whatever the case, i am 99.99% sure that this issue is related to some configuration of the proxy and has nothing to do with either django or tandoor. I know there have been issues before in this repository with people using linuxserver/swag, maybe you can find those issues and tag the people wo were involved with them, maybe they got version 0.17 already working and can provide you with a working configuration file.

vabene1111 commented 3 years ago

Ok now that i read your post again and the swag post as well: You are saying your server runs under 8080 and django makes requests to 443 which obviously get refused.

This means that the headers required for django to know under which port it runs are not set correctly thus it returns the wrong api urls. There are two options i see here, either run tandoor on 443 (which is see no reason why not to do this since you are using a reverse proxy with proper domains and having non default ports is annoying anyway) or find out which headers are required for django to know the port of your request.

I know people run tandoor on non default ports and my dev setup does this as well so it should not be a problem but i could also probably implement that you can set the port from the reverse proxcy using a header like detailed here https://docs.djangoproject.com/en/dev/ref/settings/#use-x-forwarded-port

auanasgheps commented 3 years ago

Ok now that i read your post again and the swag post as well: You are saying your server runs under 8080 and django makes requests to 443 which obviously get refused.

This means that the headers required for django to know under which port it runs are not set correctly thus it returns the wrong api urls. There are two options i see here, either run tandoor on 443 (which is see no reason why not to do this since you are using a reverse proxy with proper domains and having non default ports is annoying anyway) or find out which headers are required for django to know the port of your request.

I know people run tandoor on non default ports and my dev setup does this as well so it should not be a problem but i could also probably implement that you can set the port from the reverse proxcy using a header like detailed here https://docs.djangoproject.com/en/dev/ref/settings/#use-x-forwarded-port

Tandoor runs internally 8080 which is the default (I guess) My swag proxy server runs on 8080 instead of 443 (both the host and container) but it's making API calls to the 443 port which isn't listening. The calls I see are done to the FQDN, so they must be on 8080, regardless of which port Django is running.

My reverse proxy is instructed to serve 8080 and I can technically reach Django (ignoring all the problems now) but then Django is providing the API calls which are then made on the wrong port. Given this, do you know if it's happeniong because of Nginx (missing some weird port header) or Django (not being aware of the port)?

smilerz commented 3 years ago

The API calls are constructed based on what scheme, domain, port and path that Django sees in the headers. If Django is providing wrong absolute paths it is because there is information missing and/or wrong in the headers.

vabene1111 commented 3 years ago

Ok so i added a view on the system page that shows the header that django receives, you can update to the develop tag since i dont have time to update master right now. The output of that text field would be really interesting in debugging this issue

auanasgheps commented 3 years ago

I wanted to do a step back but Django really doesn't like the Swag container.

I adopted the full stack as per your documentation, by adding a "dummy" nginx that only serves Recipes. Works great but I'll never expose that container to the internet.

So I though to place my main swag in front of the dummy nginx. Not an elegant solution but still viable.

Well, the page errors out because the browser tries to reach recipes-nginx which is the name of the dummy nginx, but I'm connected to the swag nginx with my full domain name, so django is messing up with whatever swag is trying to do.

I still haven't found the time to update to develop branch, I won't do it before Saturday. I hope will help us understand this behaviour.

immagine

auanasgheps commented 3 years ago

@vabene1111 sorry for the delay, I got carried by other things. Do I still need the develop branch to see the headers?

vabene1111 commented 3 years ago

No this has since been released. There have also been some fixes so your problems might even be resolved.

auanasgheps commented 3 years ago

eheh the problem is still here!

So here are the headers. They look correct to me...

HTTP_CONNECTION:close
HTTP_HOST:recipes.mydomain.tld
HTTP_X_FORWARDED_FOR:10.8.0.2
HTTP_X_FORWARDED_HOST:recipes.mydomain.tld
HTTP_X_FORWARDED_PROTO:https,https
HTTP_X_FORWARDED_SSL:on
HTTP_X_REAL_IP:10.8.0.2
HTTP_SEC_CH_UA:"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"
HTTP_SEC_CH_UA_MOBILE:?0
HTTP_SEC_CH_UA_PLATFORM:"Windows"
HTTP_UPGRADE_INSECURE_REQUESTS:1
HTTP_DNT:1
HTTP_USER_AGENT:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36
HTTP_ACCEPT:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
HTTP_SEC_FETCH_SITE:same-origin
HTTP_SEC_FETCH_MODE:navigate
HTTP_SEC_FETCH_USER:?1
HTTP_SEC_FETCH_DEST:document
HTTP_REFERER:https://recipes.mydomain.tld:8080/search/
HTTP_ACCEPT_ENCODING:gzip, deflate, br
HTTP_ACCEPT_LANGUAGE:it-IT,it;q=0.9,en-GB;q=0.8,en;q=0.7,en-US;q=0.6
HTTP_SEC_GPC:1
HTTP_COOKIE:csrftoken=blablabla sessionid=blablabla

vabene1111 commented 3 years ago

is this maybe the problem HTTP_X_FORWARDED_PROTO:https,https ? i dont think its correct for https to be in there twice ? in my instance it looks like this HTTP_X_FORWARDED_PROTO:https i think one of the proxies does not remove the header and just appends the https.

Maybe try in you proxy to set and not append it to just https ? Other than that i am at a loss here, no idea on how to help you further. If you want we can shedule a meeting next week were i would look at your setup with anydesk, teamviewer or something similar to help you fix the issue.

auanasgheps commented 3 years ago

is this maybe the problem HTTP_X_FORWARDED_PROTO:https,https ? i dont think its correct for https to be in there twice ? in my instance it looks like this HTTP_X_FORWARDED_PROTO:https i think one of the proxies does not remove the header and just appends the https.

Correct, this was my mistake, I may have added this header when playing around with settings.

So I finally found the time to play around a little bit and I FIXED IT!!!!

Give me some time to wrap it up and explain. As you suspected, the issue is in Swag configurations that don't pass the headers that Django wants (but to be fair with them, they work correctly with all other sites).

I'll post updates soon.

auanasgheps commented 3 years ago

First of all, this problem only affects you if you're using linuxserver/swag and you are hosting it on a port that is not 443 (nginx port, not Recipes). The default config works when using 443!

Analysis By default, linuxserver/swag sets two headers that Django doesn't like:

proxy_set_header X-Forwarded-Host $host;
proxy_set_header Host $host;

If I change these two headers from $host to $http_host, Recipes works perfectly!

The Problem linuxserver/swag has a file with generic headers that are required by many apps, including the aforementioned headers, called proxy.conf. We can't get rid of this file, because includes a lot of configs that are also required for Recipes.

I'm not super expert in Nginx, but looks like there's no way to override a config: if I set it in the config file, it gets declared twice.

Solution There are different ways to approach the issue, the one that suits my mental order is:

Create a copy of proxy.conf
Replace proxy_set_header X-Forwarded-Host $host; and proxy_set_header Host $host; to
- proxy_set_header X-Forwarded-Host $http_host; and proxy_set_header Host $http_host;
Update recipes.subdomain.conf to use the new file
Restart the linuxserver/swag container and Recipes will work again

You could also copy the whole content of proxy.conf to recipes.subdomain.conf and set the correct values, but the file will look messy.

linuxserver/swag talks directly to Django front end, without the dedicated nginx instance.

Final words I cannot make a PR to linuxserver/swag because their default config works if using standard ports. Can we add this information to your wiki? I can make the PR if you don't have the time.

auanasgheps commented 2 years ago

@smilerz shall we document what I've discovered in the official doc?

smilerz commented 2 years ago

That would be great if you can submit a PR

vabene1111 commented 2 years ago

Possibly the FAQ or some other section you find fitting

auanasgheps commented 2 years ago

done :)

Nabukodonosor commented 2 years ago

This works for me, but it doesn't load recipe images. Anyone knows why?

TandoorRecipes / recipes

NGINX issues since 0.17 #959

Version

Bug description