search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.33k stars 9.97k forks source link

add exception in documentation with nginx and cloudflare #45522

Closed Alerinos closed 1 year ago

Alerinos commented 1 year ago

Is there an existing issue for this?

Is your feature request related to a problem? Please describe the problem.

In the nginx configuration documentation you should mention cloudflare + nginx configuration.

https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-7.0#configure-nginx

I had a problem where the blazor connection looped because there was a problem with the web socket connection.

This code solved my problem:

        proxy_set_header X-NginX-Proxy true; 
        proxy_set_header Upgrade $http_upgrade; 
        proxy_set_header Connection "upgrade";
        proxy_ssl_session_reuse off; 
        proxy_cache_bypass $http_upgrade;

and more specifically

proxy_set_header Connection "upgrade";

without it, such magic works and it kept throwing me the error 500 + ssl handshake failed

https://gyazo.com/6bae286bb75d2e9e4bef84f8057a0c48

image

My config:

    listen                    443 ssl http2;
    listen                    [::]:443 ssl http2;
    server_name               ****
    ssl_certificate           /etc/ssl/***;
    ssl_certificate_key       /etc/ssl/***;
    ssl_session_timeout       1d;
    ssl_protocols             TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_ciphers               ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_session_cache         shared:SSL:10m;
    ssl_session_tickets       off;
    ssl_stapling              off;

    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        proxy_redirect          off;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        client_max_body_size    32M;
        client_body_buffer_size 128k;
        proxy_connect_timeout   90;
        proxy_send_timeout      90;
        proxy_read_timeout      90;
        proxy_buffers           32 4k;

        proxy_set_header X-NginX-Proxy true; 
        proxy_set_header Upgrade $http_upgrade; 
        proxy_set_header Connection "upgrade";
        proxy_ssl_session_reuse off; 
        proxy_cache_bypass $http_upgrade;

        proxy_pass http://localhost:5001/;
        proxy_http_version 1.1;
    }

Describe the solution you'd like

It's worth looking into this issue and mentioning it in the documentation or solving it in some other way.

Additional context

No response

Tratcher commented 1 year ago

The current doc: https://learn.microsoft.com/en-us/troubleshoot/developer/webapps/aspnetcore/practice-troubleshoot-linux/2-2-install-nginx-configure-it-reverse-proxy#configure-nginx-as-reverse-proxy-to-route-the-requests-to-your-aspnet-core-application proxy_set_header Connection keep-alive;

I can understand that causing problems for websockets, but I thought we'd tested that for signalr. @BrennanConroy ?

BrennanConroy commented 1 year ago

Right after the nginx config snippet there is a sentence telling you to go to SignalR/Blazor specific docs for additional configuration

image

Alerinos commented 1 year ago

@BrennanConroy In fact, there is such a thing. I focused on the configuration in ssl. https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-7.0#https-configuration I think you can also add this information at the bottom, the documentation is long and can be overlooked because they focus on ssl.

I'm about to test the signal setup, need to add additional "/hubroute" location?

eg:

location / {
proxy kestrel 
}
location /hubroute {
proxy websocket
}

right?

Alerinos commented 1 year ago

@Tratcher @BrennanConroy I did everything according to the documentation, below is my config. Once in a while it gets "500 Internal Server Error" by nginx.

upstream StandService{
    server 127.0.0.1:5001;
}

server {
    listen                    443 ssl http2;
    listen                    [::]:443 ssl http2;
    server_name               ***;
    ssl_certificate           /etc/ssl/*.perm;
    ssl_certificate_key       /etc/ssl/*.key;
    ssl_session_timeout       1d;
    ssl_protocols             TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_ciphers               ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_session_cache         shared:SSL:10m;
    ssl_session_tickets       off;
    ssl_stapling              off;

    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    if ($host ~* (.*)....) {
        return 301 https://....$request_uri;
    }

    location / {
        proxy_redirect          off;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        client_max_body_size    32M;
        client_body_buffer_size 128k;
        proxy_connect_timeout   90;
        proxy_send_timeout      90;
        proxy_read_timeout      90;
        proxy_buffers           32 4k;

        proxy_set_header   Connection keep-alive;

        proxy_pass http://StandService;
        proxy_http_version 1.1;

        limit_req  zone=one burst=10 nodelay;
    }

    location /_blazor {
      # App server url
      proxy_pass http://StandService;

      # Configuration for WebSockets
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_cache off;
      # WebSockets were implemented after http/1.0
      proxy_http_version 1.1;

      # Configuration for ServerSentEvents
      proxy_buffering off;

      # Configuration for LongPolling or if your KeepAliveInterval is longer than 60 seconds
      proxy_read_timeout 100s;

      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Nginx.conf

events {
    worker_connections 4096;
    # multi_accept on;
}

http {

    ##
    # Custom settings
    ##

    map $request_uri $redirect_fbclid {
        "~^(.*?)([?&]fbclid=[a-zA-Z0-9_-]+)$"  $1;
    }

    map $http_connection $connection_upgrade {
        "~*Upgrade" $http_connection;
        default keep-alive;
    }

    limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;
    server_tokens  off;

    sendfile on;
    # Adjust keepalive_timeout to the lowest possible value that makes sense 
    # for your use case.
    keepalive_timeout   29;
    client_body_timeout 10; client_header_timeout 10; send_timeout 10;

...
}

What is wrong? I've been sitting on this for a long time and I have no idea what I could have done wrong.

image

Not once the page will load itself and _blazor gets 500. Sometimes the page gets 500... I have no idea what's going on.

Nginx error log (1 day error log ~1.5 Gb..):

2022/12/10 06:12:48 [crit] 1921#1921: accept4() failed (24: Too many open files)
2022/12/10 06:12:48 [alert] 1921#1921: *17593 socket() failed (24: Too many open files) while connecting to upstream, client: IPv6, server: *.se, request: "GET /_blazor?id=tn4K-8lJeeyoZYjffn3GHQ HTTP/1.1", upstream: "http://127.0.0.1:5001/_blazor?id=tn4K-8lJeeyoZYjffn3GHQ", host: "*.se"
2022/12/10 06:12:48 [alert] 1918#1918: *17581 socket() failed (24: Too many open files) while connecting to upstream, client: IPv6, server: *.se, request: "GET /_blazor?id=yW5wVPVsdSXmMM_FBbMlvw&_=1670649168281 HTTP/2.0", upstream: "http://127.0.0.1:5001/_blazor?id=yW5wVPVsdSXmMM_FBbMlvw&_=1670649168281", host: "*.se", referrer: "https://*.se/"
2022/12/10 06:12:48 [alert] 1918#1918: *770 socket() failed (24: Too many open files) while connecting to upstream, client: IPv6, server: *.se, request: "POST /_blazor/negotiate?negotiateVersion=1 HTTP/2.0", upstream: "http://127.0.0.1:5001/_blazor/negotiate?negotiateVersion=1", host: "*.se", referrer: "https://*.se/"

I found a hint: https://serverfault.com/a/1112731 The user claims that he had a similar problem due to a configuration error, I have a similar problem. I tried many configurations, what is in the documentation and on stackoverflow. Unfortunately, all for nothing.

image OS LXD Ubuntu 20 LTS, 64Gb memory, Ryzen 5 3600x, Main OS Proxmox

The server doesn't even use 1% of the resources. I have about 1000 users a day. The problems started the day .net 7 was released

All IPv6 addresses in the error log are addresses of users proxyed by Cloudflare. The server does not have an IPv4 address, only IPv6. It works like this: Proxmox Server (IPv4 + IPv6) -> LXD IPv4 Local IPv6 public -> Cloudflare Proxy IPv4 + IPv6

Maybe that's the problem?

Fail2ban cut all my traffic..

[nginx-req-limit]
enabled = false
filter = nginx-req-limit
action = iptables-multiport[name=ReqLimit, port="http,https", protocol=tcp]
logpath = /var/log/nginx/*error.log
findtime = 600
bantime = 7200
maxretry = 10

CF Firewall Events: image

To be sure, I did it again 1:1 according to the documentation, it didn't help.

I found bugs:

12.10.22 09:24 Warning No XML encryptor configured. Key {2e9219f8-833e-4101-896c-40dedb5eb8c5} may be persisted to storage in unencrypted form. 
12.10.22 09:24 Warning Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits. 
12.10.22 09:24 Warning Using an in-memory repository. Keys will not be persisted to storage. 
12.10.22 09:24 Warning No XML encryptor configured. Key {8c4d3301-3657-457e-944d-df3d1533ac97} may be persisted to storage in unencrypted form. 
12.10.22 09:24 Warning Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits. 
12.10.22 09:24 Warning Using an in-memory repository. Keys will not be persisted to storage. 
12.10.22 09:22 Error Unhandled exception in circuit '"UPcusR1_t3zFRYb5laCQXsZwm6hJzBtpeFrz52Zy4MU"'. 
12.10.22 09:22 Warning Unhandled exception rendering component: "JavaScript interop calls cannot be issued at this time. This is because the circuit has disconnected and is being disposed." 
12.10.22 09:22 Error Unhandled exception in circuit '"zIYVF7WRBhs9AzN7pmOsh-zStPAvbaTWslHyT3Pp0Ok"'. 
12.10.22 09:22 Warning Unhandled exception rendering component: "JavaScript interop calls cannot be issued at this time. This is because the circuit has disconnected and is being disposed." 
12.10.22 09:22 Error Unhandled exception in circuit '"exDT4XLz-d3LNN1xRCWcN7qEtnWuZbt1MMfgGH8nggM"'. 
12.10.22 09:22 Warning Unhandled exception rendering component: "JavaScript interop calls cannot be issued at this time. This is because the circuit has disconnected and is being disposed." 
12.10.22 09:21 Error Unhandled exception in circuit '"AlSGMT9UeR5bvlPJ6cGnFc-kY5grw0haGYPiLjLwZLs"'. 
12.10.22 09:21 Warning Unhandled exception rendering component: "JavaScript interop calls cannot be issued at this time. This is because the circuit has disconnected and is being disposed." 
12.10.22 09:21 Error Unhandled exception in circuit '"Ju747fLhL-rSaq2MdHViMs4xyoQMF71lEr7EtYGsChs"'. 
12.10.22 09:21 Warning Unhandled exception rendering component: "JavaScript interop calls cannot be issued at this time. This is because the circuit has disconnected and is being disposed." 
12.10.22 09:21 Error Unhandled exception in circuit '"36Ak08WGyLOELK3ba4R9kjDBusmmmQHguN79O6gYiIU"'. 
12.10.22 09:17 Warning Failed to determine the https port for redirect. 
12.10.22 09:17 Warning Failed to determine the https port for redirect. 
12.10.22 09:17 Warning No XML encryptor configured. Key {d14ec7ee-19cd-4b2e-b722-804d1d49b9cd} may be persisted to storage in unencrypted form. 
12.10.22 09:17 Warning Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits. 
12.10.22 09:17 Warning Using an in-memory repository. Keys will not be persisted to storage. 
12.10.22 09:17 Warning No XML encryptor configured. Key {6b122903-a60d-4259-8119-bca5f06da1fe} may be persisted to storage in unencrypted form. 
12.10.22 09:17 Warning Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits. 
12.10.22 09:17 Warning Using an in-memory repository. Keys will not be persisted to storage.

Is it related to this? https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-storage-providers?view=aspnetcore-3.1&tabs=visual-studio#file-system

Application without nginx no problem

Alerinos commented 1 year ago

I searched and read a lot. I solved the problem with (24: Too many open files) while connecting to upstream.

/etc/systemd/system/nginx.service.d/override.conf

add

[Service]
LimitNOFILE=65536

and nginx config:

worker_rlimit_nofile 65536;

I have a lot of websites in different technologies, this is the first time I see this problem. I think it's worth describing it in the documentation or fixing the problem from the backend side.

Next problem:

2022/12/11 06:13:49 [error] 1615#1615: *1619597 connect() failed (111: Connection refused) while connecting to upstream, client: 2400:cb00:73:1024::a29e:67bd, server: ***, request: "GET /_blazor?id=Etfs5lISKABRMAM-CMEZhQ HTTP/1.1", upstream: "http://127.0.0.1:5001/_blazor?id=Etfs5lISKABRMAM-CMEZhQ", host: "***"
2022/12/11 06:13:49 [error] 1616#1616: *1614358 connect() failed (111: Connection refused) while connecting to upstream, client: 2400:cb00:73:1024::a29e:6643, server: ***, request: "POST /_blazor/negotiate?negotiateVersion=1 HTTP/2.0", upstream: "http://127.0.0.1:5001/_blazor/negotiate?negotiateVersion=1", host: "***", referrer: "https://**"
ghost commented 1 year ago

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

amcasey commented 1 year ago

Triage: We should make the first sample support websockets.

halter73 commented 1 year ago

Both https://learn.microsoft.com/en-us/troubleshoot/developer/webapps/aspnetcore/practice-troubleshoot-linux/2-2-install-nginx-configure-it-reverse-proxy#configure-nginx-as-reverse-proxy-to-route-the-requests-to-your-aspnet-core-application and https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-7.0#configure-nginx should probably be updated. The latter at least links to the SignalR docs, but we should make sure all our sample nginx.conf files support WebSockets so it's impossible to miss.

darkedges commented 1 year ago

This what I have working for Kubernetes ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"   
    nginx.ingress.kubernetes.io/server-snippets: |
      location / {
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection keep-alive;
        proxy_set_header   Connection "Upgrade";
        proxy_set_header   Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
      }
  name: pingfedqrcodeauth-ingress
  namespace: spring
spec:
  ingressClassName: nginx
  rules:
  - host: qrcodeauth.xxx.com
    http:
      paths:
      - backend:
          service:
            name: pingfedqrcodeauth
            port:
              number: 443
        path: /
        pathType: Prefix

I further had to stop CloudFlare from stripping out comments to get the WebSocket Circuit connection to work correctly. https://stackoverflow.com/a/75424394/6719700

Is there anything I can do in my Application to stop Blazor from using comments to store details needed to make it work?

BrennanConroy commented 1 year ago

@darkedges file a new issue.

BrennanConroy commented 1 year ago

Both docs mentioned in https://github.com/dotnet/aspnetcore/issues/45522#issuecomment-1347475238 have been updated to include websocket support by default.