search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
3.95k stars 533 forks source link

Issues with rendering MeshCentral when reverse proxied #4320

Closed takuy closed 9 months ago

takuy commented 2 years ago

Describe the bug So... sorry for the vague bug report but looking for a hint as where to look. Will provide config/logs with sensitive data removed tomorrow.

We're publishing the web UI for our instance of Mesh to the internet via an apache reverse proxy. Sometimes, the page needs to be refreshed 3 times for it to render properly. This is published on a different domain name from what our agents talk to; as we're only using the MeshAgent inside of the network.

So, we use Docker to host Mesh and make it available outside the server using Nginx as a reverse proxy, which we then proxy again with Apache for external access. This worked fine inside of our network before introducing Apache.

The first time you go to the page, it 400s, just renders the text "Bad Request". Next refresh, most of Mesh renders but some assets also 400:. Third refresh, everything works fine! Wait a few minutes, refresh, goes back to bad request and repeat... this is consistent. 

Navigated to https://mydomain.com/mesh/
mydomain.com/:11          GET https://mydomain.com/mesh/styles/style.css net::ERR_ABORTED 400 (Bad Request)
mydomain.com/:14          GET https://mydomain.com/mesh/styles/xterm.css net::ERR_ABORTED 400 (Bad Request)
mydomain.com/:13          GET https://mydomain.com/mesh/styles/ol3-contextmenu.min.css net::ERR_ABORTED 400 (Bad Request)
mydomain.com/:15          GET https://mydomain.com/mesh/styles/flatpickr.min.css net::ERR_ABORTED 400 (Bad Request)
mydomain.com/:17          GET https://mydomain.com/mesh/scripts/common-0.0.1.js net::ERR_ABORTED 400 (Bad Request)
mydomain.com/:672

I'm just not sure why it would only work sometimes. Will provide cleaned apache config tomorrow.

Server Software (please complete the following information):

OS: RHEL 7 Virtualization: Docker Network: WAN, Nginx reverse proxy, Apache Reverse Proxy Version: 1.0.60 Node: 12.22.5 Browser: Chrome

si458 commented 2 years ago

this seems like an issue with the fact you are having a double reverse proxy? what is your config.json also your config for apache and nginx?

takuy commented 2 years ago

this seems like an issue with the fact you are having a double reverse proxy? what is your config.json also your config for apache and nginx?

For sure, the double proxy introduces this but I'm not clear as to why. 3 refreshes consistently to make it work seems like something that can be fixed.

Here are the cleaned config:

config.json:

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "server-1.com",
    "WANonly": true,
    "_LANonly": true,
    "trustedProxy": ["proxy.proxy_network", "otherproxy ip", "another one"],
    "sessionKey": "THE KEY",
    "port": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "_agentLogDump": true,
    "AgentPong": 300,
    "SelfUpdate": false,
    "AllowFraming": "false",
    "cookieIpCheck": false,
    "WebRTC": "false",
    "MariaDB": {
      "host": "sqlserver.com",
      "user": "dbuser",
      "password": "totallythepassword",
      "database": "meshcentral"
    }
  },
  "domains": {
    "mesh": {
      "title": "Meshcentral",
      "title2": "server-1.com mesh central",
      "newAccounts": false,
      "footer": "server-1",
      "unknownUserRootRedirect": "/mesh/auth-saml",
      "showPasswordLogin": false,
      "CertUrl": "https://server-1.com:443/",
      "consentMessages": {
        "consentTimeout": 30,
        "autoAcceptOnTimeout": true
      },
      "ManageAllDeviceGroups": [ "group", "~saml:user1", "~saml:user2", "~saml:user3" ],
      "authStrategies": {
        "saml": {
          "callbackurl": "https://server1.com/mesh/auth-saml-callback",
          "_disableRequestedAuthnContext": true,
          "newAccounts": true,
          "newAccountsUserGroups": [ "ugrp/mesh/group" ],
          "_newAccountsRights": [ "nonewgroups", "notools" ],
          "entityid": "https://server1.com/mesh/auth-saml-callback",
          "idpurl": "https://idpserver.com/simplesaml/saml2/idp/SSOService.php",
          "cert": "saml_prod.crt",
          "logouturl": "https://ssologout.com/idp/profile/Logout"
        }
      }
    }
  }
}

nginx config (to expose to services outside of Docker)

server {
    #listen       80;
    #listen  [::]:80;
    listen              443 ssl;
    ssl_certificate     /etc/nginx/conf.d/cert.crt;
    ssl_certificate_key /etc/nginx/conf.d/private.key;
    ssl_protocols       TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    server_name  server-1.com server1.com;
    #access_log  /var/log/nginx/host.access.log  main;
    proxy_pass_request_headers on;
    client_body_in_file_only on;
    client_body_buffer_size 100M;
    client_max_body_size 100M;
    lingering_time 60s;
    resolver 127.0.0.11;

    location /mesh/ {
        proxy_read_timeout 300;
        proxy_pass https://meshcentral.proxy_network;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        #Websocket Support
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "Upgrade";
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Apache config (to expose to internet)

<VirtualHost 8.8.8.8:443 >
ServerName server1.com
SSLEngine On
SSLProxyEngine On
SSLCertificateFile /etc/pki/tls/certs/server1/cert.crt
SSLCertificateKeyFile /etc/pki/tls/certs/server1/private.key
SSLCertificateChainFile /etc/pki/tls/certs/server1/intermediate.crt

<Location /mesh>

    Require all granted
    RewriteEngine on
    ProxyPreserveHost On
    RequestHeader set "Host" "server1.com"
    ProxyPass  "https://server-1.com/mesh"
    ProxyPassReverse "https://server-1.com/mesh"
    RewriteCond ${HTTP:UPGRADE} websocket [NC]
    RewriteCond ${HTTP:CONNECTION} Upgrade [NC]
    ProxyPass  "wss://server-1.com/mesh"
    ProxyPassReverse "wss://server-1.com/mesh"
</Location>

ErrorLog "/var/log/httpd/utils_error"
TransferLog "/var/log/httpd/utils_access"
#LogLevel debug

</VirtualHost>
Ylianst commented 2 years ago

I don't see any obvious problems with any of the configuration files, they all look great. I am going to take a complete guess as to what is going on. Reverse proxies will route requests based on the DNS name that was used for the request, so if you type "server1.sample.com" and "server2.sample.com" they may hit the save IP address, but the reverse proxy will forward the requests to two different servers. Anyway...

I have found that if no DNS name is provided, NGINX (and probably Apache) will just round robin direct the request to various servers since it has no idea which server to send the request to.

My complete guess is that the second (inner) reverse proxy is getting request that do not have the DNS name in the request and does not know how to forward that connection. So, if you have two servers, one MeshCentral and another one, half the connections will go to the wrong one.

Each time you reload, 50% of the connections will be routed correctly and slowly, you end up getting the entire MeshCentral page by chance.

Of course, this is not a MeshCentral issue. You will need to look at the logs of the inner reverse proxy to see why it's not routing the connections right and may need to modify the config of the outer one to pass the DNS of the request correctly.

Anyway, this is my guess. Let me know if I happen to guess right.

takuy commented 2 years ago

So @Ylianst it's weird. As far as we can tell, per a tcpdump and nginx debug logs, the request is always making it nginx and then to the proper container but is being killed randomly on 2 of the 3 page loads. No request shows up in MeshCentral's trace function though. We're pretty stuck. The fact that it's consistent (3 refreshes to fix) is super weird to me. Any further ideas to troubleshoot?

silversword411 commented 2 years ago

You looked at your session limits nginx proxy settings? Maybe it set to kill the connection?

takuy commented 2 years ago

You looked at your session limits nginx proxy settings? Maybe it set to kill the connection?

This doesn't seem to be the case.

Here's the nginx debug log. I don't see anything obvious standing out, except that it instantly 400s. For context: 111.111.x.x is used in place of our internal network IP. 222.222.x.x is my home IP.

"cph-utils.mydomain.com" is our internal name for the server (nginx is used as a proxy for the containers on this server) "cphutils.mydomain.com" is our external proxy for the server (apache)

2022/08/02 23:46:03 [debug] 24#24: *397 accept: 111.111.164.95:34118 fd:85
2022/08/02 23:46:03 [debug] 24#24: *397 event timer add: 85: 60000:554770717
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 1
2022/08/02 23:46:03 [debug] 24#24: *397 epoll add event: fd:85 op:1 ev:80002001
2022/08/02 23:46:03 [debug] 24#24: *397 http check ssl handshake
2022/08/02 23:46:03 [debug] 24#24: *397 http recv(): 1
2022/08/02 23:46:03 [debug] 24#24: *397 https ssl handshake: 0x16
2022/08/02 23:46:03 [debug] 24#24: *397 tcp_nodelay
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 0
2022/08/02 23:46:03 [debug] 24#24: *397 SSL server name: null
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_do_handshake: -1
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_get_error: 2
2022/08/02 23:46:03 [debug] 24#24: *397 SSL handshake handler: 0
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_do_handshake: 1
2022/08/02 23:46:03 [debug] 24#24: *397 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 1
2022/08/02 23:46:03 [debug] 24#24: *397 http wait request handler
2022/08/02 23:46:03 [debug] 24#24: *397 malloc: 000055E94C24B080:1024
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: -1
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_get_error: 2
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C24B080
2022/08/02 23:46:03 [debug] 24#24: *397 http wait request handler
2022/08/02 23:46:03 [debug] 24#24: *397 malloc: 000055E94C24B080:1024
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: 1024
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: avail:0
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 0
2022/08/02 23:46:03 [debug] 24#24: *397 posix_memalign: 000055E94C275820:4096 @16
2022/08/02 23:46:03 [debug] 24#24: *397 http process request line
2022/08/02 23:46:03 [debug] 24#24: *397 http request line: "GET /mesh/ HTTP/1.1"
2022/08/02 23:46:03 [debug] 24#24: *397 http uri: "/mesh/"
2022/08/02 23:46:03 [debug] 24#24: *397 http args: ""
2022/08/02 23:46:03 [debug] 24#24: *397 http exten: ""
2022/08/02 23:46:03 [debug] 24#24: *397 posix_memalign: 000055E94C27CA80:4096 @16
2022/08/02 23:46:03 [debug] 24#24: *397 http process request header line
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Host: cph-utils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Upgrade-Insecure-Requests: 1"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Sec-Fetch-Site: none"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Sec-Fetch-Mode: navigate"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Sec-Fetch-User: ?1"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Sec-Fetch-Dest: document"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103""
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "sec-ch-ua-mobile: ?0"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "sec-ch-ua-platform: "Windows""
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Accept-Encoding: gzip, deflate, br"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Accept-Language: en-US,en;q=0.9"
2022/08/02 23:46:03 [debug] 24#24: *397 http alloc large header buffer
2022/08/02 23:46:03 [debug] 24#24: *397 posix_memalign: 000055E94C266850:512 @16
2022/08/02 23:46:03 [debug] 24#24: *397 malloc: 000055E94C27EBA0:32768
2022/08/02 23:46:03 [debug] 24#24: *397 http large header alloc: 000055E94C27EBA0 32768
2022/08/02 23:46:03 [debug] 24#24: *397 http large header copy: 380
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: 535
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: 43
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_read: -1
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_get_error: 2
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Cookie: domain.cookies.euCompliance.dismissed=0; _gcl_au=1.1.2002223468.1658443547; _fbp=fb.1.1658443546675.865341403; _cc_id=6a3d8af693f12b7b6aa16412853e0149; _ga_SHC8M7ZQVS=GS1.1.1658704199.2.0.1658704202.0; _ga_6JS8P20QE2=GS1.1.1658726007.3.0.1658726011.0; _ga_18X6B7F78P=GS1.1.1658875535.4.0.1658875535.60; _ga_Z3YPKC3YMJ=GS1.1.1658875535.4.0.1658875535.0; _ga=GA1.2.114940629.1658371487; sso_1466609665382=VND_RSSO_V2.eyJpYXQiOjE2NTk0MDA1ODM2MjgsInNydiI6Imh0dHA6Ly9yc3NvLXNlcnZpY2U6ODA4MC9yc3NvIiwicmxtIjoiKiIsInRva2VuSWQiOiJfYjBhMmFhZDaEtMWJkNS00MmU4LWJjZWEtYmQ1NDZmYWNlZmI3In0=; xid=eyJwYXNzcG9ydCI6eyJ1c2VyIjoifnNhbWw6dHVlNTkwMzYifSwidXNlcmlkIjoidXNlci9tZXNoL35zYW1sOnR1ZTU5MDM2IiwieCI6IlQ3RUpJMGZlIiwidCI6Mjc2NTgwNjN9; xid.sig=VCd5NC4Z49TL0BCsNsawlBW9WbfYikBTQ9tZQeLKir-SNRVkflpgrw3mbrjqLz5C"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "X-Forwarded-For: 222.222.219.236"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "X-Forwarded-Host: cph-utils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "X-Forwarded-Server: cphutils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Upgrade: WebSocket"
2022/08/02 23:46:03 [debug] 24#24: *397 http header: "Connection: Upgrade"
2022/08/02 23:46:03 [debug] 24#24: *397 http header done
2022/08/02 23:46:03 [debug] 24#24: *397 event timer del: 85: 554770717
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 0
2022/08/02 23:46:03 [debug] 24#24: *397 rewrite phase: 1
2022/08/02 23:46:03 [debug] 24#24: *397 test location: "/itable/"
2022/08/02 23:46:03 [debug] 24#24: *397 test location: "/portainer/"
2022/08/02 23:46:03 [debug] 24#24: *397 test location: "/mesh/"
2022/08/02 23:46:03 [debug] 24#24: *397 using configuration "/mesh/"
2022/08/02 23:46:03 [debug] 24#24: *397 http cl:-1 max:104857600
2022/08/02 23:46:03 [debug] 24#24: *397 rewrite phase: 3
2022/08/02 23:46:03 [debug] 24#24: *397 post rewrite phase: 4
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 5
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 6
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 7
2022/08/02 23:46:03 [debug] 24#24: *397 access phase: 8
2022/08/02 23:46:03 [debug] 24#24: *397 access phase: 9
2022/08/02 23:46:03 [debug] 24#24: *397 access phase: 10
2022/08/02 23:46:03 [debug] 24#24: *397 post access phase: 11
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 12
2022/08/02 23:46:03 [debug] 24#24: *397 generic phase: 13
2022/08/02 23:46:03 [debug] 24#24: *397 http init upstream, client timer: 0
2022/08/02 23:46:03 [debug] 24#24: *397 epoll add event: fd:85 op:3 ev:80002005
2022/08/02 23:46:03 [debug] 24#24: *397 posix_memalign: 000055E94C2438D0:4096 @16
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "Host"
2022/08/02 23:46:03 [debug] 24#24: *397 http script var: "cph-utils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "X-Real-IP"
2022/08/02 23:46:03 [debug] 24#24: *397 http script var: "111.111.164.95"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "X-Forwarded-For"
2022/08/02 23:46:03 [debug] 24#24: *397 http script var: "222.222.219.236, 111.111.164.95"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "X-Forwarded-Proto"
2022/08/02 23:46:03 [debug] 24#24: *397 http script var: "https"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "Upgrade"
2022/08/02 23:46:03 [debug] 24#24: *397 http script var: "WebSocket"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "Connection"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: "Upgrade"
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: ""
2022/08/02 23:46:03 [debug] 24#24: *397 http script copy: ""
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Upgrade-Insecure-Requests: 1"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Sec-Fetch-Site: none"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Sec-Fetch-Mode: navigate"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Sec-Fetch-User: ?1"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Sec-Fetch-Dest: document"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103""
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "sec-ch-ua-mobile: ?0"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "sec-ch-ua-platform: "Windows""
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Accept-Encoding: gzip, deflate, br"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Accept-Language: en-US,en;q=0.9"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Cookie: domain.cookies.euCompliance.dismissed=0; _gcl_au=1.1.2002223468.1658443547; _fbp=fb.1.1658443546675.865341403; _cc_id=6a3d8af693f12b7b6aa16412853e0149; _ga_SHC8M7ZQVS=GS1.1.1658704199.2.0.1658704202.0; _ga_6JS8P20QE2=GS1.1.1658726007.3.0.1658726011.0; _ga_18X6B7F78P=GS1.1.1658875535.4.0.1658875535.60; _ga_Z3YPKC3YMJ=GS1.1.1658875535.4.0.1658875535.0; _ga=GA1.2.114940629.1658371487; sso_1466609665382=VND_RSSO_V2.eyJpYXQiOjE2NTk0MDA1ODM2MjgsInNydiI6Imh0dHA6Ly9yc3NvLXNlcnZpY2U6ODA4MC9yc3NvIiwicmxtIjoiKiIsInRva2VuSWQiOiJfYjBhMmFhZDaEtMWJkNS00MmU4LWJjZWEtYmQ1NDZmYWNlZmI3In0=; xid=eyJwYXNzcG9ydCI6eyJ1c2VyIjoifnNhbWw6dHVlNTkwMzYifSwidXNlcmlkIjoidXNlci9tZXNoL35zYW1sOnR1ZTU5MDM2IiwieCI6IlQ3RUpJMGZlIiwidCI6Mjc2NTgwNjN9; xid.sig=VCd5NC4Z49TL0BCsNsawlBW9WbfYikBTQ9tZQeLKir-SNRVkflpgrw3mbrjqLz5C"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "X-Forwarded-Host: cph-utils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "X-Forwarded-Server: cphutils.mydomain.com"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header:
2022/08/02 23:46:03 [debug] 24#24: *397 http cleanup add: 000055E94C27D568
2022/08/02 23:46:03 [debug] 24#24: *397 get rr peer, try: 1
2022/08/02 23:46:03 [debug] 24#24: *397 stream socket 90
2022/08/02 23:46:03 [debug] 24#24: *397 epoll add connection: fd:90 ev:80002005
2022/08/02 23:46:03 [debug] 24#24: *397 connect to 172.20.0.2:443, fd:90 #398
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream connect: -2
2022/08/02 23:46:03 [debug] 24#24: *397 posix_memalign: 000055E94C25F5C0:128 @16
2022/08/02 23:46:03 [debug] 24#24: *397 event timer add: 90: 60000:554770731
2022/08/02 23:46:03 [debug] 24#24: *397 http finalize request: -4, "/mesh/?" a:1, c:2
2022/08/02 23:46:03 [debug] 24#24: *397 http request count:2 blk:0
2022/08/02 23:46:03 [debug] 24#24: *397 http run request: "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream check client, write event:1, "/mesh/"
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream request: "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream send request handler
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream send request
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream send request body
2022/08/02 23:46:03 [debug] 24#24: *397 chain writer buf fl:1 s:1671
2022/08/02 23:46:03 [debug] 24#24: *397 chain writer in: 000055E94C276810
2022/08/02 23:46:03 [debug] 24#24: *397 writev: 1671 of 1671
2022/08/02 23:46:03 [debug] 24#24: *397 chain writer out: 0000000000000000
2022/08/02 23:46:03 [debug] 24#24: *397 event timer del: 90: 554770731
2022/08/02 23:46:03 [debug] 24#24: *397 event timer add: 90: 300000:555010731
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream request: "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream process header
2022/08/02 23:46:03 [debug] 24#24: *397 malloc: 000055E94C267DA0:4096
2022/08/02 23:46:03 [debug] 24#24: *397 recv: eof:1, avail:-1
2022/08/02 23:46:03 [debug] 24#24: *397 recv: fd:90 103 of 4096
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy status 400 "400 Bad Request"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Connection: close"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Content-type: text/html"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header: "Content-Length: 11"
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy header done
2022/08/02 23:46:03 [debug] 24#24: *397 HTTP/1.1 400 Bad Request
2022/08/02 23:46:03 [debug] 24#24: *397 write new buf t:1 f:0 000055E94C27D8A8, pos 000055E94C27D8A8, size: 156 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 http write filter: l:0 f:0 s:156
2022/08/02 23:46:03 [debug] 24#24: *397 http cacheable: 0
2022/08/02 23:46:03 [debug] 24#24: *397 http proxy filter init s:400 h:0 c:0 l:11
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream process upstream
2022/08/02 23:46:03 [debug] 24#24: *397 pipe read upstream: 1
2022/08/02 23:46:03 [debug] 24#24: *397 pipe preread: 11
2022/08/02 23:46:03 [debug] 24#24: *397 readv: eof:1, avail:0
2022/08/02 23:46:03 [debug] 24#24: *397 readv: 1, last:3993
2022/08/02 23:46:03 [debug] 24#24: *397 pipe recv chain: 0
2022/08/02 23:46:03 [debug] 24#24: *397 pipe buf free s:0 t:1 f:0 000055E94C267DA0, pos 000055E94C267DFC, size: 11 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 pipe length: 11
2022/08/02 23:46:03 [debug] 24#24: *397 input buf #0
2022/08/02 23:46:03 [debug] 24#24: *397 pipe write downstream: 1
2022/08/02 23:46:03 [debug] 24#24: *397 pipe write downstream flush in
2022/08/02 23:46:03 [debug] 24#24: *397 http output filter "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http copy filter: "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http postpone filter "/mesh/?" 000055E94C243F78
2022/08/02 23:46:03 [debug] 24#24: *397 write old buf t:1 f:0 000055E94C27D8A8, pos 000055E94C27D8A8, size: 156 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 write new buf t:1 f:0 000055E94C267DA0, pos 000055E94C267DFC, size: 11 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 http write filter: l:0 f:0 s:167
2022/08/02 23:46:03 [debug] 24#24: *397 http copy filter: 0 "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 pipe write downstream done
2022/08/02 23:46:03 [debug] 24#24: *397 event timer: 90, old: 555010731, new: 555010732
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream exit: 0000000000000000
2022/08/02 23:46:03 [debug] 24#24: *397 finalize http upstream request: 0
2022/08/02 23:46:03 [debug] 24#24: *397 finalize http proxy request
2022/08/02 23:46:03 [debug] 24#24: *397 free rr peer 1 0
2022/08/02 23:46:03 [debug] 24#24: *397 close http upstream connection: 90
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C25F5C0, unused: 48
2022/08/02 23:46:03 [debug] 24#24: *397 event timer del: 90: 555010731
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 0
2022/08/02 23:46:03 [debug] 24#24: *397 http upstream temp fd: -1
2022/08/02 23:46:03 [debug] 24#24: *397 http output filter "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http copy filter: "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http postpone filter "/mesh/?" 00007FFFB1FE64F0
2022/08/02 23:46:03 [debug] 24#24: *397 write old buf t:1 f:0 000055E94C27D8A8, pos 000055E94C27D8A8, size: 156 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 write old buf t:1 f:0 000055E94C267DA0, pos 000055E94C267DFC, size: 11 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 write new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
2022/08/02 23:46:03 [debug] 24#24: *397 http write filter: l:1 f:0 s:167
2022/08/02 23:46:03 [debug] 24#24: *397 http write filter limit 2097152
2022/08/02 23:46:03 [debug] 24#24: *397 malloc: 000055E94C277BB0:16384
2022/08/02 23:46:03 [debug] 24#24: *397 SSL buf copy: 156
2022/08/02 23:46:03 [debug] 24#24: *397 SSL buf copy: 11
2022/08/02 23:46:03 [debug] 24#24: *397 SSL to write: 167
2022/08/02 23:46:03 [debug] 24#24: *397 SSL_write: 167
2022/08/02 23:46:03 [debug] 24#24: *397 http write filter 0000000000000000
2022/08/02 23:46:03 [debug] 24#24: *397 http copy filter: 0 "/mesh/?"
2022/08/02 23:46:03 [debug] 24#24: *397 http finalize request: 0, "/mesh/?" a:1, c:1
2022/08/02 23:46:03 [debug] 24#24: *397 set http keepalive handler
2022/08/02 23:46:03 [debug] 24#24: *397 http close request
2022/08/02 23:46:03 [debug] 24#24: *397 http log handler
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C267DA0
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C275820, unused: 0
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C27CA80, unused: 0
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C2438D0, unused: 2072
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C24B080
2022/08/02 23:46:03 [debug] 24#24: *397 hc free: 0000000000000000
2022/08/02 23:46:03 [debug] 24#24: *397 hc busy: 000055E94C27BFE8 1
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C27EBA0
2022/08/02 23:46:03 [debug] 24#24: *397 free: 000055E94C277BB0
2022/08/02 23:46:03 [debug] 24#24: *397 reusable connection: 1
2022/08/02 23:46:03 [debug] 24#24: *397 event timer add: 85: 65000:554775732
dinger1986 commented 9 months ago

@si458 stale can close