"Failed to load web certificate" when behind a forward proxy

[KyleMaas]

Describe the bug My MeshCentral server is behind both a forward (for outbound requests) and reverse (for inbound requests) proxy. However, when I try to run the server and add an agent, I get the following error:

Oct  2 19:14:02 (hostname censored) node[2958]: Failed to load web certificate at: "https://(public address censored)", host: "(public address censored)"

This seems to be because this is not going through the forward proxy so it's not able to retrieve the certificate from the reverse proxy servers.

To Reproduce Steps to reproduce the behavior:

1. Install MeshCentral behind both a forward and reverse proxy
2. Configure to enable TLS offloading, set NPM Proxy to forward proxy, set agent proxy settings to forward proxy
3. Restart MeshCentral
4. Try to add a client
5. Message shows up in log

Expected behavior I would expect that it would go through the forward proxy to retrieve the certificate it's supposed to be using

Screenshots Not applicable

Server Software (please complete the following information):

• OS: Ubuntu 22.04
• Virtualization: The entire Ubuntu install is virtualized, but the software is installed directly mostly using the guide at https://meshcentral-community.com/doku.php?id=howto:installation:manual:ubuntu:moresecure
• Network: WAN-only, reverse proxy, forward proxy, TLS offloading
• Version: 1.1.11
• Node: 12.22.9

Client Device (please complete the following information):

• Device: Desktop
• OS: Gentoo
• Network: Remote over WAN
• Browser: Chromium
• MeshCentralRouter Version: N/A

Remote Device (please complete the following information):

• Device: Server
• OS: Windows Server
• Network: Remote over WAN
• Current Core Version (if known): Can't get it to add to the server

Additional context Add any other context about the problem here.

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "MongoDb": "mongodb://127.0.0.1:27017/meshcentral",
    "cert": "(address censored)",
    "WANonly": true,
    "_LANonly": true,
    "_sessionKey": "MyReallySecretPassword1",
    "port": 443,
    "_aliasPort": 443,
    "_redirPort": 80,
    "_redirAliasPort": 80,
    "_trustedProxy": "(censored list of reverse proxies, not applicable because ignored)",
    "tlsOffload": true,
    "npmProxy": "http://proxy:3128/",
    "agentNoProxy": "http://proxy:3128/",
    "agentTimeStampProxy": "http://proxy:3128/"
  },
  "domains": {
    "": {
      "title": "(censored)",
      "certURL": "https://(address censored)",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "myemail@mydomain.com",
    "names": "myserver.mydomain.com",
    "skipChallengeVerification": true,
    "production": false
  }
}

[silversword411]

You probably don't have it configured correctly https://www.youtube.com/watch?v=YSmiLyKSX2I

[Ylianst]

So, obviously the URL you provide to "CertURL" must be valid and working from MeshCentral's location in your network. You can try "wget" or "curl" to check that the URL works. If it does work and it's a MeshCentral specific issue, let us know, but i suspect that curl or wget would also fail.

One alternative you have is to save your cer to a file and put:

"certurl": "file://\tmp\cert.cer"
"certurl": "file://C:\cert\cert.cer"

This way, MeshCentral will load your cert from a file. You must provide a full path after the file://. The problem with this is that each time you rotate your reverse proxy cert, it will cause MeshCentral to stop working until you file the file and restart MeshCentral... so it's best to provide a HTTPS URL if possible.

Hope that helps.

[KyleMaas]

@Ylianst

I am able to access it using both curl and wget on the command line of the machine running MeshCentral. I can even

su -s /bin/bash meshcentral

...just to make sure it's the same for the meshcentral user, and then run both curl and wget and both of them work and are able to retrieve the login page just fine.

Using a file would only be a temporary workaround because the frontend reverse proxies' certificates are managed separately and this system wouldn't have access to them, so I'd have to just periodically pull them some other way. Having it access the certificate and use that instead would make that process maintenance-free.

[KyleMaas]

AFAICT, this line here is the culprit:

https://github.com/Ylianst/MeshCentral/blob/b99fb686750ee49cac64478ac895acc236dbd94c/certoperations.js#L546C36-L546C36

And I don't see anywhere in the documentation for the tls module where the connect() method accepts a proxy. So that might have to be retrieved another way.

[si458]

just curious, have u tried updating ur node to the latest LTS version (18) ?

also try running it in debug mode for cert like node node_modules/meshcentral --debug cert and see what output you get

also what is the proxy you are using for outbound filtering?

also agentNoProxy should be a boolean value When enabled, all newly installed MeshAgents will be instructed to no use a HTTP/HTTPS proxy even if one is configured on the remote system defaults to false

[KyleMaas]

To be honest, I couldn't afford to spend any more time trying things. So I built a script that just pulls the certificate through the proxy and installs it into MeshCentral. The problem's still there, but I had to get this thing into production.

@si458 Haven't tried the latest LTS version, nor did I try running in debug mode. The outbound proxy is Squid. But there's no point in my trying that because I can see in the code that it's not feeding the proxy to the tls module anywhere, nor does the tls module support it, so as far as I can tell the only way to fix this would be to switch to a different library that is capable of using a forward proxy.