Apache Reverse Proxy, No TLSOffload, and incorrect IP addresses

[MordyT]

Mesh Server sitting behind an Apache Reverse Proxy (configs can be seen here: https://github.com/Ylianst/MeshCentral/issues/317#issuecomment-569991844), but TLS Offload is disabled. All IPs show as the reverse proxy internal IP address.

(note I don't use TLSOffload because something breaks and Mesh dies. Behind an NGINX server, works fine, behind Apache it fails to start. I don't really care to offload TLS one way or the other, I can just add resources to the correct server to deal with the load. Apache is not on same machine.)

How can I fix this?

[Ylianst]

So, in your setup, is Apache just forwarding the TLS connections from port 443 as-is? Obviously, MeshCentral will see all connections coming from the Apache IP address in this case.

When you have Apache perform TLS and forward the connection to MeshCentral, Apache (like NGINX and others) will add a HTTP header (X-Forwarded-Host) with the real source IP address of the connection. So, MeshCentral knows that is need to use that IP address for logs, etc.

I am not sure there are any solutions for fixing the IP address with pass-thru forwarding of the TLS connection. I would have Apache offload the TLS and forward the connection with X-Forwarded-Host header added. I would then add the following to the settings section of config.json:

"tlsoffload": "127.0.0.1"

Note that "127.0.0.1" must be the IP address of the Apache reverse-proxy. When the connection comes from that IP address and contains a X-Forwarded-Host header, MeshCentral will know to use it, otherwise it's ignored.

Hope that helps. Ylian

[MordyT]

@Ylianst

So my Apache Reverse Proxy is on another machine (not the MC server).

When I enable "tlsoffload": "10.0.1.50" MC crashes and won't start. This shows in the log:

MeshCentral HTTP redirection server running on port 80.
MeshCentral v0.5.0-h, Hybrid (LAN + WAN) mode.
MeshCentral Intel(R) AMT server running on mesh.domain.tld:4433.
MeshCentral HTTP server running on port 443.
Failed to load web certificate at: "https://mesh.domain.tld/", host: "mesh.domain.tld"
Failed to load web certificate at: "https://mesh.domain.tld/", host: "mesh.domain.tld"
Error: Command failed: C:\Program Files\nodejs\node.exe C:\Program Files\Open Source\MeshCentral\winservice\winservice.js --launch 6016

    at ChildProcess.exithandler (child_process.js:301:12)
    at ChildProcess.emit (events.js:224:7)
    at maybeClose (internal/child_process.js:1027:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:285:5) {
  killed: false,
  code: 4294967295,
  signal: null,
  cmd: 'C:\\Program Files\\nodejs\\node.exe C:\\Program Files\\Open Source\\MeshCentral\\winservice\\winservice.js --launch 6016'
}
ERROR: MeshCentral failed with critical error, check MeshErrors.txt. Restarting in 5 seconds...

What additional logging can I review to find out why?

[Ylianst]

Can you look in "meshcentral-data/mesherrors.txt" to see if there is any other details there?

Also, can you change "certurl": "mesh.domain.tld" to "certurl": "10.0.1.50"?

[MordyT]

Unfortunately there is no mesherrors.txt created.

I've updated the certurl to the IP and it does start, but now I can't load the webpage. Even from the MC server, I get:

ERR_SSL_PROTOCOL_ERROR

(This is the same with the internal 10.0.1.50 or mesh.domain.tld, from the MC server, other PCs on LAN or external)

EDIT: It appears that when using TLSOffload, that 443 is now used for HTTP MeshCentral HTTP server running on port 443. vs MeshCentral HTTPS server running on mesh.domain.tld:443.

We run a hybrid (devices on LAN and WAN) MC server - and have split DNS to support this: (mesh.domain.tld internal answers at 10.0.1.51, external it's at 1.2.3.4)

How should this be setup?

[Ylianst]

So, you need to configure Apache to perform TLS on port 443 and forward the connection to MeshCentral without TLS. It looks like your just forwarding the TCP connection as-is in Apache which you should not be doing. You need to take a look at your Apache configuration.

[MordyT]

So I got it all working... But I think there is room for improvement here.

Once I understand that tlsoffloading basically means MC shuts down all forms of HTTPS, I was able to modify Apache to http:// passthrough (on port 443), wss to ws passthough (on port 443), etc.

Then I had to change my internal DNS to route all traffic to the proxy, which means all internal traffic goes out the WAN and comes back in (not great from a bandwidth perspective)

I get that TLSOffload means MC shouldn't do anything TLS related. My proposed change/enhancement is what spawned this... MC could use X-Forwarded-For (or similar) to get the real IP instead of having to offload TLS completely.

[Ylianst]

If Apache adds the "X-Forwarded-Host" header and the request comes from the IP address in "tlsoffload", then MeshCentral should use the IP address specified in X-Forwarded-Host.

[MordyT]

If Apache adds the "X-Forwarded-Host" header and the request comes from the IP address in "tlsoffload", then MeshCentral should use the IP address specified in X-Forwarded-Host.

Correct, it does that today. I'm proposing that if Apache adds the X-Forwarded-For and tlsoffload isn't used that mesh still uses that IP.

Perhaps a variable "trustedipprovider" could be added and MC, without TLS Offload, will trust the IPs provided from that.

I'll draw this up in a few minutes and send it to you.

EDIT: Blue is current, Green is proposed

[Ylianst]

Well, I can add that, it would not be difficult. I want to make it super clear that the reverse proxy would have to terminate TLS, add the header and re-encrypt using TLS to send the traffic to MeshCentral. I will add it because, maybe there is a need for this, but generally you don't need to re-apply TLS between the reverse proxy and MeshCentral.

If the TLS connection is directly forwarded to MeshCentral, then there is no way for Apache to add the "X-Forwarded-Host" header. Even worst, a bad agent could insert a "X-Forwarded-Host" header and foul the server into thinking it's coming from a different IP address.

I imagine one would not generally undo TLS, process and redo-TLS in the reverse proxy?

[Ylianst]

Ha. I just looked at the code and I already have it. Instead of:

"tlsoffload": "172.16.1.50"

do this:

"trustedproxy": "172.16.1.50"

That will cause x-forwarded-for to be trusted. Let me know if that works.

[MailYouLater]

Once I understand that tlsoffloading basically means MC shuts down all forms of HTTPS, I was able to modify Apache to http:// passthrough (on port 443), wss to ws passthough (on port 443), etc. Then I had to change my internal DNS to route all traffic to the proxy, which means all internal traffic goes out the WAN and comes back in (not great from a bandwidth perspective)

Wait a second, from your description it sounds like the reverse proxy is out on the internet somewhere rather than being on the same LAN as your server. If I'm not misunderstanding what your setup is like, dropping to http between the reverse proxy and the server could also be a huge security issue and you should definitely be using "trustedproxy": "<ip address>" instead of "tlsoffload": "<ip address>".

[MordyT]

@Ylianst I will have to try this and get back tomorrow to you on it. I had disabled the proxy because of CIRA not working (It wasn't the proxy, I have other CIRA issues) and will have to schedule time to bring it back behind the proxy.

@MailYouLater 2 things: 1) I try to go as secure as possible. ZeroTrust means I don't even trust the LAN. So yes, Proxy is on same LAN as MC server, but I don't want unencrypted traffic from proxy to MC just in case someone else on LAN is malicious. 2) If you run a hybrid setup (agents on both LAN and WAN), and TLS is on proxy, then agents need to go through proxy. So it increases traffic as the agents need to exit LAN and re-enter on WAN side.

[OutbackMatt]

Trusted proxy works for me, using Apache reverse proxy and SSL - thanks

[MordyT]

I'm going to close this based on OutbackMatt's feedback and will open a new request if I have issues. I won't be able to test for some time for unrelated reasons.