Issues with agent functionality on IIS reverse proxy setup

[chk-mytoys]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior: Create and deploy a device of type agent and deploy it. The device will connect and appear in mesh cental- but only the tabs general, events, detail, console is shown. No details is shown, no desktop sharing is available. Verified with Windows & Mac agent. Also: No output is given for commands in the console windows - except when the tiny core is active

Mesh Central Assistant is not affected and works correctly.

Expected behavior Full functionality is available with the agent.

Screenshots

Server Software (please complete the following information):

• OS: Alpine Linux
• Virtualization: Docker
• Network: IIS reverse proxy with Lets Encrypt Certificates and SSL Offloading
• Version: 1.1.21
• Node: whatever is in the image

Client Device (please complete the following information):

• Device: Laptop
• OS: Windows, Mac OS
• Network: Remote over WAN
• Browser: [e.g. Google Chrome]
• MeshCentralRouter Version: [if applicable]

Remote Device (please complete the following information):

• Device: Laptop
• OS:Windows, Mac OS
• Network: Remote over WAN
• Current Core Version (if known): querying does not work

• Additional context

  {
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
  "plugins":{"enabled": true},
  "_mongoDb": null,
  "cert": "mc.mydoamain.net",
  "_WANonly": true,
  "_LANonly": true,
  "sessionKey": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "port": 8086,
  "aliasPort": 443,
  "redirPort": 80,
  "_redirAliasPort": 80,
  "AgentPong": 300,
  "TLSOffload": "192.168.6.5",
  "SelfUpdate": false,
  "AllowFraming": false,
  "WebRTC": false
  },
  "domains": {
  "": {
    "_title": "MyServer",
    "_title2": "Servername",
    "minify": true,
    "NewAccounts": true,
    "localSessionRecording": false,
    "_userNameIsEmail": true,
    "certUrl": "https://192.168.6.5:46443"
  }
  },
  "_letsencrypt": {
  "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
  "_email": "myemail@mydomain.com",
  "_names": "myserver.mydomain.com",
  "production": false
  }
  }

Switch to Tinycore:

[chk-mytoys]

Problem seems to be checking or deploying the mesh core from the server over the reverse proxy. If I begin without reverse proxy, add the agent host and let it start - it works. If I then switch to the reverse proxy configuration, it continue to work for clients that already had the core deployed.

[si458]

one thing to try is the following

1. go to a device in web ui and into the console tab
2. click 'Agent Actionand runClear the core`
3. wait about 60 seconds
4. then click 'Agent Action' and run Upload Default Server Core
5. wait about 60 seconds
6. in theory the tabs should re-appear!

sometimes the meshcore.js can get itselfs confused, so simply removing the core, waiting and re-uploading can fix the issue

[si458]

also your certUrl seems weird? the certUrl should be the value of your domain you are using so meshcentral KNOWS where to get the certificate and what to expect from your reverse proxy https://192.168.6.5:46443 -> https://mc.mydomain.net

[chk-mytoys]

one thing to try is the following

1. go to a device in web ui and into the console tab
2. click 'Agent Actionand runClear the core`
3. wait about 60 seconds
4. then click 'Agent Action' and run Upload Default Server Core
5. wait about 60 seconds
6. in theory the tabs should re-appear!

sometimes the meshcore.js can get itselfs confused, so simply removing the core, waiting and re-uploading can fix the issue

Already tried that multiple times and in all imaginable combinations - it does not work. Only the tinycore can be loaded - never the full/"fat" one.

Agent says: Timeout waiting for Server, launching cached meshcore...

[chk-mytoys]

also your certUrl seems weird? the certUrl should be the value of your domain you are using so meshcentral KNOWS where to get the certificate and what to expect from your reverse proxy https://192.168.6.5:46443 -> https://mc.mydomain.net (It has it'S own port, just to be sure that he can only get the correct certificate from the many ones present on the reverse proxy).

That should be correct I think and is also working as expected. He gets the outer/reverse proxy cert from this address. I also tried entering the outer URL there - there was no change in behaviour.

[si458]

so is the url you use to access meshcentral like this https://mc.mydomain.net ? no extra ports or anything?

and your network similar to this? internet (mc.mydomain.net) -> IIS PROXY (using port 443) -> MESHCENTRAL (192.168.6.99:8086)

i dont see why the certurl includes the port 46443 ?

[chk-mytoys]

so is the url you use to access meshcentral like this https://mc.mydomain.net ? no extra ports or anything?

and your network similar to this? internet (mc.mydomain.net) -> IIS PROXY (using port 443) -> MESHCENTRAL (192.168.6.99:8086)

i dont see why the certurl includes the port 46443 ?

Yes, that is the setup. The port is not strictly necessary - on port 443 there is SNI an multiple certificates depending on the SNI. On 46443 there is only the MC-certificate. I had 443 previously there - and there is no difference in functionality (e.g. same bug).

[chk-mytoys]

FYI - importing the core locally with MeshAgent.exe dbTool.js import CoreModule also "works" - all functionality is there, but core update does not work. I could also verify that the MeshCore is sent out by the reverse proxy - the problem seems to be the agent not processing it.

[chk-mytoys]

It seems to be a bug in the agentcore.c when trying to reassemble larger requests in 65k chunks. After setting the chunk size zo 650kb (allowing the file to be consumed in one go) - it works.

Likely the clients relies on some idiosyncrasies of the nodeJS web server module to work in standalone mode.

[si458]

I still believe this is a reverse proxy issue, as other people use trafeik, nginx, apache, etc... and they all work? You even said urself if u remove the proxy and use it directly it works... So it might be IIS isn't passing the data in chunks correctly

[chk-mytoys]

Well could be, but when waying the odds between ISS ARR having a fundamental issue and a bug in a smaller project with comments like "// TODO: Confirm with Bryan that this is how partial data works" in the sections with the issue - I think the MeshCentral is the more likely culprit. :)

Could for example be, that the other servers compress the http stream and because of that do not reach the 65kb limit,.

[chk-mytoys]

The problem seems to be, hat the state machine in agentcore.c / MeshServer_OnResponse:

Seems to go: ILibWebClient_ReceiveStatus_Partial ... ILibWebClient_ReceiveStatus_Partial ILibWebClient_ReceiveStatus_LastPartial

and then there should be a:

ILibWebClient_ReceiveStatus_MoreDataToBeReceived (in reality means: no more data to be recieved)

... but that is never fired.

[chk-mytoys]

I analyzed it a little further - an the problem is:

• The agent implementation for fragmentation is not finished and does not work
• The the nodeJS webserver does not seem to observe the fragmentation limit and the agent does not enforce it - IIS does

Options:

1. Fix the agent (agentcore.c / MeshServer_OnResponse)
2. Raise the buffer limit (agentcore.c / MeshServerConnectEx / Line 4148) for example to 1-2Mb as a temporary workarround

[si458]

Both options would require new agent builds which won't be anytime soon

But if u qant to do new builds urself and do some testing then plz do!

If u think ur patch works then submit a PR in the meshagent repo and I can look at it for you as we would need to check every agent works with the PR

[inzi]

FWIW - I run meshcentral behind ARR without issue, except for a few things like file distribution which I've just not spent the time to resolve.

It's been a while since I set it up, but IIRC, I had to configure MC to know it was behind a proxy, I setup the ARR mappings, and URL rewrite rules inbound and outbound.

[inzi]

@chk-mytoys - you might search issues for my posts, which are likely closed at this point.

[inzi]

My ARR setup is as follows, YMMV.

You'll need to replace some variables I used for sanitization purposes:

YOUR_MC_SERVER_IP_HERE -> The IP of your MC server behind ARR (ex: 10.0.0.1) YOUR_MC_SERVER_HERE -> The IP of your MNC server behind ARR (ex: 10.0.0.1) YOUR_URL_HERE -> Your public URL (ex: https://meshcentral.example.com )

Here are my http and https web configurations.

Here is the http web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <clear />
                <rule name="AlwaysToUpgrade" enabled="false">
                    <match url="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    </conditions>
                    <serverVariables>
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="None" />
                </rule>
                <rule name="ReverseProxyInboundRule1" enabled="true" stopProcessing="false">
                    <match url="(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                    <serverVariables>
                        <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                        <set name="HTTP_X_Fowarded_Proto" value="http" replace="false" />
                        <set name="HTTP_X_Forwarded_Host" value="{SERVER_NAME}:{SERVER_PORT}" replace="false" />
                        <set name="HTTP_CF_Connecting_IP" value="{REMOTE_ADDR}" replace="false" />
                        <set name="HTTP_CONNECTION" value="upgrade" />
                        <set name="ORIGINAL_URL" value="{HTTP_HOST}" />
                    </serverVariables>
                    <action type="Rewrite" url="http://YOUR_URL_HERE:88/{R:1}" />
                </rule>
            </rules>
            <outboundRules>
                <clear />
                <rule name="Rewrite Location Header" preCondition="IsRedirection" stopProcessing="false">
                    <match filterByTags="None" serverVariable="RESPONSE_Location" pattern="^http://[^/]+/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="https://{ORIGINAL_URL}" />
                </rule>
                <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding" enabled="true">
                    <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
                </rule>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1" stopProcessing="false">
                    <match filterByTags="A, Form, Img" pattern="^http(s)?://YOUR_MC_SERVER_IP_HERE:88/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="http{R:1}://YOUR_URL_HERE/{R:2}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="NeedsRestoringAcceptEncoding">
                        <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
                    </preCondition>
                    <preCondition name="HOST_HasValue">
                        <add input="{HOST}" pattern="^(.*)" />
                    </preCondition>
                    <preCondition name="IsRedirection">
                        <add input="{RESPONSE_STATUS}" pattern="3\d\d" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
        <tracing>
            <traceFailedRequests>
                <add path="*">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI,WebSocket" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions statusCodes="500" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

Here is the https web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <clear />
                <rule name="AlwaysToUpgrade" enabled="true">
                    <match url="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    </conditions>
                    <serverVariables>
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="None" />
                </rule>
                <rule name="ReverseProxyInboundRule1" enabled="true" stopProcessing="false">
                    <match url="(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                    <serverVariables>
                        <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                        <set name="HTTP_X_Fowarded_Proto" value="https" replace="false" />
                        <set name="HTTP_X_Forwarded_Host" value="{SERVER_NAME}:{SERVER_PORT}" replace="false" />
                        <set name="HTTP_CF_Connecting_IP" value="{REMOTE_ADDR}" replace="false" />
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="Rewrite" url="http://YOUR_MC_SERVER_HERE/{R:1}" />
                </rule>
            </rules>
            <outboundRules>
                <clear />
                <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding" enabled="true">
                    <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
                </rule>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1" stopProcessing="false">
                    <match filterByTags="A, Form, Img" pattern="^http(s)?://YOUR_MC_SERVER_IP_HERE:480/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="https://YOUR_URL_HERE/{R:2}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="NeedsRestoringAcceptEncoding">
                        <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
                    </preCondition>
                    <preCondition name="HOST_HasValue">
                        <add input="{HOST}" pattern="^(.*)" />
                    </preCondition>
                    <preCondition name="IsRedirection">
                        <add input="{RESPONSE_STATUS}" pattern="3\d\d" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
        <tracing>
            <traceFailedRequests>
                <add path="*">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI,WebSocket" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions statusCodes="500" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

HTH

[chk-mytoys]

I tried your configuration - but it is still not possible to sucessfully upload the agent core with this configuration.

My ARR setup is as follows, YMMV.

You'll need to replace some variables I used for sanitization purposes:

YOUR_MC_SERVER_IP_HERE -> The IP of your MC server behind ARR (ex: 10.0.0.1) YOUR_MC_SERVER_HERE -> The IP of your MNC server behind ARR (ex: 10.0.0.1) YOUR_URL_HERE -> Your public URL (ex: https://meshcentral.example.com )

Here are my http and https web configurations.

Here is the http web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <clear />
                <rule name="AlwaysToUpgrade" enabled="false">
                    <match url="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    </conditions>
                    <serverVariables>
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="None" />
                </rule>
                <rule name="ReverseProxyInboundRule1" enabled="true" stopProcessing="false">
                    <match url="(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                    <serverVariables>
                        <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                        <set name="HTTP_X_Fowarded_Proto" value="http" replace="false" />
                        <set name="HTTP_X_Forwarded_Host" value="{SERVER_NAME}:{SERVER_PORT}" replace="false" />
                        <set name="HTTP_CF_Connecting_IP" value="{REMOTE_ADDR}" replace="false" />
                        <set name="HTTP_CONNECTION" value="upgrade" />
                        <set name="ORIGINAL_URL" value="{HTTP_HOST}" />
                    </serverVariables>
                    <action type="Rewrite" url="http://YOUR_URL_HERE:88/{R:1}" />
                </rule>
            </rules>
            <outboundRules>
                <clear />
                <rule name="Rewrite Location Header" preCondition="IsRedirection" stopProcessing="false">
                    <match filterByTags="None" serverVariable="RESPONSE_Location" pattern="^http://[^/]+/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="https://{ORIGINAL_URL}" />
                </rule>
                <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding" enabled="true">
                    <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
                </rule>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1" stopProcessing="false">
                    <match filterByTags="A, Form, Img" pattern="^http(s)?://YOUR_MC_SERVER_IP_HERE:88/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="http{R:1}://YOUR_URL_HERE/{R:2}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="NeedsRestoringAcceptEncoding">
                        <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
                    </preCondition>
                    <preCondition name="HOST_HasValue">
                        <add input="{HOST}" pattern="^(.*)" />
                    </preCondition>
                    <preCondition name="IsRedirection">
                        <add input="{RESPONSE_STATUS}" pattern="3\d\d" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
        <tracing>
            <traceFailedRequests>
                <add path="*">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI,WebSocket" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions statusCodes="500" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

Here is the https web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <clear />
                <rule name="AlwaysToUpgrade" enabled="true">
                    <match url="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                    </conditions>
                    <serverVariables>
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="None" />
                </rule>
                <rule name="ReverseProxyInboundRule1" enabled="true" stopProcessing="false">
                    <match url="(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                    <serverVariables>
                        <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                        <set name="HTTP_X_Fowarded_Proto" value="https" replace="false" />
                        <set name="HTTP_X_Forwarded_Host" value="{SERVER_NAME}:{SERVER_PORT}" replace="false" />
                        <set name="HTTP_CF_Connecting_IP" value="{REMOTE_ADDR}" replace="false" />
                        <set name="HTTP_CONNECTION" value="upgrade" />
                    </serverVariables>
                    <action type="Rewrite" url="http://YOUR_MC_SERVER_HERE/{R:1}" />
                </rule>
            </rules>
            <outboundRules>
                <clear />
                <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding" enabled="true">
                    <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
                </rule>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1" stopProcessing="false">
                    <match filterByTags="A, Form, Img" pattern="^http(s)?://YOUR_MC_SERVER_IP_HERE:480/(.*)" />
                    <conditions logicalGrouping="MatchAll" trackAllCaptures="true" />
                    <action type="Rewrite" value="https://YOUR_URL_HERE/{R:2}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="NeedsRestoringAcceptEncoding">
                        <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
                    </preCondition>
                    <preCondition name="HOST_HasValue">
                        <add input="{HOST}" pattern="^(.*)" />
                    </preCondition>
                    <preCondition name="IsRedirection">
                        <add input="{RESPONSE_STATUS}" pattern="3\d\d" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
        <tracing>
            <traceFailedRequests>
                <add path="*">
                    <traceAreas>
                        <add provider="ASP" verbosity="Verbose" />
                        <add provider="ISAPI Extension" verbosity="Verbose" />
                        <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI,WebSocket" verbosity="Verbose" />
                    </traceAreas>
                    <failureDefinitions statusCodes="500" />
                </add>
            </traceFailedRequests>
        </tracing>
    </system.webServer>
</configuration>

HTH