Terminal Emulation on Windows without ConPTY results in ANSI Escape Code flooding

[rowds15]

Describe the bug Terminal Emulation on Windows 7 and Server 2008R2 not rendering correctly

To Reproduce Steps to reproduce the behavior:

1. Go to 'Open Terminal'
2. See error

Expected behavior A clear and concise description of what you expected to happen.

Server Software (please complete the following information):

• OS: [e.g. AWS OS]
• Network: LAN/WAN, Nginx reverse proxy
• Version: [e.g. 1.0.71]
• Node: [e.g. 16.17.0]

Client Device (please complete the following information):

• Device: Laptop
• OS: Windows 10
• Network: Local to Meshcentral, Remote over WAN]
• Browser: Firefox & Google Chrome]

Remote Device (please complete the following information):

• Device: PC/Server
• OS: Windows 7 and Server 2008
• Network: Local to Meshcentral, Remote over WAN]
• Current Core Version (if known): Apr 4 2022, 3863795360 & Mar 25 2022, 3863795360

Issue started with version 1.0.70 and yes I know Windows 7 and server 2008 are EOL but.....

[sblosser]

Experiencing this as well.

[krayon007]

The main issue with these two OS is they do not support MSFT's pseudo console API (ConPTY) which is why the behavior is wierd because we had to do some hacks to fake the terminal. But some apps aren't tolerable to how it works, which is why it renders incorrectly at time

What specific app are you running, and how does it render?

[Ylianst]

The screen shot above does not show any applications. It's just the Windows command shell.

[rowds15]

I'm not running any Applications it's just the command shell

this is what types out when i type ipconfig [4;29H[0mi[4;30H[0mp[4;31H[0mc[4;32H[0mo[4;33H[0mn[4;34H[0mf[4;35H[0mi[4;36H[0mg[6;1H

[MC-PM]

Same is happening since a couple of days on several Windows 10 and Windows Server 2012 devices. CMD is appearing kind of missing layout. This is also happening on the PS option:

[krayon007]

Can you run the osinfo console command, for your win 10 device that is messed up. I'm guessing it says ConPTY: No?

[rowds15]

osinfo from working and non working os's

OS: Microsoft Windows Server 2012 R2 Standard - 9600 [ConPTY: NO] OS: Microsoft Windows Server 2008 R2 Standard - 7601 [ConPTY: NO] OS: Microsoft Windows 7 Professional - 7601 [ConPTY: NO] OS: Microsoft Windows 10 Pro - 21H2/19044 [ConPTY: YES]

[Matt-CyberGuy]

We've started noticing this issue as well more and more

I've definitely seen it on a few Windows 10 systems, and Windows 2019 server

[adsgreen]

It used to be "Ok" - whilst not perfect it could be used for basic tasks as you only generally encountered an issue when text scrolled off the bottom of the screen. However now it's not usable with the control codes being output.

[krayon007]

This is really interesting. I can't reproduce it right now on any of my machines that match match what you guys are running. It looks like it is an issue with the xterm environment in the browser, as that is what is supposed to be interpreting those vt100 codes. I'm looking into it to see if I can find a system that has this issue, then I can test with different server versions to see what happened, or if its a config issue, etc.

[krayon007]

Does it make any difference if you append the following query string to the url in your browser, before connecting the terminal?

&xterm=0

[sblosser]

Does it make any difference if you append the following query string to the url in your browser, before connecting the terminal?

&xterm=0

I checked a few systems and this did not help.

[adsgreen]

Same here. Didn’t help and I tried a number of clients including mobile.

[krayon007]

I'm curious if all the people having issue with this, are all running reverse proxies of some type? I'm having a hard time trying to reproduce this issue. I even built multiple test clients running the same versions of windows, and tried the versions of the servers mentioned above. For these systems, the agent side is very simplistic in how it sends the escape codes. It almost seems like something in between is escaping the escape codes... Are all the clients affected on your guys network, or just these windows machines that lack ConPTY?

[si458]

@krayon007 ive just tried an old Windows 7 Pro pc we still have around and im experiencing the same, it just pours our garbage, i am not using a proxy either, im running meshcentral directly on 80+443 and its handling LE too, the server is external ip, and the remote client is also external, im using hybrid mode, also 1.0.85 MC Agent Version: Aug 25 2022, 1360394467 OS: Microsoft Windows 7 Professional - 7601 [ConPTY: NO]

[sblosser]

No reverse proxies here. I'm running the same config as @si458.

[adsgreen]

No reverse proxy here either. running the letsencrypt https service.

it doesn’t impact all our clients

[krayon007]

Ok, can someone try this:

Using Firefox, go to the terminal tab, then hit F-12 to bring up the browser console... Then select network Then click connect on the terminal... Then in the network tab of Firefox, select the one that says meshrelay, which should be one of the first ones. Then on the right side, select response

And then show me what it receives on the websocket... I want to see if the escape codes are arriving correctly, or if they are escaped, etc... Here's what it looks like from one of my Windows machines that lacks ConPTY:

[sblosser]

Here is what a WIndows Server 2016 (no ConPTY) looks like (agent branding/folder redacted):

[sblosser]

In case it is needed, here is the output from a Server 2019 (with ConPTY):

[krayon007]

So from the same browser, some agents work and some dont? This is very interesting, becuase your output looks identical to mine. Seems like the escape codes are arriving correctly, but for some reason aren't getting interpreted correctly by the web application... I'll have to dig deeper...

[si458]

@krayon007 ok im seeing the same as @sblosser for my Windows 7 machine

[krayon007]

Can you click on one of the entries, so it shows the raw data? Like this:

[si458]

[1;1H[0mMicrosoft Windows [Version 6.1.7601]                                            
[2;1H[0mCopyright (c) 2009 Microsoft Corporation.  All rights reserved.                 
[3;1H[0m                                                                                
[4;1H[0mC:\Program Files\Hestor Ltd\Mesh Agent>                                         

any use?

[adsgreen]

When connecting to terminal I get this in the browsers network debug on two messages: OS: Microsoft Windows 10 Pro - 21H2/19044 [ConPTY: YES]

00000000: 1b5b 324a 1b5b 6d1b 5b48 4d69 6372 6f73  .[2J.[m.[HMicros
00000001: 6f66 7420 5769 6e64 6f77 7320 5b56 6572  oft Windows [Ver
00000002: 7369 6f6e 2031 302e 302e 3139 3034 342e  sion 10.0.19044.
00000003: 3138 3839 5d1b 5d30 3b43 3a5c 5749 4e44  1889].]0;C:\WIND
00000004: 4f57 535c 5379 7374 656d 3332 5c63 6d64  OWS\System32\cmd
00000005: 2e65 7865 071b 5b3f 3235 68              .exe..[?25h

00000000: 0d0a 2863 2920 4d69 6372 6f73 6f66 7420  ..(c) Microsoft 
00000001: 436f 7270 6f72 6174 696f 6e2e 2041 6c6c  Corporation. All
00000002: 2072 6967 6874 7320 7265 7365 7276 6564   rights reserved
00000003: 2e0d 0a1b 5b34 3758 0d0a 433a 5c50 726f  ....[47X..C:\Pro
00000004: 6772 616d 2046 696c 6573 5c4d 6573 6820  gram Files\Mesh 
00000005: 4167 656e 743e 1b5b 3139 581b 5d30 3b41  Agent>.[19X.]0;A
00000006: 646d 696e 6973 7472 6174 6f72 3a20 433a  dministrator: C:
00000007: 5c57 494e 444f 5753 5c53 7973 7465 6d33  \WINDOWS\System3
00000008: 325c 636d 642e 6578 6507                 2\cmd.exe.

However on the node that has weird behaviour : OS: Microsoft Windows 10 Enterprise 2016 LTSB - 14393 [ConPTY: NO]

00000000: 5b32 3b31 485b 306d 2863 2920 3230 3136  [2;1H[0m(c) 2016
00000001: 204d 6963 726f 736f 6674 2043 6f72 706f   Microsoft Corpo
00000002: 7261 7469 6f6e 2e20 416c 6c20 7269 6768  ration. All righ
00000003: 7473 2072 6573 6572 7665 642e            ts reserved.

00000000: 5b34 3b31 485b 306d 433a 5c50 726f 6772  [4;1H[0mC:\Progr
00000001: 616d 2046 696c 6573 5c4d 6573 6820 4167  am Files\Mesh Ag
00000002: 656e 743e                                ent>

And this renders on the terminal browser screen as: [1;1H[0mMicrosoft Windows [Version 10.0.14393] [2;1H[0m(c) 2016 Microsoft Corporation. All rights reserved. [3;1H[0m [4;1H[0mC:\Program Files\Mesh Agent> [5;1H[0m [6;1H[0m [7;1H[0m [8;1H[0m [9;1H[0m [10;1H[0m [11;1H[0m [12;1H[0m [13;1H[0m [14;1H[0m [15;1H[0m [16;1H[0m [17;1H[0m [18;1H[0m [19;1H[0m [20;1H[0m [21;1H[0m [22;1H[0m [23;1H[0m [24;1H[0m [25;1H[0m [1;1H[0mMicrosoft Windows [Version 10.0.14393][2;1H[0m(c) 2016 Microsoft Corporation. All rights reserved.[4;1H[0mC:\Program Files\Mesh Agent>

[krayon007]

It looks like the ESC character got dropped... I'll have to check to see if I can figure out where it got dropped...

[KommunistGangsta]

I had this problem, after installing the 1.0.85 version in a Windows Server. Restarted the Mesh Server, and it seems to have fixed the problem in my end...

[DirectITServicesUK]

Same. This is what I get typing "ipconfig"

[krayon007]

Does anyone have a system experiencing this issue that they are willing to grant remote access to me, so I can try to remote troubleshoot what is going on? I can't reproduce this issue with any of my systems, so I'm not sure where/how/why the ESC character is getting dropped. If anyone would like to volunteer, please email me :)

[Matt-CyberGuy]

I have a few I can give you access to, was going to DM you, but I guess github doesn't have that feature?

[krayon007]

You can click on my profile, and email me directly.

git.bryan.roe@techsavvydude.com

[Matt-CyberGuy]

Got it... I added you in our portal. It should have hopefully sent you an invite. You might need to check your junk mail. I added a low-production 2012 server that's experiencing the issue.

[krayon007]

@Matt-CyberGuy can you email me your config.json for your server? Feel free to redact anything sensitive. I connected to your test system, and I installed another instance of the same agent, but configured to talk to my server... Since that system does NOT have ConPTY, the two agents will connect to the same console... The agent connected to your server is dropping the ESC character, and displaying the terminal with the VT100 codes all over the place... However, the agent that is connected to my server, using the same console instance, does not have this behavior:

This is very interesting...

[Matt-CyberGuy]

I think I got all the secret stuff out. Below is our current config: I don't know if it matters for this or not, but the TlsOffload address is our internal NGinx reverse proxy address.

{
    "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
    "settings": {
        "_WANonly": true,
        "_LANonly": true,
        "_sessionKey": "MyReallySecretPassword1",
        "port": 443,
        "aliasPort": 443,
        "redirPort": 80,
        "_redirAliasPort": 80,
        "ignoreagenthashcheck": true,
        "cert": "remote.CompanyName.com",
        "WebRTC": true,
        "selfupdate": false,
        "noagentupdate": true,
        "BrowserPong": 90,
        "AgentPong": 90,
        "TlsOffload": "172.0.0.101",
        "RelayDNS":[
            "1.remote.CompanyName.com",
            "2.remote.CompanyName.com",
            "3.remote.CompanyName.com"
            ],

        "autoBackup": {
          "backupIntervalHours": 24,
          "keepLastDaysBackup": 30,
          "backupPath": "meshcentral-backup"
        }
    },
    "domains": {
    "": {
        "mstsc": true,
        "agentCustomization": {
            "displayName": "CompanyName Remote Agent",
            "description": "CompanyName remote agent for remote monitoring, management and assistance.",
            "companyName": "CompanyName",
            "serviceName": "CompanyName-Remote",
            "fileName": "CompanyName-Remote",
            "image": "agent-logo.png"
        },
        "agentFileInfo": {
            "icon":  "CompanyName-Remote.ico",
            "fileDescription": "CompanyName remote agent for remote monitoring, management and assistance.",
            "legalCopyright":  "CompanyName Remote Agent (c)",
            "originalFilename":  "CompanyName-Remote.exe",
            "productName": "CompanyName Remote Agent"
        },
        "altMessenging": {
            "name": "Video Chat",
            "url": "https://meet.CompanyName.com/{0}"
        },
        "terminal": {
            "launchCommand": {
            "linux": ""
            }
        },
        "authStrategies": {        
            "azure": {
                "newAccounts": true,

            }
        },
        "welcomePicture": "CompanyName-wallpaper.jpg",
        "welcomePictureFullScreen": true,
        "loginPicture": "welcome-logo.png",
        "title": "CompanyName",
        "title2": "<br/><h2>Remote Monitoring Portal</h2>",
        "welcomeText": "Please enter your email and given password above.<br/>If you are experiencing difficulty connecting to endpoints, need to add/remove users or systems, <br/>or you would like to have your password reset, please contact us at <a href='mailto:support@CompanyName.com'>support@CompanyName.com</a>.",
        "titlePicture": "logo.png",
        "minify": true,
        "newAccounts": false,
        "userNameIsEmail": true,
        "PasswordRequirements": { 
            "min": 8,
            "max": 64,
            "upper": 1,
            "lower": 1,
            "numeric": 1,
            "nonalpha": 1,
            "hint": true,
            "reset": 90,
            "_force2factor": true
        },
        "ManageAllDeviceGroups":["email@addresses.com"],
        "Footer": "<a target=_blank href='https://CompanyName.com'>CompanyName</a>",
        "_agentInviteCodes": true,
        "certUrl": "https://remote.CompanyName.com",
        "_userAllowedIp": "ExternalListCurrentlyNotWorking.txt"
        }
    },
     "smtp": {
         "host": "smtp.companyname.com",
         "port": 587,
         "from": "No-Reply | CompanyName Remote Portal <no-reply@CompanyName.com>",
         "user": "CompanyNamegroup@email.com",

         "tls": false
     },
    "_letsencrypt": {
        "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
        "email": "support@CompanyName.com",
        "names": "remote.CompanyName.com",
        "production": true
    }
}

[adsgreen]

Worth confirming the sever versions as this used to work with the same node and server and I think a recent update may have caused this.

[krayon007]

Worth confirming the sever versions as this used to work with the same node and server and I think a recent update may have caused this.

The server I had it connect to is 1.0.85. I'm not sure what server version Matt was running that the agent connected to on his side.

[Matt-CyberGuy]

I can confirm the issue also seemed to be version related for us. We are running 1.0.85 currently, but I believe this has been affecting us for awhile.

I actually run a personal mesh server with a more basic config file and noticed the issue occurring on it as well. If it's helpful Bryan, I can start forcing the version down on my personal lab server to see which version the ESC characters started dropping off.

[rowds15]

for me the Issue started with version 1.0.70

[Matt-CyberGuy]

Ok... I can also confirm, when I rolled one of my personal mesh server back to 0.9.77, terminal for Windows 7 began functioning correctly.

I don't think I have a way to update to in between versions, so will have to take @rowds15 word for non-ConPTY terminal functioning in that version

[krayon007]

Interesting. I'll do some testing with these versions to see if I can spot something. Just for giggles I'll start with an older version and start upgrading to newer versions, as maybe something in the update process screwed something up.

[silversword411]

It will...of course....but something ridiculously inconsequential...like a comma, or a slash in the wrong place 🤣

[krayon007]

All of you guys that are having issues.... Do you all have minify set to true?

[silversword411]

"Minify": 1,

was in my config.

Removing that line...made no difference on my Windows Server 2012 R2 terminal...still had the [21;1H[0m 's on terminal Admin Shell session

[krayon007]

I figured it out. It was indeed related to minification. I'm not sure if when you restarted your server, if minification was undone, it may have still got the setting from the db. I think you need to set it to 0, not remove it...

But anyways, the problem was that when win-terminal.js was minified, it for some reason removed the ESC character sequence. I'll have to play around with the minifier he used to get it to not do that...

But anyways, there are a few ways to work around this... Either disable minification.... Or, go to your server, go to the agents folder, and go to the modules_meshcore_min folder. In there, you'll find the problem win-terminal.min.js

If you edit that file, search for: function GetEsc(e, t)

and you'll see the definition as: return Buffer.from("[" + t.join(";") + e)

you'll need to change it to: return Buffer.from("\x1B[" + t.join(";") + e)

Or, alternately, you can just copy agents\modules_meshcore\win-terminal.js and save it into: agents\modules_meshcore_min\win-terminal.min.js

[krayon007]

On a side note, I tested this on Windows Server 2012 R2. Ylian made changes to the minifier a few versions ago, so this is probably how it broke in a recent version.

[si458]

@krayon007 I don't use minify and I have this issue? Are u sure it's a minify issue?

[krayon007]

@krayon007 I don't use minify and I have this issue? Are u sure it's a minify issue?

Well, in my case, I was never able to reproduce this issue, until I turned minification on. Then I was able to get it to go back and forth between working and not working, with the above mentioned change. From a usage point of view, everything seems to line up. The ESC sequence was deleted by the minifier, and the minifier was modified in version 1.0.70, which is when people reported this issue happening. And looking at the logs people posted earlier, looking at the raw sequences, it looked like the ESC character was missing, which would explain why the VT100 codes simply printed to the output, instead of being eaten by the console.

If you go to your server, do you have an agents\modules_meshcore_min folder? If you never enabled minification, that folder should not exist.

[si458]

If you go to your server, do you have an agents\modules_meshcore_min folder? If you never enabled minification, that folder should not exist.

the folder does exists at /home/simon/meshcentral/node_modules/meshcentral/agents/modules_meshcore_min but i dont have minify listed in my config.json ? and never enabled it?

  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "meshcentral.myserver.com",
    "port": 443,
    "noAgentUpdate": 1,
    "redirPort": 80,
    "AgentPong": 300,
    "authLog": "authlog.log",
    "relayDNS": "relay.meshcentral.myserver.com",
    "autoBackup": {
      "backupIntervalHours": 12,
      "keepLastDaysBackup": 7,
      "backupPath": "meshcentral-backup"
    }
  },
  "domains": {
    "": {
      "agentInviteCodes": true,
      "title": "myserver",
      "agentCustomization": {
        "companyName": "myserverLtd",
        "image": "128x128.png"
      },
      "assistantCustomization": {
        "image": "128x128.png"
      },
      "androidCustomization": {
        "image": "128x128.png"
      }
    }
  },
  "letsencrypt": {
    "email": "simon@myserver.com",
    "names": "meshcentral.myserver.com,relay.meshcentral.myserver.com",
    "skipChallengeVerification": false,
    "production": true
  }
}

which network url request should i be inspecting/looking out for with firefox? as it never asks for win-terminal.min.js ?

EDIT: on a fresh npm install meshcentral in a clean folder, its downloading the agents\modules_meshcore_min folder?

[krayon007]

which network url request should i be inspecting/looking out for with firefox? as it never asks for win-terminal.min.js ?

You won't see that in the browser, becuase it's used by the agent, not the browser. As far as npm, I'm not sure what the default state is from npm, but I know when pulled the server from GIT, it was never there... One way to test if it is a screwed up minification, is to copy the win-terminal.js file from the non minified folder on your server, and place it on your agent manually, in the same folder as the agent... Then either restart the agent, or go to the console tab, and select "clear core", and "load default core".... The agent should use the copy that you manually copied over if the timestamp is newer.

But like I said, maybe it will work, if you specify minify in your config, and specify the value as false.

[krayon007]

If you wanted to test what your agent was actually using, to verify your server config... Copy the modules\dbTool.js file from the agent repository, and place it in the same folder as your agent... Then run the following from an elevated command prompt: MeshAgent dbTool.js export CoreModule This will export the meshcore the agent received from the server, into a file called CoreModule.js. Open it in a text editor... If the file is human readable, it is not minified. If the whole file is on a single line, it was minified.