HW Connect fails on AMT version 15.0.42

[4pack]

Describe the bug Attempting to use HW Connect on AMT version 15.0.42 fails. When clicking HW Connect, the screen goes to Setup, and then just disconnects immediately. This happens in both the desktop tab as well as under Remote Desktop in the Intel AMT tab.

Using a version before 15.0.42 works perfectly, so something definitely broke in 15.0.42.

To Reproduce Steps to reproduce the behavior:

1. Set up CIRA with ACM on a device running AMT 15.0.42
2. Go to the Desktop tab
3. Click HW Connect
4. Screen says "Setup" for a few seconds and then disconnects and displays the HW Connect button again.

Expected behavior User gets the PIN prompt, user enters PIN, AMT remote desktop works properly

Screenshots

https://user-images.githubusercontent.com/3211393/200551755-1efc07c8-806b-43b5-a589-9d89c065154c.mp4

Server Software (please complete the following information):

• OS: Ubuntu 20.04.4
• Virtualization: LXC
• Network: LAN/WAN Hybrid, Apache2 Reverse Proxy
• Version: 1.0.85
• Node: v18.11.0

Client Device (please complete the following information):

• Device: Desktop
• OS: Windows 10
• Network: Remote over WAN
• Browser: Chrome

Remote Device (please complete the following information):

• Device: Laptop
• OS: Windows 10 21H2
• Network: Remote CIRA
• AMT Version: 15.0.42
• Current Core Version (if known): N/A

Your config.json file

{
  "__comment__" : "This is a sample configuration file, edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "settings": {
    "Cert": "meshcentral.mydomain.com",
    "MongoDb": "mongodb://127.0.0.1:27017",
    "_MongoDbName": "meshcentral",
    "_MongoDbChangeStream": true,
    "_WANonly": true,
    "_LANonly": true,
    "_Minify": 1,
    "_SessionTime": 30,
    "SessionKey": "SUPERSECRETPASSWORD",
    "_SessionSameSite": "strict",
    "_DbEncryptKey": "SUPERSECRETPASSWORD",
    "DbRecordsEncryptKey": "SUPERSECRETPASSWORD",
    "_DbRecordsDecryptKey": "SUPERSECRETPASSWORD",
    "_DbExpire": {
      "events": 1728000,
      "powerevents": 864000
    },
    "Port": 8081,
    "RedirPort": 8082,
    "AliasPort": 443,
    "_AllowLoginToken": true,
    "_AllowFraming": true,
    "_WebRTC": false,
    "_Nice404": false,
    "_ClickOnce": false,
    "_SelfUpdate": true,
    "_AgentPing": 60,
    "AgentPong": 300,
    "_AgentIdleTimeout": 150,
    "_MeshErrorLogPath": "c:\\tmp",
    "_NpmPath": "c:\\npm.exe",
    "_NpmProxy": "http://1.2.3.4:80",
    "_AllowHighQualityDesktop": true,
    "_UserAllowedIP": "127.0.0.1,192.168.1.0/24",
    "_UserBlockedIP": "127.0.0.1,::1,192.168.0.100",
    "_AgentAllowedIP": "192.168.0.100/24",
    "_AgentBlockedIP": "127.0.0.1,::1",
    "_LocalDiscovery": {
      "name": "Local server name",
      "info": "Information about this server"
    },
    "TlsOffload": "10.22.254.1",
    "_MpsPort": 44330,
    "_MpsAliasPort": 4433,
    "_MpsAliasHost": "mps.mydomain.com",
    "MpsTlsOffload": false,
    "_No2FactorAuth": true,
    "_Log": "main,web,webrequest,cert",
    "_WebRtConfig": {
      "iceServers": [
        { "urls": "stun:stun.services.mozilla.com" },
        { "urls": "stun:stun.l.google.com:19302" }
      ]
    },
    "_AutoBackup": {
      "backupIntervalHours": 24,
      "keepLastDaysBackup": 10,
      "zipPassword": "MyReallySecretPassword3",
      "_backupPath": "C:\\backups"
    },
    "_Redirects": {
      "meshcommander": "https://www.meshcommander.com/"
    },
    "__MaxInvalidLogin": "Time in minutes, max amount of bad logins from a source IP in the time before logins are rejected.",
    "MaxInvalidLogin": { "time": 10, "count": 10, "coolofftime": 10 },
    "_Plugins": {
        "enabled": true
    }
  },
  "domains": {
    "": {
      "Title": "MyCompany MeshCentral",
      "Title2": "Device Management",
      "_TitlePicture": "title-sample.png",
      "_UserQuota": 1048576,
      "_MeshQuota": 248576,
      "_NewAccounts": true,
      "_UserNameIsEmail": true,
      "_NewAccountEmailDomains": [ "sample.com" ],
      "_NewAccountsRights": [ "nonewgroups", "notools" ],
      "NewAccounts": false,
      "Footer": "<a href='https://helpdesk.mydomain.com'>MyCompany HelpDesk</a>",
      "CertUrl": "https://meshcentral.mydomain.com/",
      "_PasswordRequirements": { "min": 8, "max": 128, "upper": 1, "lower": 1, "numeric": 1, "nonalpha": 1, "reset": 90, "force2factor": true, "skip2factor": "127.0.0.1,192.168.2.0/24" },
      "_AgentNoProxy": true,
      "_GeoLocation": true,
      "_UserAllowedIP": "127.0.0.1,192.168.1.0/24",
      "_UserBlockedIP": "127.0.0.1,::1,192.168.0.100",
      "_AgentAllowedIP": "192.168.0.100/24",
      "_AgentBlockedIP": "127.0.0.1,::1",
      "___UserSessionIdleTimeout__" : "Number of user idle minutes before auto-disconnect",
      "_UserSessionIdleTimeout" : 30,
      "__UserConsentFlags__" : "Set to: 1 for desktop, 2 for terminal, 3 for files, 7 for all",
      "_UserConsentFlags" : 7,
      "_Limits": {
        "_MaxDevices": 100,
        "_MaxUserAccounts": 100,
        "_MaxUserSessions": 100,
        "_MaxAgentSessions": 100,
        "MaxSingleUserSessions": 10
      },
      "deviceMeshRouterLinks": {
        "rdp": true,
        "ssh": true,
        "scp": true,
        "extralinks": [
          {
            "name": "HTTP",
            "protocol": "http",
            "port": 80
          },
          {
            "name": "HTTPS",
            "protocol": "https",
            "port": 443
      },
      {
        "name": "Fox TLS",
        "protocol": "custom",
        "port": 4911,
        "filter": [ "mesh//pYiAqw1xHRVzuEc4MtCkXFqX$iUD8aEYU9uFyBscV8C8$wBu2mskNK3fqdNiRLhO" ]
      },
      {
        "name": "Platform TLS",
        "protocol": "custom",
        "port": 5011,
        "filter": [ "mesh//pYiAqw1xHRVzuEc4MtCkXFqX$iUD8aEYU9uFyBscV8C8$wBu2mskNK3fqdNiRLhO" ]
      }
        ]
      },
      "auth": "ldap",
      "authStrategies": {
    "azure": {
          "callbackurl": "https://meshcentral.mydomain.com/auth-azure-callback",
          "newAccounts": true,
      "clientid": "SUPERSECRETPASSWORD",
      "clientsecret": "SUPERSECRETPASSWORD",
      "tenantid": "SUPERSECRETPASSWORD"
    }
      },
      "ldapOptions": {
        "url": "ldaps://corp.mydomain.com:636",
        "bindDN": "CN=MeshCentral Auth User,OU=Internal Service Users,DC=corp,DC=MyCompany,DC=com",
        "bindCredentials": "SUPERSECRETPASSWORD",
        "searchBase": "CN=Users,DC=corp,DC=MyCompany,DC=com",
        "searchFilter": "(sAMAccountName={{username}})",
        "tlsOptions": {
          "ca": "SUPERSECRETPASSWORD"
        }
      },
      "_AmtAcmActivation": {
        "log": "amtactivation.log",
        "certs": {
          "mycertname": {
            "certfiles": [ "amtacm-leafcert.crt", "amtacm-intermediate1.crt", "amtacm-intermediate2.crt", "amtacm-rootcert.crt" ],
            "keyfile": "amtacm-leafcert.key"
          }
        }
      },
      "_Redirects": {
        "meshcommander": "https://www.meshcommander.com/"
      },
      "_yubikey": { "id": "0000", "secret": "xxxxxxxxxxxxxxxxxxxxx", "_proxy": "http://myproxy.domain.com:80" },
      "_httpheaders": {
        "Strict-Transport-Security": "max-age=360000",
        "x-frame-options": "SAMEORIGIN",
        "Content-Security-Policy": "default-src 'none'; script-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; frame-src 'self'; media-src 'self'"
      },
      "_agentConfig": [ "webSocketMaskOverride=1" ],
      "_SessionRecording": {
        "_filepath": "C:\\temp",
        "_index": true,
        "__protocols__": "Is an array: 1 = Terminal, 2 = Desktop, 5 = Files, 100 = Intel AMT WSMAN, 101 = Intel AMT Redirection",
        "protocols": [ 1, 2, 101 ]
      }
    },
    "_customer1": {
      "_DNS": "customer1.myserver.com",
      "_Title": "Customer1",
      "_Title2": "TestServer",
      "_NewAccounts": 1,
      "_Auth": "sspi",
      "_Footer": "Test",
      "_CertUrl": "https://192.168.2.106:443/"
    },
    "_info": {
      "_share": "C:\\ExtraWebSite"
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 10.12 or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "myemail@myserver.com",
    "names": "myserver.com,customer1.myserver.com",
    "rsaKeySize": 3072,
    "production": false
  },
  "_peers": {
    "serverId": "server1",
    "servers": {
      "server1": { "url": "wss://192.168.2.133:443/" },
      "server2": { "url": "wss://192.168.1.106:443/" }
    }
  },
  "smtp": {
    "host": "365relay.mydomain.com",
    "port": 25,
    "from": "meshcentral@mydomain.com",
    "__tls__": "When 'tls' is set to true, TLS is used immidiatly when connecting. For SMTP servers that use TLSSTART, set this to 'false' and TLS will still be used.",
    "tls": false,
    "___tlscertcheck__": "When set to false, the TLS certificate of the SMTP server is not checked.",
    "_tlscertcheck": true,
    "__tlsstrict__": "When set to true, TLS cypher setup is more limited, SSLv2 and SSLv3 are not allowed.",
    "_tlsstrict": true
  }
}

[justinswall]

Can confirm I see the same results.

I first noticed this issue on Tuesday, October 26, 2022, after the update to v1.0.91. I'm not sure if that is a coincidence or not, however.

I was doing a presentation on MeshCentral for the Oregon Computer Consultants Association that evening, and the machine I was using to test HW Remote on I could connect to in the first half of the meeting, and then I no longer could in the second half of the meeting.

The problem is, two things occurred at roughly the same time:

1. The server was restarted and upgraded from v1.0.90 to v1.0.91
2. The workstation we were testing against was rebooted as well.

As such, the workstation could have installed the update to v15.0.42 during that reboot (I didn't note its AMT version before or after the reboot); the server update from v1.0.90 to v1.0.91 may be a red herring. I'm not sure. There is a Vimeo recording of the presentation; if needed, I could go back and double-check the exact order of events.

[justinswall]

This issue was first discussed on Reddit at https://www.reddit.com/r/MeshCentral/comments/ypb6vk/intel_amt_v15042_fails_to_connect_to_hardware_kvm/

[justinswall]

I looked back at the Vimeo recording and found that the system in question was running Intel AMT v15.0.41 at the beginning of the recording. So this looks more like an Intel AMT issue than a MeshCentral issue. :/

[4pack]

What's really strange is if you have KVM User Consent set and try to initiate HW Connect through MeshCommander, it properly displays the 6 digit pin in the remote computer, but upon entering said pin, it drops the connection like in my original post.

[justinswall]

I can report that behavior as well. Regardless of ACM or CCM, the connection cannot establish when using Intel AMT v15.0.42

[unguzov]

Same results for me. After BIOS/firmware update on a new Dell PCs when clicking "HW Connect", the screen goes to "setup", and then just disconnects immediately. AMT Flash: "15.0.42" Intel®AMT tab works just fine, except "Remote Desktop" functionality.

[4pack]

That's good to know that it's not down to a manufacturer problem. I'm experiencing this on a Lenovo Thinkpad P15 Gen 2.

[unguzov]

EDIT: Network driver crash was caused by bug in the backup software, not AMT related.

I'm not sure if there is a connection between the new AMT 15.0.42 and losing network connectivity on wired interface on the pc (connected to MikroTik switch). This starts to happen after the AMT update. Ethernet interface in OS just freeze all traffic, but physical link is ON. Only AMT connectivity works at this moment and sometimes OS gets BSOD pointing to the ethernet adapter. Mainboard is changed and this still happens from time to time.

[4pack]

That could be a funky driver issue for sure. I haven't noticed that on any of my Lenovo laptops running the same version.

[ib-mlatin]

Just confirmed that 15.0.41 is the last version that works. I guarantee that it will not work once the pending firmware update gets applied.

[mesmariusz]

What is the status here ?

Seems that AMT technology is not useful anymore. Not possible to connect graphical remote desktop anymore, so currently the AMT functionality is similar to a smart socket (you can connect to AMT web panel when the PC is off and turn it on). You can read the hardware description, but nothing more.

[si458]

@mesmariusz please read #4795, all development has stopped for the moment

[SurrounDTech]

I know MeshCentral is not actively being worked on but did anyone find a fix for this at all? We are starting to get machines with AMT version 16.1.25 and the issue of not being out able to connect through AMT to get to the BIOS is becoming problematic.

[justinswall]

That's interesting. I don't have any problems connecting to v16 AMT machines, it was just v15 as was mentioned in the original post. All of our AMT v16 machines have been Dell or Microsoft at this point though... so maybe your problem is OEM-specific?

As for the hopes of there being a fix, Ylian mentioned in another thread that without access to the equipment he had access to at Intel he doesn't have any way of troubleshooting problems with AMT. So, at least from Ylian, I don't see any AMT fixes coming in the future, unfortunately.

[SurrounDTech]

Thanks for the reply, it could very well be OEM specific as we are running HP devices but similar to the original post anything above v15.0.42 and remote desktop just flashes up "setup" and that's it. I might check to see if there are anymore updates on the HP site to what we have just received on the new devices (currently v16.1.25).

Yeah I have read about Ylian not being able to maintain the AMT side anymore which is a real shame as Mesh is such a great product especially for the AMT management side.

[thermionic]

@4pack & @SurrounDTech
If you run in the console eval SMBiosTables.amtInfo does it show "kvm":false or "kvm":true I'm looking at a HP 400 Mini G9 which shows false )-:

edit: because it only has vPro Essentials...