AMT activated in ACM. But CIRA: Disconnected

[ZENAdmin-Ops]

Describe the bug Hello,

I'm raising this as a bug as I have this issue on 4 devices and have not been able to determine a resolution. And the discussion questions that I have seen have not been responded to.

This could be a bug, or maybe I just need some troubleshooting advice.

I have other AMT devices (and agents) that are Activated and are connecting fine to the MC server. So the MC server is functional - at least to some degree.

To Reproduce Steps to reproduce the behavior:

1. Have an Intel AMT only group
2. Intel AMT Policy
3. Simple ACM
4. Keep existing password
5. Deactivate CCM
6. CIRA - Connect to server
7. Deploy setup file using meshcmd
8. AMT device appears in group, but IntelAMT tab is diconnected with no option to connect
9. Run meshcmd amtinfo on device.
10. Intel AMT v12.0.93, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

Expected behavior Expect CIRA to be connected

Screenshots If applicable, add screenshots to help explain your problem.

Server Software (please complete the following information):

• OS: Windows 10 Pro
• Virtualization: Hyper-V VM
• Network: WANonly
• Version: 1.1.24
• Node: 20.15.0

Client Device (please complete the following information):

• Device: OptiPlex 7070
• OS: Windows 10 Pro
• Network: [Remote over WAN]
• Browser: [N/A]
• MeshCentralRouter Version: [if applicable]

Remote Device (please complete the following information):

• Device: [e.g. Laptop]
• OS: [e.g. Windows 10 21H2]
• Network: [e.g. Local to Meshcentral, Remote over WAN]
• Current Core Version (if known): [HINT: Go to a device then console Tab then type info]

Additional context Add any other context about the problem here.

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are ignored. Edit a section and remove the _ in front of the name. Refer to the user's guide for details.",
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "mc1.zen.net.au",
    "WANonly": true,
    "_LANonly": true,
    "sessionKey": "### have changed ###",
    "_port": 443,
    "aliasPort": 5443,
    "MpsAliasPort": 54433,
    "_redirPort": 80,
    "_redirAliasPort": 80,  
    "mongodb": "mongodb://127.0.0.1:27017/meshcentral",
    "mongodbcol": "meshcentral",
    "autoBackup": {
    "mongoDumpPath": "C:/Program Files/MongoDB/Server/7.0/bin/mongodump.exe",
    "backupIntervalHours": 24,
    "keepLastDaysBackup": 10,
    "zipPassword": "#### have changed ####",
    "backupPath": "C:/MeshCentral/meshcentral-backups",
    "maxFiles": 10
    }
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "info@zen.net.au",
    "names": "mc1.zen.net.au",
    "skipChallengeVerification": true,
    "production": false
  }
}

[si458]

you say Keep existing password, in the General tab of the device, is the an option saying invalid credentials next to the AMT bit at all? share screenshot of you can?

[ZENAdmin-Ops]

Some screenshots from MeshCentral

[si458]

people keep reporting this type of issue and sadly i havent been able to replicate the issue? BUT i only have a single AMT v7 device which works no matter what, so without hardware to test, i cannot fix the bug (if it is a bug).

a few things you can try:

1. go into the console tab under My Server then type amtpasswords then check the device thats listed in there AND the password for the AMT (mebx) is infact in that list.

2. check you can access the AMT web panel on the device itself https://localhost:16992 or https://localhost:16993

3. if you can access the web ui, check you can login with a password listed in amtpasswords for that device

4. run meshcentral with amt debug node node_modules/meshcentral --debug amt

[ZENAdmin-Ops]

Hello,

Have I done the amtpasswords step correctly?

[si458]

@ZENAdmin-Ops erm yes and im not sure why its blank?

[ZENAdmin-Ops]

Nothing else has appeared in the console

[si458]

@ZENAdmin-Ops can you just try all the other options for me tho please

[ZENAdmin-Ops]

I am able to logon to the device using https://localhost:16993

But there are no passwords listed in amtpasswords so I am using the credentials that I know are correct for the device

[ZENAdmin-Ops]

For step 4.

Where will the logs / diagnostic info be reported that I will need to submit?

[si458]

@ZENAdmin-Ops sorry first stop meshcentral, then rerun it with that command, the logs will show on the screen

you can also go into the web ui, then my server, then trace, click tracing, then enable ALL the intel amt sections, then it will show any debug stuff in the web ui (dont close the page!)

if you use the web ui, its best to shut down computer, unplug its power, wait say 2 mins, then plug its power back in, switch computer on, then watch the web ui output

if you get no output in either the console or the trace ui, then you might have the amt port 4433 blocked in your firewall

[ZENAdmin-Ops]

OK. These systems are going to be in use this evening.

So I will try the tracing tomorrow and report back then.

Ironically I had access to AMT via VNC Plus prior to trying to switch them to MeshCentral. And at the moment AMT access via MeshCentral for these devices is broken.

So, I can't do the remote restart and wait 2 mins.

I'll just try a remote restart tomorrow. Worst case I'll have to go to the office to perform a power-drain, that will be sometime in the next week.

I'd really like to get to the bottom of this. So I'll do whatever testing is needed.

Appreciate your prompt assistance.

[si458]

thats ok! in theory you can do the web ui trace now if you wanted while its running,

go into the web ui, then my server, then trace, click tracing, then enable ALL the intel amt sections, then it will show any debug stuff in the web ui (dont close the page!)

then watch the logs as you try different things and see what happens,

but what you need to do is really unplug/plug the remote device, that will restart AMT into connecting to your meshcentral, and if you NEVER see any connections from that device happen when you power cycle it, then something isnt right with the AMT

strange one (side note), can you use the meshcommander application and connect into the device? if you can then you can check the internet settings tab and see if your meshcentral server dns is listed there, same with the security settings, check its certificate is listed there

[ZENAdmin-Ops]

The server trace doesn't appear to identify the host.

And I have AMT devices that are working as well as 3 or 4 that are stuffed.

[ZENAdmin-Ops]

Internet Settings looks correct

I'm using a redirected port.

Can you tell me where I check the Trusted Root certificate?

[si458]

plz can you share a few more screenshots? just trying to compare with mine System Status, Network Settings, Internet Settings, Security Settings ?

[si458]

ive just spotted, it says talk to port 54433, which you have set as your alias port for amt do you use a reverse proxy server at all? as the default port should be 4433, and the alias port only tells remote device which port to connect to, but you havent specified an amt port so its going to use the default 4433 port so 54433 != 4433

[ZENAdmin-Ops]

plz can you share a few more screenshots? just trying to compare with mine System Status, Network Settings, Internet Settings, Security Settings ?

[si458]

the dns suffix for the computer in incorrect! from my understanding and knowledge, the dns suffix MUST match your meshcentral dns name!? so you should be able to click the name & domain in network settings, and change the dns suffix

[ZENAdmin-Ops]

ive just spotted, it says talk to port 54433, which you have set as your alias port for amt do you use a reverse proxy server at all?

No reverse proxy server.

as the default port should be 4433, and the alias port only tells remote device which port to connect to, but you havent specified an amt port so its going to use the default 4433 port so 54433 != 4433

I don't understand the comment above.

The public AMT port is 54433 which forwards to 4433 on the MeshCentral server.

[si458]

The public AMT port is 54433 which forwards to 4433 on the MeshCentral server.

ah right ok thats fine! so port forwarding 54433 to 4433, thats ok makes sense! my mistake!

[ZENAdmin-Ops]

the dns suffix for the computer in incorrect! from my understanding and knowledge, the dns suffix MUST match your meshcentral dns name!? so you should be able to click the name & domain in network settings, and change the dns suffix

Well I have devices at other client sites that working in MeshCentral and their domain suffix does not match our dns suffix

If all the devices have to match MeshCentral dns suffix that will be a showstopper for us. We will need a work-around

Does this mean that I would need to set up multiple domains within MeshCentral. There is a video on multiple domains, but I haven't watched it yet.

[si458]

hmm ok that is weird? and forgive me but i could be mistaken!

in theory you can click the name & domain, then change name sharing to dedicated, different from os, then set a computer name and your meshcentral dns name in there (computer1.meshcentral.myserver.com) and then click OK/Save, it then WONT effect your windows OS domain as its seperate :)

[ZENAdmin-Ops]

hmm ok that is weird? and forgive me but i could be mistaken!

in theory you can click the name & domain, then change name sharing to dedicated, different from os, then set a computer name and your meshcentral dns name in there (computer1.meshcentral.myserver.com) and then click OK/Save, it then WONT effect your windows OS domain as its seperate :)

So you would suggest that the name should be: bbox4.zen.net.au ?

[si458]

So you would suggest that the name should be: bbox4.zen.net.au ?

no it would need to match the DNS name you use to access meshcentral, so from the screenshots/config.json something like comp1.mc1.zen.net.au

[ZENAdmin-Ops]

I've just changed the name to:

bbox4.mc1.zen.net.au

No change in amtinfo CIRA: Disconnected

Do I need to restart?

[si458]

@ZENAdmin-Ops possibly the remote machine yes, by unplugging/plugging back in, but maybe just give it chance to connect, as i do know/have seen my AMT can take 5-10mins before it connects back

[si458]

@ZENAdmin-Ops what is the full output now if you run the amtinfo with meshcmd?

[ZENAdmin-Ops]

@ZENAdmin-Ops what is the full output now if you run the amtinfo with meshcmd?

I have restarted

C:\ZEN\MeshCentral\ZEN>meshcmd amtinfo Intel AMT v12.0.93, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

[si458]

ok i am confused because its missing a section which should show the dns suffix? but at the same time the amtinfo command has NEVER worked on my V7 due to a bug i will try fix my bug here next week, and then have a look at whats going on if i can im sorry i cant be of much more help, i will ask @Ylianst when i get chance, see if he has any suggestions as we must be missing something!

[ZENAdmin-Ops]

This is from a system that is working.

Later version of AMT.

C:\ZEN\MeshCentral>meshcmd amtinfo Intel AMT v16.1.30, activated in Admin Control Mode (ACM). Wired Enabled, DHCP, CC:96:E5:37:4A:7F DNS suffix: alleanza.local Connection Status: Outside, CIRA: Connected to mc1.zen.net.au, Periodic.

What determines Connection Status: Outside versus Direct?

[ZENAdmin-Ops]

Actually that demonstrates (at least in that instance) that the DNS suffix doesn't need to match the MeshCentral server DNS suffix.

[si458]

ok so the dns suffix isnt the problem! thats ok! im still learning all the AMT bits n bob so my mistake! AMT seems to be registering as direct and thinks it has direct access to your meshcentral server rather than registering as OUTSIDE the network and needs to use CIRA. so are both of the AMT machines (v12 and v16) on the same network OR different networks with like different DNS servers? one other thing to check is use meshcommander again, go into Internet Settings and click Environment detection and see what DNS names are listed in the pop up, it should list a RANDOM name

[ZENAdmin-Ops]

These systems are on completely different networks.

But the AMT systems that aren't working are all registered as Direct.

So that could be the issue here.

[si458]

one other thing to check is use meshcommander again, go into Internet Settings and click Environment detection and see what DNS names are listed in the pop up, it should ONLY list a RANDOM name, is this the case? (screenshot hehe)

[ZENAdmin-Ops]

one other thing to check is use meshcommander again, go into Internet Settings and click Environment detection and see what DNS names are listed in the pop up, it should ONLY list a RANDOM name, is this the case? (screenshot hehe)

[si458]

so the is defo something wrong with the AMT then? are the IP addresses assigned to the DNS name different from the working ones and non-working ones? (like internal ip VS external ip?)

[ZENAdmin-Ops]

The systems that aren't working.

Are at the same physical site (not the same subnet) as the MeshCentral server.

But they're accessing the MeshCentral server via the same router (so the same network path) as the Outside host.

So, from my perspective they should consider the MeshCentral server to be outside. But perhaps the AMT / MeshCentral logic is "smart enough" to tell the difference.

If that is the case, I'd like to understand the logic, so that I can make it so that these AMT systems will see the MeshCentral server as 'Outside' and not 'Direct'.

[ZENAdmin-Ops]

so the is defo something wrong with the AMT then? are the IP addresses assigned to the DNS name different from the working ones and non-working ones? (like internal ip VS external ip?)

The internal IP's are completely different subnets.

The external IP could be the problem here.

With the Outside systems, they are at a different location so the Public IP address is different.

With the Direct systems, they are at the same site, and the Public IP address is the same.

But in both cases, the same firewall rules are used. So it is the same FQDN:port which then forwards to the same internal IP/ port.

[si458]

@ZENAdmin-Ops in theory the logic is already set, you shared the screenshot for Environment detection and if you read it, IF the local dns suffix is the same as those listed in the list then it conciders the AMT as LOCAL/DIRECT BUT if they arent the same, its concidered AWAY/OUTSIDE

but for some reason the AMT seems to think it has DIRECT access to your meshcentral server which doesnt make sense?

one thing you can try is REMOVE/UNCOMMENT WANonly from your config.json, restart meshcentral, let it enter into hybrid mode, see if the devices suddenly connect! if they do, the machines seem to think they can talk directly to meshcentral without CIRA support

[ZENAdmin-Ops]

one thing you can try is REMOVE/UNCOMMENT WANonly from your config.json, restart meshcentral, let it enter into hybrid mode, see if the devices suddenly connect! if they do, the machines seem to think they can talk directly to meshcentral without CIRA support

Genius.

Two of the systems are now connected!

[ZENAdmin-Ops]

But no change to amtinfo

This is from a system now online and working in MeshCentral

C:\ZEN\MeshCentral\ZEN>meshcmd amtinfo Intel AMT v12.0.93, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

That doesn't make sense to me.

[si458]

@ZENAdmin-Ops glad its almost sorted but wow that is weird !? it must be the machines are seeing the meshcentral server as LAN rather than WAN for some reason? AMT is very confusing to understand BUT an amazing technology/idea!

im assuming the amtinfo hasnt changed BECAUSE those machines that arent working can talk directly to meshcentral rather than using the CIRA, only thing i can think off!

i will dig into the meshcmd tho when i get chance as i said before i have a BUG which means it doesnt work on mine at all.

[ZENAdmin-Ops]

There are also still two systems whose status hasn't changed, and I can't think what to do to them to get them to connect.

Other than deleting them out of MeshCentral and re-running meshcmd amtconfig

I think there is a bug here.

[si458]

There are also still two systems whose status hasn't changed, and I can't think what to do to them to get them to connect.

Other than deleting them out of MeshCentral and re-running meshcmd amtconfig

I think there is a bug here.

i would say WAIT about 10-15mins see if they connect themselves, if they dont then UNPROVISION them using the meshcmd or the mebx, then reconfig them and see what happens

its all trial and error now!

[ZENAdmin-Ops]

When you say UNPROVISION using meshcmd.

Which command?

I used AmtAcmDeactivate on one system and completely lost control of AMT.

I had to go to site to access MEBX directly.

[si458]

yes that command! AmtAcmDeactivate its very dangerous as it unprovisions the AMT, it basically removed all certs, details, etc and resets to default the AMT (not the password tho), so yes you would need to be on site for that

[ZENAdmin-Ops]

Unprovision is a shitter.

That cannot be done remotely.

[ZENAdmin-Ops]

Hello,

As I expected the two unconnected hosts have not connected.

I'd like to try options to tweak the AMT settings to switch the connection from Direct to CIRA.

All the AMT systems that are connecting via CIRA are working smoothly.

When you next are able to speak with @Ylianst if you could ask what options I might be able to try here to get these AMT's to switch to CIRA.

That would be great.

[ZENAdmin-Ops]

Hello,

I'm returning to this issue.

A recap. I have computers at 3 sites connected to MeshCentral. Both meshagents and AMT.

MeshAgents work at all 3 sites.

AMT agents only connect from 2 sites.

The 3rd site, the "problem" site. Had the MeshCentral server.

So the troubleshooting above assumed that the issue with the AMT endpoints at the 3rd site was somehow related to the AMT endpoints being "local" to the MeshCentral server.

I have now setup a MeshCentral server at another location and performed a backup and restore of the MeshCentral server to that location.

The behaviour of the agents is as before.

All the MeshAgents connect.

The AMT agents at two sites connect, but not the 3rd site.

So I want to resume troubleshooting the AMT endpoints at the 3rd site.

I had deleted the AMT endpoints from the 3rd site in MeshCentral. The endpoints appear in MeshCentral, but they always show as offline / disconnected. As per screenshot at bottom of this post.

I also 'reset' AMT on these endpoints.

• meshcmd amtacmdeactivate
• Then reconfigured AMT via MEBX (i.e. entered IP configuration for LAN)

Here is output from attempt to connect to MeshCentral C:\ZEN\MeshCentral\ZEN>meshcmd amtconfig --url wss://mc1.zen.net.au:5443/apf.ashx --id --serverhttpshash (I have hidden some of the info in case it is sensitive) Setting up MEI... Starting Intel AMT configuration... Started APF tunnel... Checking Intel AMT state... Intel AMT connected. Performing Commit... Enabled TLS, holding 10 seconds... Intel AMT connected. Added server root certificate. Created new MPS server. Created new MPS policy. Environment detection set. Enabled redirection features. Enabled KVM. Fetching hardware inventory. Done.

C:\ZEN\MeshCentral\ZEN>meshcmd amtinfo Intel AMT v12.0.94, activated in Admin Control Mode (ACM). Wired Enabled, Static, E4:54:E8:92:A7:FB, 10.254.242.100 Connection Status: Direct, CIRA: Disconnected.

This meshcmd amtconfig results in the endpoint appearing in MeshCentral as per the screenshot. But the endpoints are offline and cannot be connected to.

The issue from my perspective here. Is that the "working" AMT endpoints report: Connection Status: Outside rather than Direct

See example below.

C:\ZEN\MeshCentral>meshcmd amtinfo Intel AMT v16.1.30, activated in Admin Control Mode (ACM). Wired Enabled, DHCP, CC:96:E5:37:4A:7F DNS suffix: alleanza.local Connection Status: Outside, CIRA: Connected to mc1.zen.net.au, Periodic.

The Connection Status of 'Outside' versus 'Direct' is what led to me thinking that the MeshCentral server being "local" was the issue. However the MeshCentral server is no longer "local" and the behaviour of the AMT agent is unchanged.

Diagnostics go into the console tab under My Server then type amtpasswords then check the device thats listed in there AND the password for the AMT (mebx) is infact in that list.

Nothing appears. Refer screenshot.

check you can access the AMT web panel on the device itself https://localhost:16992 or https://localhost:16993

if you can access the web ui, check you can login with a password listed in amtpasswords for that device

Refer screenshots

run meshcentral with amt debug node node_modules/meshcentral --debug amt

The AMT agents that are connecting appear in the console log. The AMT agents that are not connecting (e.g. BBOX4) do not appear in the console log.

[ZENAdmin-Ops]

Hello,

Can you advise with amtinfo the difference between Connection Status:

Direct Outside