search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
3.99k stars 536 forks source link

Bug? AMT-enabled device does not show AMT connectivity in Agent mesh but does in AMT mesh #643

Closed veitw closed 4 years ago

veitw commented 4 years ago

Hi,

we just bought a bunch of new devices, all of them are the same model (HP EliteDesk 800 G5 DM 7PF52EA), and after deploying the first 59 of them, while

MC shows the following odd behaviour on just 2 of those devices:

The machines show up with the agent connection in the associated agent mesh in MC, but are not recognised as AMT-capable. But I can connect to AMT via browser to http://:16992/ using the IP address from the agent connection as shown in MC. Also when I create an AMT only mesh in MC and add the device with this IP address, MC recognises the device and shows it online, but only in the AMT mesh, while the device stays offline in the Agent mesh or shows agent only connection when powered on.

The network switches are not the culprit as I already switched port of one of the odd devices with another device that correctly shows up on both ports while the odd one keeps staying AMT offline in the agent mesh.

There is also no IP address collision or firewall issue as I switched IPs with a working device and the odd behaviour moved along to the new IP.

Any hints on how to debug this or anything else to try?

Could this be some kind of e.g. an AMT identifier collision or bug in MC?

Best regards, // Veit

jsastriawan commented 4 years ago

What is the system UUID shown in meshconmander panel? In some cases AMT may have a non unique UUID.

You can also quickly query AMT UUID using MeshCmd AMTUUID command.

Not sure how you define AMT connection here but AMT CIRA connection uses this UUID as device identifier. So if multiple devices connected to MC2 MPS Server using the same UUID, definitely there will be collusion.

veitw commented 4 years ago

Thanks for your hints, @jsastriawan!

What is the system UUID shown in meshconmander panel? In some cases AMT may have a non unique UUID.

This is what I meant by "AMT identifier collision". :)

You can also quickly query AMT UUID using MeshCmd AMTUUID command.

Unfortunately (or furtunately? ;)), all of them are unique:

$ for i in {1..59}; do ./meshcmd amtuuid --pass 'xxx' --host x.x.x.$i; done >amt_uuids.txt
$ wc -l <amt_uuids.txt
59
$ sort amt_uuids.txt | uniq | wc -l
59

Not sure how you define AMT connection here but AMT CIRA connection uses this UUID as device identifier. So if multiple devices connected to MC2 MPS Server using the same UUID, definitely there will be collusion.

We use direct AMT connections for this local network as we experienced CIRA/CILA to be much slower in the local network.

jsastriawan commented 4 years ago

Ok,

Here is another situation which may caused this. How are your device is connected to network?

  1. MC2 detects AMT via RMCP ping. Unfortunately AMT will only reply that via LAN interface. If your system is connected via Wifi, although AMT can support that WiFi, I don't think MC2 could detect the presence of AMT via that WiFi network interface.

  2. If my first assumption was not completely correct, you may need to add WiFi profile to AMT manually using MeshCommander locally.

I hope my answer could give some way to figure out the situation.

Regards,

Joko Sastriawan

veitw commented 4 years ago

Is there some documentation on how MC actually recognises AMT devices to be online?

Because at another site we have another odd behaviour with some older AMT 11.0 devices in AMT only meshes (no agent installed there) that only show up when powered on, but go offline a short while after powered off, while all devices in that network that are either newer or older stay online. Here we also can connect via browser to AMT and the devices work well in other remote management solution employing AMT. AMT is configured to be online in all power states and both ICMP and RMCP pings are enabled, so I suspect that the ME answers slower when the system is powered off and this is caused by some kind of timeout that might be chosen too aggressive in MC.

veitw commented 4 years ago

Here is another situation which may caused this. How are your device is connected to network?

1. MC2 detects AMT via RMCP ping. Unfortunately AMT will only reply that via LAN interface. If your system is connected via Wifi, although AMT can support that WiFi, I don't think MC2 could detect the presence of AMT via that WiFi network interface.

The devices are all wired. I even tried to swap switch ports with working devices to no avail.

veitw commented 4 years ago

I just tried to configure the device for CIRA on the Agent mesh; the script applied correctly, but the MC does not show a CIRA connection either.

jsastriawan commented 4 years ago

Here are what I know how MC detects the presence of AMT on a platform.

  1. MeshAgent MEI query and SMBIOS query.

MeshAgent will try to connect to Management Engine Interface to query AMT and also parse certain SMBIOS table to identify if the device supports AMT. You can check amt-mei.js in agent modules to see how it is being done. See also Ylian's MeshAgent repo where all the magic happens. @krayon007 could explain better. Once this detection completed, the data is sent to MC2 and added to node metadata. Detection is primarily based on MeshAgent and CIRA connection for WAN type connection, for LAN based setup, it will typically use MC2 periodic detection via RMCP ping. @Ylianst correct me if I am wrong here.

  1. CIRA connection If a device is connected via CIRA, it is automatically recognized as AMT machine. The state of connection is dependent on CIRA connection status.
veitw commented 4 years ago 
1. MeshAgent MEI query and SMBIOS query.

Ah, okay, so this might actually be caused by the Agent not being able to locally connect to AMT, causing MC not even trying to connect to AMT via network?

jsastriawan commented 4 years ago

You may try installing Intel Manageability Engine driver from the OEM who sells you the hardware. A simple test if MEI driver is installed (typically also accompanied with LMS), you should be able to access AMT webui via localhost:16992.

Regards,

Joko

Ylianst commented 4 years ago

If you don't see the Intel AMT line in the device tab with the Intel AMT version and state, the MEI driver is probably not installed on the remote machine.

veitw commented 4 years ago

Thanks to you both for this hint.

I forwarded this to a colleague for verification, as I will not hit the office the next two weeks.

veitw commented 4 years ago

Sorry for the late reply.

We did an automatic OS reinstallation again. Although it all looks the same as before, AMT is detected now. We have no real clue, but @jsastriawan 's hint for the LMS was right.

Thank you very much for your support!