Feature Request: Enabling KVM redirection and configuring AMT consent in AMT Fully Automatic mode

[veitw]

Hi!

I started moving some PCs from my old AMT/CIRA config with my manually created setup.bin to a group in AMT Fully Automatic mode and its setup.bin for ACM activation.

AMT Fully Automatic mode worked fine for me with two exceptions:

1. I discovered that the "HW connect" button pops up in the Desktop tab but did not work. After checking the AMT tab I found that KVM redirection was disabled and needed manual activation. Disabled KVM redirection seems to be the default for unprovisioned HP computers.
2. For ACM activated devices, I expected AMT console consent to get synchronised with my device group's consent policy, but there does not seem to be any AMT consent configuration at all at the moment.

It would be great if the AMT Fully Automatic mode would enable KVM redirection automatically. Synchronising the KVM console consent policy to the device group's consent policy (or a new AMT/ACM consent policy flag per device group as I think MC user/usergroup consent setting can not be replicated to AMT) would be great, too. Getting these features, I could stop running a meshcommander script against every freshly provisioned computer to configure this, rendering it actually fully automatic.

Thank you very much and best regards, // Veit

[Ylianst]

This is interesting, MeshCentral should automatically enable the redirection port and KVM by default now. This is part of the check that is done each time MeshCentral takes a look at Intel AMT. Both of these are disabled by default, so it's odd it's not enabled automatically. I will try to replicate this.

For user consent, right now MeshCentral will disable it by default. Your right that I should sync it with the device group policy... that would be a smart thing to do! I will do that.

[Ylianst]

When running 'meshcmd amtconfig' (agentless) or typing the 'amtconfig' and 'amtevents' commands in the agent console tab (agent). I see the following:

13:27:29, User LMS tunnel closed.
13:27:54, User LMS tunnel start.
13:27:55, Checking Intel AMT state...
13:27:56, Intel AMT connected with TLS.
13:28:00, Enabled redirection features.                <-----------
13:28:00, Enabled KVM.                                 <-----------
13:28:00, Done.
13:28:00, User LMS tunnel closed.

I first disabled Intel AMT KVM and port redirection and let MeshCentral configure AMT which happens at regular intervals... and it did re-enable these features automatically. So, not sure what is going on.

[Ylianst]

Ok, KVM consent policy will now follow device group desktop prompt for user consent policy. This will be in MeshCentral v0.8.89.

[veitw]

Hi @Ylianst,

thank you very much for implementing this!

Regarding auto-enable KVM redirection: I updated MeshCentral from 0.8.87 to 0.8.89, unprovisioned the device's AMT completely through UEFI, provisioned ACM using the setup.bin and booted the device. KVM redir was automatically enabled after the provisioning had finished. User consent has also automatically been disabled as the device group has user consent for desktop disabled. Works as expected. 👍

Regarding user consent sync: I enabled user consent prompt in the device group my test device is in. The device is powered on and agent is connected.

• While using the agent remote desktop, user consent prompt works as expected, connecting using AMT still does not require user consent. ❌
• Switching the device group's AMT policy to "No policy" and then back to "Fully Automatic" to force AMT reconfiguration (I do not know a better way to trigger this -- this is how I force reconfiguration so far...), I still can connect without consent. ❌
• Waiting for 20 minutes, AMT consent is still not configured. ❌
• Moving the device to another device group (without AMT policy) and back to the device group with AMT Fully Automatic mode, AMT consent stays disabled. ❌
• Issuing "amtconfig" and "amtevents" to the agent console, configuration does not really seem to start (first two commands in screenshot below). ❌
• Unprovisioning ME and reactivating to ACM via stick and issuing the commands, does not reconfigure AMT either (commands 3 to 5 below), device now stays at status "Not Activated (Pre)". Connectivity shows "Intel AMT", but neither "HW connect" nor the AMT tab show up. ❌
• Deleting the device from MC, rebooting the device, reinstalling the agent do bring AMT functionality back. ❌

Before I unprovisioned ME, I also noticed that the device is no longer set up for CIRA but was configured for regular AMT; I was able connect to the web interface via both HTTP on port 16992 and HTTPS on port 16993, but I was unable to log in using admin and the MEBx password I configured using the setup.bin, so the user:pass seems something that MeshCentral configured.

MC is version 0.8.89, AMT is version 11.8.86, device is a HP EliteDesk 800 G2 SFF.

[veitw]

Update:

• Unprovisioning ME via UEFI and switching AMT policy to "No Policy" and back to "Fully Automatic" brought back AMT in CCM mode, but now again AMT exposed to LAN instead of CIRA.
• Unprovisioning ME again and applying setup.bin, then switching AMT policy to "No Policy" and back to "Fully Automatic"; AMT stayed in "pre", AMT neither accessible through MC nor via LAN.
• Rebooting the device and once agent was disconnected deleting the device, it automatically re-added itself and switching group's AMT policy to "No Policy" and back to "Fully Automatic" brought back AMT in ACM mode, but again AMT exposed to LAN instead of CIRA.

[veitw]

Update2:

• After enabling the device group's desktop user consent requirement, neither switching AMT policy to "No Policy" and back to "Fully Automatic" nor issuing "amtconfig" (at 00:50:11, which worked this time) activated user consent requirement for the device.

[veitw]

Update 3: Once the system is shut down, I lose access to AMT in both MC and also via webbrowser to port 16992/16993 (which sadly is now configured by MC instead of CIRA, no idea why). Thus MeshCentral seems not to configure ME to be available in soft-off and/or hibernation (Windows fast start-up) mode.

[veitw]

Update 4: After booting the device again, AMT connectivity does not come back, neither CIRA nor via local AMT nor can I access AMT directly via port 16992/16993. :( MC still shows status "Activated ACM, v11.8.86, TLS" but has no AMT or CIRA connection. Completely unpowering the systems does not help. Looking at the agent, AMT looks quite dead again, just like with the problems before:

[veitw]

Okay, I just fetched this from the system log (while MC's server log was empty):

Jul 30 13:15:51 mcp kernel: IN=en_dmz2 OUT= MAC=xxx SRC=xxx DST=xxx LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=45089 DF PROTO=TCP SPT=623 DPT=4433 WINDOW=8688 RES=0x00 SYN URGP=0 
Jul 30 13:15:52 mcp node: TypeError: Assignment to constant variable.
Jul 30 13:15:52 mcp node: at ProcessCommand (/home/mesh/meshcentral/node_modules/meshcentral/mpsserver.js:642:38)
Jul 30 13:15:52 mcp node: at TLSSocket.<anonymous> (/home/mesh/meshcentral/node_modules/meshcentral/mpsserver.js:538:26)
Jul 30 13:15:52 mcp node: at TLSSocket.emit (events.js:314:20)
Jul 30 13:15:52 mcp node: at addChunk (_stream_readable.js:297:12)
Jul 30 13:15:52 mcp node: at readableAddChunk (_stream_readable.js:268:11)
Jul 30 13:15:52 mcp node: at TLSSocket.Readable.push (_stream_readable.js:213:10)
Jul 30 13:15:52 mcp node: at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)

So happened everytime a clients tries to connect to the MPS service, this seems the reason why CIRA is not established. I just upgraded MC 0.8.89 to 0.8.91 and CIRA is back.

Thank you!

[veitw]

Hi @Ylianst,

• Synchronising AMT remote desktop consent now works when I trigger AMT reconfiguration by switching the device group AMT policy to "No Policy" and back to "Fully Automatic". :+1:
• Reconfiguring remote desktop consent by issuing "amtconfig" to the agent worked now. :+1:
• Reconfiguring remote desktop consent by rebooting the client (thus restarting the agent) did not work. :x: I hoped restarting the client would trigger a reconfiguration of AMT policy just as it triggers configuration of an unprovisioned device.
• Moving a device from one AMT Policy: Fully Automatic group to another Fully Automatic group with different user consent policy now reconfigures user consent correctly. :+1:

Three questions remain:

• Is automatic reconfiguration/synchronisation of AMT configuration performed on a regular schedule in Fully Automatic mode and is this configurable?
• Is there a simpler/better way to trigger reconfiguration/synchronisation of all devices' AMT in a device group than switching AMT policy away and back to Fully Automatic?
• I noticed that I have to wait some time between switching to No Policy and back to Fully Automatic for the reconfiguration to actually be performed -- how long do I actually need to wait to reliably trigger reconfiguration?

[Ylianst]

Apologies for the delay, just catching up on this thread. Lots of good data, I have been doing Intel AMT bug fixes in the last few days. For your 3 questions:

• Is automatic reconfiguration/synchronisation of AMT configuration performed on a regular schedule in Fully Automatic mode and is this configurable?

Yes, there is a component that I call the Intel AMT manager that should take a look at your Intel AMT devices periodically. It will happen each time the MeshAgent or Intel AMT connects, or when "meshcmd amtconfig" is run for agent-less device groups. The manager will look to see if everything is setup right, update hardware inventory information and sync the AMT clock if needed. If you run in agent-less mode, it's ok to run "meshcmd amtconfig" periodically to make sure everything is in sync.

• Is there a simpler/better way to trigger reconfiguration/synchronisation of all devices' AMT in a device group than switching AMT policy away and back to Fully Automatic?

Not right now, but I could look into putting that in place. In general , I was hoping the server could update devices automatically when needed. If you want to sync a single device, click on the device, go to the "console" tab and type "amtconfig". That will force the AMT manager to take a look at the device right away. You can also type "amtevents" to see previous events from the AMT manager.

• I noticed that I have to wait some time between switching to No Policy and back to Fully Automatic for the reconfiguration to actually be performed -- how long do I actually need to wait to reliably trigger reconfiguration?

Intel AMT is very chatty and I did not want the Intel AMT manager to look at 1000+ machines at once, so Intel AMT machines are put in a queue and a bunch are being worked on at any given time. Also, because there is a lot of round trips with AMT, the manager has to start by getting a lot of Intel AMT state which takes time. You can use the agent console "amtconfig" and "amtevents" to get a feel for the speed. Certainly switching AMT policy is not going to be instant.

Let me know if that helps.

[Ylianst]

FYI. Next version of MeshCentral will have an improved "amtconfig" command in the agent console that will show you what the server is doing in real time, just like "meshcmd amtconfig".

[veitw]

Hi @Ylianst,

many thanks for your answers!

So if I could make a wish to complete this feature, I'd like to see a button in the device's general tab like "reconfigure AMT" to put the device in the AMT manager queue for reconfiguration, and a button in the device group's general tab to queue all devices in this group for AMT reconfiguration.

I noticed one more bug that I am unsure whether it was introduced by changes for this feature, so I am unsure whether I should open a new issue for this or not:

We have "Maintenance" device groups with user consent disabled, that we move devices to if we need to do maintenance work on devices while they are not in use by a user, esp. for repairing or for setting up new devices. When these maintenance device groups have AMT policy "fully automatic", too, consent now gets correctly disabled when moving devices to that group. Unfortunately, if the device is shut down or rebooted or AMT/CIRA connection is lost, it cannot be re-established by the device as the device tries to register to the device group it was originally registered to but is no longer a member of. From the log:

Jul 30 16:32:19 mcp node: CIRA connection for unknown node. groupid: mesh//xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Moving the device back to its original device group, CIRA comes up again after a few seconds.

I think moving a CIRA device to a new device group also needs to update the groupid, at least when coming from and/or going to a device group with "fully automatic" policy.

This problem is similar to an older issue from like 2019 that I thought I opened, but I was unable to find it. Maybe it was reported by someone else and I commented on it when we experienced the same issue.

Thank you and best regards, // Veit