open-amt-cloud-toolkit / rpc-go

The Remote Provisioning Client (RPC) communicates with the Manageability Engine Interface (MEI) and Remote Provisioning Service (RPS) interfaces.
https://open-amt-cloud-toolkit.github.io/docs/
Apache License 2.0
13 stars 15 forks source link

PRC client prerequisites #49

Closed michaelmumenthaler closed 2 years ago

michaelmumenthaler commented 2 years ago

We are currently testing the OpenAMT stack for a bunch of HP Z Workstations but ran into some issues with the RPC-Go application.

As far as I understood the documentation I should be able to pull the client out of the box with factory settings, install an OS (Ubuntu server 22.04 in our case) and build / run the RPC application. But we always run into the same error when trying to add the device to OpenAMT.

We run the following command:

sudo ./rpc activate -profile testprof200 -u wss://192.168.200.200/activate -n -v

and then we get the following output:

TRAC[0000] Seeing if existing LMS is already running....
DEBU[0000] connecting to lms
TRAC[0000] nope!
DEBU[0000] closing connection to lms
INFO[0000] no connection to close
Unable to launch MicroLMS. Check that Intel ME is present, MEI driver installed and run this executable as administrator.

We verified that the clients are AMT compatible which they 100% are. I tried to find some information in the documentation if some pre-configuration in the BIOS/UEFI has to be done beforehand but could not find anything specific.

Output from sudo ./rpc amtinfo:

Version                 : 14.1.60
Build Number            : 1807
SKU                     : 8
UUID                    : [redacted]
Control Mode            : pre-provisioning state
DNS Suffix              :
DNS Suffix (OS)         :
Hostname (OS)           : ws001
RAS Network             : unknown
RAS Remote Status       : not connected
RAS Trigger             : user initiated
RAS MPS Hostname        :
---Wired Adapter---
DHCP Enabled            : true
DHCP Mode               : passive
Link Status             : up
IP Address              : 0.0.0.0
MAC Address             : [redacted]
---Wireless Adapter---
DHCP Enabled            : false
DHCP Mode               : passive
Link Status             : down
IP Address              : 0.0.0.0
MAC Address             : 00:00:00:00:00:00

The client and the docker host with the OpenAMT stack are in the same network without any firewall policies in between.

Any help would be greatly appreciated!

michaelmumenthaler commented 2 years ago

Alright, so in a stroke of genius I activated the Intel AMT option in the BIOS/UEFI. The client now connects to the OpenAMT WSS and is visible in the web GUI as "disconnected". When running the RPC activate command i receive a new error:

INFO[0000] no connection to close
Starting MicroLMS.
INFO[0005] connecting to wss://192.168.200.200/activate
INFO[0005] connected to wss://192.168.200.200/activate
ERRO[0026] {"Status":"Client control mode","Network":"Failed"}

sudo ./rpc amtinfooutput:

Version                 : 14.1.60
Build Number            : 1807
SKU                     : 16392
UUID                    : [redacted]
Control Mode            : activated in client control mode
DNS Suffix              :
DNS Suffix (OS)         :
Hostname (OS)           : ws002
RAS Network             : direct
RAS Remote Status       : not connected
RAS Trigger             : user initiated
RAS MPS Hostname        :
---Wired Adapter---
DHCP Enabled            : true
DHCP Mode               : passive
Link Status             : up
IP Address              : 192.168.200.4
MAC Address             : [redacted]
---Wireless Adapter---
DHCP Enabled            : false
DHCP Mode               : passive
Link Status             : down
IP Address              : 0.0.0.0
MAC Address             : 00:00:00:00:00:00

Is there another setting I'm missing?

Thanks again

rsdmike commented 2 years ago

Hi @michaelmumenthaler , Yes, sometimes AMT is shipped disabled in the BIOS as you discovered, and that varies for each manufacturer. Glad you got the first issue taken care of! Are you able to post your logs from the server side (RPS)? that should give some indication of what went wrong during network configuration. Also, are you using CIRA? the RAS Network mode should be "outside enterprise" instead of direct if you are and a RAS Trigger of periodic.

michaelmumenthaler commented 2 years ago

Hi @rsdmike, yes the following log entries were created when we tried to register the device:

2022-05-04T13:30:51.409Z error: NetworkConfig - 2d8656fe-cdf4-4b9e-98cc-e340d5047bf8 : Failed to configure network settings : TypeError: Cannot set properties of undefined (setting 'DHCPEnabled')
2022-05-04T13:33:07.113Z error: NetworkConfig - 3f9f9e1e-0df6-44e2-bf32-caa60382be9c : Failed to configure network settings : TypeError: Cannot set properties of undefined (setting 'DHCPEnabled')
2022-05-04T13:47:34.274Z error: NetworkConfig - 5dee0e39-21be-48e9-a392-f45bf6b17104 : Failed to configure network settings : TypeError: Cannot set properties of undefined (setting 'DHCPEnabled')
2022-05-04T14:20:48.786Z error: NetworkConfig - f4159895-1d97-4587-aa7e-746dbece160c : Failed to configure network settings : TypeError: Cannot set properties of undefined (setting 'DHCPEnabled')
2022-05-04T14:21:57.825Z error: NetworkConfig - c7893221-b854-4547-acbf-89b11204fa2e : Failed to configure network settings : TypeError: Cannot set properties of undefined (setting 'DHCPEnabled')

As for CIRA, yes I created a CIRA profile analog to the documentation, is there a setting I need to set on the client that I'm missing? Is CIRA even needed? Basically what we want to do is being able to control the client's power state via remote and having full control over the power states at all times. Then create a script which uses the microservice's APIs to control the powerstates automatically.

rsdmike commented 2 years ago

Hey @michaelmumenthaler, can you try with the latest release of the toolkit that we released last week and see if that fixes your issue?

michaelmumenthaler commented 2 years ago

Hey @rsdmike, thanks for the update. I tried again and it got a bit further but I'm now getting the following error:

[...]
DEBU[0031] sending message to RPS
DEBU[0031] received APF_CHANNEL_CLOSE
TRAC[0031] {MessageType:97 RecipientChannel:17}
DEBU[0031] sending APF_CHANNEL_CLOSE_MESSAGE
TRAC[0031] {[97 0 0 0 17] 0 0}
ERRO[0031] {"Status":"Client control mode","Network":"Failed"}
DEBU[0031] closing connection to lme

AMT Info:

Version                 : 14.1.60
Build Number            : 1807
SKU                     : 16392
UUID                    : [REDACTED UUID]
Control Mode            : activated in client control mode
DNS Suffix              :
DNS Suffix (OS)         :
Hostname (OS)           : gpu002
RAS Network             : direct
RAS Remote Status       : not connected
RAS Trigger             : user initiated
RAS MPS Hostname        :
---Wired Adapter---
DHCP Enabled            : true
DHCP Mode               : passive
Link Status             : up
IP Address              : 192.168.200.4
MAC Address             : [REDACTED MAC]
---Wireless Adapter---
DHCP Enabled            : false
DHCP Mode               : passive
Link Status             : down
IP Address              : 0.0.0.0
MAC Address             : 00:00:00:00:00:00

In the attached file is the full verbose output. rpclog.txt

Edit: Like before the client appears in the webgui with a disconnected state.

michaelmumenthaler commented 2 years ago

Ok, so I updated the whole docker stack as well and the client now registers properly and can also be controlled via the WebUI. The only problem I am facing now is that once the client is shut down i cannot start it in the WebUI. I tried with a few different BIOS power settings but to no avail.

The status in the WebUI says: Connected (power: )

Do you perhaps have an idea why this could be or what else i might be missing?

Thanks again!

michaelmumenthaler commented 2 years ago

Sorry for the spam, I figured out that when I use Windows to register the device etc. I can start the device from the WebUI. So I guess it's an issue with the linux driver (or the lack of) to set the ethernet powerstate when the machine is off.

I guess I'll have to switch to Windows to provision the clients but this is not an issue of OpenAMT. So this issue is closed.

Thanks again for the support!