Open abjoseph opened 1 week ago
Node 14 is EOL, the min requirement is node 16, and recommended node 18. So please update node and try again
Node 14 is EOL, the min requirement is node 16, and recommended node 18. So please update node and try again
@si458 When I updated MeshCentral to v1.1.24, node was also updated to v18 and that's the result that was screenshot above where the AMT agents aren't connecting.
Config all looks OK at first glance? What's your reverse proxy config? I'm guessing the AMT port is also proxied via ur reverseproxy?
Do u get any errors when u start meshcentral?
Config all looks OK at first glance? What's your reverse proxy config? I'm guessing the AMT port is also proxied via ur reverseproxy?
root@meshcentral:~# systemctl status meshcentral.service
* meshcentral.service - MeshCentral Service
Loaded: loaded (/etc/systemd/system/meshcentral.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2024-06-23 16:49:27 UTC; 11min ago
Main PID: 1473 (node)
Tasks: 22 (limit: 76917)
Memory: 169.3M
CPU: 5.969s
CGroup: /system.slice/meshcentral.service
|-1473 /usr/bin/node meshcentral
`-1485 /usr/bin/node /root/meshcentral/node_modules/meshcentral/meshcentral --launch 1473
Jun 23 16:49:30 meshcentral node[1473]: Code signed MeshCmd64.exe. Jun 23 16:49:30 meshcentral node[1473]: Code signed MeshService.exe. Jun 23 16:49:30 meshcentral node[1473]: Code signed MeshService64.exe. Jun 23 16:49:33 meshcentral node[1473]: Code signed MeshCmd.exe. Jun 23 16:49:36 meshcentral node[1473]: Code signed MeshCmdARM64.exe. Jun 23 16:49:36 meshcentral node[1473]: MeshCentral Intel(R) AMT server running on remote.example.com:44330, alias port 4433. Jun 23 16:49:36 meshcentral node[1473]: MeshCentral HTTP server running on port 4430, alias port 443. Jun 23 16:49:36 meshcentral node[1473]: Loaded web certificate from "https://remote.example.com:443/", host: "remote.example.com" Jun 23 16:49:36 meshcentral node[1473]: SHA384 cert hash: 306b824ab8ba70ae49a609cfb827a9df68db3c9d165a924d6b772070dbcef5ca288d31400609ac147bbd7cab7a9dc00f Jun 23 16:49:36 meshcentral node[1473]: SHA384 key hash: 9d59e535a5b179e64b678875e7b69cc598aef257956912afd86d8e9abd601df17529e04735692d379bf4315524b6d5bf
Was there a change/update to the discovery and connectivity logic for AMT in v1.1.22 and later??
P.S - I've created a clone of LXC machine with the v1.1.20 installation via my virtualization host and making NO OTHER changes besides updating Meshcentral to v1.1.24 via the self-update feature, all the agent-based machines come online however, the AMT machines never come back online.
The Only variable is the meshcentral version.
Somethings changed.
Code signed MeshService64.exe
It's codesigned all ur exes again? And this only happens if a dns name or certificate has changed?
How did u update?
Do u have any agents, or is it only AMT devices?
Edit.
Also check its full output journalctl -u meshcentral.service
Try the following
cd /root/meshcentral
node node_modules/meshcentral --debug amt
See what logs appear? Anything helpful there?
@si458 I ran the node node_modules/meshcentral --debug amt
for both the v1.1.20 and v1.1.24 installations, results below. The v1.1.24 installation never outputs anything for AMT, which makes sense since the agents never come online.
The third output is after I power on the two machines, Servyr-M920q and Servyr-NUC11. There is an issue with the v1.1.24 installation when machines are off whereas this was working before in v1.1.20.
root@meshcentral:~# systemctl stop meshcentral.service
root@meshcentral:~# cd meshcentral/
root@meshcentral:~/meshcentral# node node_modules/meshcentral --debug amt
WARNING: MeshCentral will require Node v16 or above in the future, your current version is v14.21.1.
MeshCentral HTTP redirection server running on port 800.
MeshCentral v1.1.20, Hybrid (LAN + WAN) mode.
Code signed MeshService64.exe.
Code signed MeshCmd64.exe.
Code signed MeshCmd.exe.
Code signed MeshCmdARM64.exe.
Code signed MeshServiceARM64.exe.
Code signed MeshService.exe.
MeshCentral Intel(R) AMT server running on remote.example.com:44330, alias port 4433.
MeshCentral HTTP server running on port 4430, alias port 443.
Loaded web certificate from "https://remote.example.com:443/", host: "remote.example.com"
SHA384 cert hash: 306b824ab8ba70ae49a609cfb827a9df68db3c9d165a924d6b772070dbcef5ca288d31400609ac147bbd7cab7a9dc00f
SHA384 key hash: 9d59e535a5b179e64b678875e7b69cc598aef257956912afd86d8e9abd601df17529e04735692d379bf4315524b6d5bf
AMT: Start Management node//yPbYCzLdse0WxhjZOwu$@IDKJm2PggFbtQtqEJfjZPyzwjOpWeUvpRYqYTjFIWjK 3
AMT: Servyr-NUC11 Checking Intel AMT state...
AMT: Servyr-NUC11 Attempt Initial Contact Local
AMT: Servyr-NUC11 Attempt Initial Local Contact 3 Servyr-NUC11.example.org
AMT: Servyr-NUC11 Direct-Connect TLS Servyr-NUC11.example.org admin
AMT: Start Management node//VAfyxz1KHnxHyLcW7XtkUcD$TCN32h7SAxMvZMjOOLHXrwAQ8QemHcAlZFpls3Ub 3
AMT: Start Management node//GL6B7$2vKcDtxcftjwpR1By25a6a5X98tO9a$X0ZLlKL3375xV$iIbKm@ED0bRj$ 3
AMT: Servyr-DevBox Checking Intel AMT state...
AMT: Servyr-DevBox Attempt Initial Contact Local
AMT: Servyr-DevBox Attempt Initial Local Contact 3 Servyr-DevBox.example.org
AMT: Servyr-DevBox Direct-Connect TLS Servyr-DevBox.example.org admin
AMT: Servyr-M920q Checking Intel AMT state...
AMT: Servyr-M920q Attempt Initial Contact Local
AMT: Servyr-M920q Attempt Initial Local Contact 3 Servyr-M920q.example.org
AMT: Servyr-M920q Direct-Connect TLS Servyr-M920q.example.org admin
AMT: Servyr-NUC11 Initial Contact Response 200
AMT: Servyr-NUC11 Intel AMT connected with TLS.
AMT: Servyr-DevBox Initial Contact Response 200
AMT: Servyr-DevBox Intel AMT connected with TLS.
AMT: Servyr-M920q Initial Contact Response 200
AMT: Servyr-M920q Intel AMT connected with TLS.
AMT: Servyr-NUC11 Fetching hardware inventory.
AMT: Servyr-DevBox Fetching hardware inventory.
AMT: Servyr-M920q Fetching hardware inventory.
AMT: Servyr-NUC11 Done.
AMT: Servyr-M920q Done.
AMT: Servyr-DevBox Done.
root@meshcentral:~/meshcentral# node node_modules/meshcentral --debug amt
MeshCentral HTTP redirection server running on port 800.
MeshCentral v1.1.24, Hybrid (LAN + WAN) mode.
Code signed MeshService64.exe.
Code signed MeshService.exe.
Code signed MeshCmdARM64.exe.
Code signed MeshServiceARM64.exe.
Code signed MeshCmd64.exe.
Code signed MeshCmd.exe.
MeshCentral Intel(R) AMT server running on remote.example.com:44330, alias port 4433.
MeshCentral HTTP server running on port 4430, alias port 443.
Loaded web certificate from "https://remote.example.com:443/", host: "remote.example.com"
SHA384 cert hash: 306b824ab8ba70ae49a609cfb827a9df68db3c9d165a924d6b772070dbcef5ca288d31400609ac147bbd7cab7a9dc00f
SHA384 key hash: 9d59e535a5b179e64b678875e7b69cc598aef257956912afd86d8e9abd601df17529e04735692d379bf4315524b6d5bf
root@meshcentral:~/meshcentral# node node_modules/meshcentral --debug amt
MeshCentral HTTP redirection server running on port 800.
MeshCentral v1.1.24, Hybrid (LAN + WAN) mode.
Code signed MeshCmd64.exe.
Code signed MeshCmdARM64.exe.
Code signed MeshServiceARM64.exe.
Code signed MeshService.exe.
Code signed MeshService64.exe.
Code signed MeshCmd.exe.
MeshCentral Intel(R) AMT server running on remote.example.com:44330, alias port 4433.
MeshCentral HTTP server running on port 4430, alias port 443.
Loaded web certificate from "https://remote.example.com:443/", host: "remote.example.com"
SHA384 cert hash: 306b824ab8ba70ae49a609cfb827a9df68db3c9d165a924d6b772070dbcef5ca288d31400609ac147bbd7cab7a9dc00f
SHA384 key hash: 9d59e535a5b179e64b678875e7b69cc598aef257956912afd86d8e9abd601df17529e04735692d379bf4315524b6d5bf
AMT: Start Management node//GL6B7$2vKcDtxcftjwpR1By25a6a5X98tO9a$X0ZLlKL3375xV$iIbKm@ED0bRj$ 3
AMT: Servyr-M920q Checking Intel AMT state...
AMT: Servyr-M920q Attempt Initial Contact Local
AMT: Servyr-M920q Attempt Initial Local Contact 3 Servyr-M920q.example.org
AMT: Servyr-M920q Direct-Connect TLS Servyr-M920q.example.org admin
AMT: Servyr-M920q Initial Contact Response 200
AMT: Servyr-M920q Intel AMT connected with TLS.
AMT: Servyr-M920q Fetching hardware inventory.
AMT: Servyr-M920q Done.
AMT: Start Management node//KR@Lev7CYbYEZDfDZL0sEjf$CGEvbrGAuIHiLHGOiAu6UJ1vW5X36JbSL9EkC@az 3
AMT: Servyr-NUC11 Checking Intel AMT state...
AMT: Servyr-NUC11 Attempt Initial Contact Local
AMT: Servyr-NUC11 Attempt Initial Local Contact 3 Servyr-NUC11.example.org
AMT: Servyr-NUC11 Direct-Connect TLS Servyr-NUC11.example.org admin
AMT: Servyr-NUC11 Initial Contact Response 200
AMT: Servyr-NUC11 Intel AMT connected with TLS.
AMT: Servyr-NUC11 Fetching hardware inventory.
AMT: Servyr-NUC11 Done.
Something isn't right with ur setup?
Why is it code-signing the applications every time u start meshcentral?
It should only do that once and never again.
What's the layout of ur meshcentral-data folder?
P.S - Below is the output when I issue a AMT power off command for the two machines:
...
AMT: Servyr-NUC11 Fetching hardware inventory.
AMT: Servyr-NUC11 Done.
performPowerAction node//GL6B7$2vKcDtxcftjwpR1By25a6a5X98tO9a$X0ZLlKL3375xV$iIbKm@ED0bRj$ 8
AMT: Servyr-M920q performPowerAction 8
performPowerAction node//KR@Lev7CYbYEZDfDZL0sEjf$CGEvbrGAuIHiLHGOiAu6UJ1vW5X36JbSL9EkC@az 8
AMT: Servyr-NUC11 performPowerAction 8
AMT: Servyr-M920q Stop Management node//GL6B7$2vKcDtxcftjwpR1By25a6a5X98tO9a$X0ZLlKL3375xV$iIbKm@ED0bRj$ 3
AMT: Servyr-M920q Remove device node//GL6B7$2vKcDtxcftjwpR1By25a6a5X98tO9a$X0ZLlKL3375xV$iIbKm@ED0bRj$ 3 1
AMT: Servyr-NUC11 Stop Management node//KR@Lev7CYbYEZDfDZL0sEjf$CGEvbrGAuIHiLHGOiAu6UJ1vW5X36JbSL9EkC@az 3
AMT: Servyr-NUC11 Remove device node//KR@Lev7CYbYEZDfDZL0sEjf$CGEvbrGAuIHiLHGOiAu6UJ1vW5X36JbSL9EkC@az 3 1
Something isn't right with ur setup?
Why is it code-signing the applications every time u start meshcentral?
It should only do that once and never again.
What's the layout of ur meshcentral-data folder?
I understand that "Code signed" thing seems unusual to you but I'm primarily concerned about the AMT agents not showing online and since the "Code signed" output is the same for both versions, then logic would imply that it might not be the cause of the AMT issue. Just my opinion.
root@meshcentral:~/meshcentral/meshcentral-data# ls -lha
total 1.1M
drwxr-xr-x 3 root root 23 Jun 23 17:47 .
drwxr-xr-x 7 root root 8 Jun 13 23:33 ..
-rw-r--r-- 1 root root 2.5K Jun 27 2020 agentserver-cert-private.key
-rw-r--r-- 1 root root 1.5K Jun 27 2020 agentserver-cert-public.crt
-rw-r--r-- 1 root root 2.5K Jun 8 2022 codesign-cert-private.key
-rw-r--r-- 1 root root 1.6K Jun 8 2022 codesign-cert-public.crt
-rw-r--r-- 1 root root 4.7K Jun 27 2020 config.exampple.json
-rw-r--r-- 1 root root 4.9K Aug 8 2023 config.json
-rw-r--r-- 1 root root 4.7K Jun 27 2020 config.json~
-rw-r--r-- 1 root root 3.6M Jun 23 17:44 meshcentral-events.db
-rw-r--r-- 1 root root 0 Jul 16 2021 meshcentral-plugins.db
-rw-r--r-- 1 root root 44K Jun 23 17:44 meshcentral-power.db
-rw-r--r-- 1 root root 245K Jun 23 17:44 meshcentral-stats.db
-rw-r--r-- 1 root root 303K Jun 23 17:44 meshcentral.db
-rw-r--r-- 1 root root 1.7K Jun 27 2020 mpsserver-cert-private.key
-rw-r--r-- 1 root root 1.5K Jun 27 2020 mpsserver-cert-public.crt
-rw-r--r-- 1 root root 2.5K Jun 27 2020 root-cert-private.key
-rw-r--r-- 1 root root 1.6K Jun 27 2020 root-cert-public-backup.crt
-rw-r--r-- 1 root root 1.6K Jun 27 2020 root-cert-public.crt
-rw-r--r-- 1 root root 198 Jun 23 17:34 serverstate.txt
drwxr-xr-x 2 root root 8 Apr 24 2023 signedagents
-rw-r--r-- 1 root root 2.5K Jun 27 2020 webserver-cert-private.key
-rw-r--r-- 1 root root 1.7K Jun 27 2020 webserver-cert-public.crt
root@meshcentral:~/meshcentral/meshcentral-data#
Try rerunning amtconfig
in the console tab of one of the devices and see what it outputs (might take a min to finish with logs)
Then shut the device down and see if u can still see it online
I'm confused why it says Remov Device?
I don't have an amt 11 device only 7, and I'm not seeing this issue? But will look more into it tomorrow if I can
The problem is the code-siging only happens if the dns name changes, which in turn regenerates certificates, like the amt certificate, which in turn would explain WHY they can't connect because the certificate has changed?
Try also running node node_modules/meshcentral --debug
This will output ALL debug logs and might give more of an insight!
Note the will be loads of logs too
Try rerunning
amtconfig
in the console tab of one of the devices and see what it outputs (might take a min to finish with logs)Then shut the device down and see if u can still see it online
I'm confused why it says Remove Device?
I don't have an amt 11 device only 7, and I'm not seeing this issue? But will look more into it tomorrow if I can
> amtconfig
Enabled live view of Intel AMT configuration events, "amtevents off" to disable.
13:54:13, LMS tunnel start.
13:54:14, Checking Intel AMT state...
13:54:17, Intel AMT connected.
13:54:23, Done.
13:54:23, LMS tunnel closed.
> amtconfig
Enabled live view of Intel AMT configuration events, "amtevents off" to disable.
13:55:59, LMS tunnel start.
13:56:00, Checking Intel AMT state...
13:56:03, Intel AMT connected.
13:56:10, Done.
13:56:10, LMS tunnel closed.
The problem is the code-siging only happens if the dns name changes, which in turn regenerates certificates, like the amt certificate, which in turn would explain WHY they can't connect because the certificate has changed?
Understood.
Try also running
node node_modules/meshcentral --debug
This will output ALL debug logs and might give more of an insight!
Note the will be loads of logs too
Output below, v1.1.24
The full log above, did you switch the computers on? Where they already switched on? Did they show in the Web panel as online but offline?
The full log above, did you switch the computers on? Where they already switched on? Did they show in the Web panel as online but offline?
FYI - I have a few of these mini-PCs with Intel ME, if you're in the states, I can send you one on loan to use for testing. I appreciate the work that's been done with MeshCentral and I wouldn't mind lending one for a good cause. The Intel ME version would be v12.0.90.
Oh wow, any hardware I can use for testing is appreciated! Do email me! Check my github page
sorry just realised u said states, dont worry about it, im UK 🇬🇧
these are the changes we did between 1.1.20 and 1.1.24 - https://github.com/Ylianst/MeshCentral/compare/1.1.20...1.1.24
we changed 0% to do with AMT so im totally confused?
altho my test machine which uses AMT 7, is showing as Intel AMT Cira
and Hybernating
are your AMT machines local to your meshcentral server? because in theory as ur using a reverse proxy they should be connecting over CIRA as they arent local to meshcentral
also what is ur group settings for AMT? these are mine
Just to interject- I am running 1.1.24 and all my AMT devices are working properly. I don't use a reverse proxy and my agents are code signed.
Just to interject- I am running 1.1.24 and all my AMT devices are working properly. I don't use a reverse proxy and my agents are code signed.
@PathfinderNetworks No problem, the additional data point is appreciated. Can you share the same information about your setup that I did in the original issue creation? Also, can you add the following bit of information about your setup for comparison's sake?
Thanks again and appreciate any information that you can share.
Describe the bug After upgrading from 1.1.20 to 1.1.24 my AMT agents are no longer showing as connected (online) on the "My Devices" page. This is while the agents are powered down into one of soft-off states (e.g. S5). Only after powering on the machines, do the AMT agent shows as online and I'm able to "Connect" to the Intel AMT portal.
To Reproduce Steps to reproduce the behavior:
Expected behavior I expected that after updating to v1.1.24 my previously registered AMT Agents would continue to show as online and that I would be able to access the Intel AMT portal even when the machine is in one of the soft-off states (e.g. S3-S5) without the machine having to be powered-on.
Screenshots If applicable, add screenshots to help explain your problem.
AMT Agents online (v1.1.20)
AMT Agents offline (after updating to v1.1.24)
Server Software (please complete the following information):
Client Device (please complete the following information):
Remote Device (please complete the following information):
Additional context Add any other context about the problem here.
Your config.json file