search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.14k stars 554 forks source link

Linux Agents won't connect to MC #1536

Closed vitko-bg closed 3 years ago

vitko-bg commented 4 years ago

When installing Agent on Linux , I have tried to install on Debian and Ubuntu( both are freshly installed VMs ) - no devices are coming up on MC and on both Linux machines meshagent.log is with the same content:

[2020-06-24 10:53:05 PM] Info: No certificate was found in db

On both linux machines installation went with no problems or errors.

The installation link I have on MC for Linux is:

(wget "https://remote.domain.com/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh || wget "https://remote.domain.com/meshagents?script=1" --no-proxy --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo -E ./meshinstall.sh https://remote.domain.com '$hnAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' || ./meshinstall.sh https://remote.domain.com '$hnAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

Output is here install_output.txt - all seems fine.

On MC server "systemctl status meshcentral" result is Jun 24 21:42:46 MC node[14809]: Agent connected with bad signature, holding connection (172.19.X.X4:44117). Jun 24 21:51:33 MC node[14809]: Agent connected with bad signature, holding connection (172.19.X.X3:59247).

Windows agents freshly downloaded and installed on PCs on the same network and at the same time are fine.

Not sure if the problem is in my MC installation?!?!!

vitko-bg commented 4 years ago

Today have tried it with completely different fresh installed Linux machine, outside of my network to prevent network related issues - result is same - "No certificate was found in db" Is that agent installation issue for not getting certificate or MC issue for not providing certificate to agent??? Is there a way to manually start agent with the certificate that is looking for?

krayon007 commented 4 years ago

That's not an error, that is just an informational entry. I put that there to log all the cases when an agent will generate a new certificate.

If the agent does not connect, that is a sepat issue. Did you verify the URL in the meshagent.msh file?

If you stop the service and run the binary directly, you'll see everything the agent is doing on the screen.

vitko-bg commented 4 years ago

Looks like it is fine and connected, but not on MC as device ......

root@myserver:/usr/local/mesh# ./meshagent MeshCentral2 Agent Connecting to: wss://remote.domain.com:449/agent.ashx Connected.

Also, in meshagent.msh ServerID and MeshServer are correct and are same as on Windows machines.........

So it is a MC problem then

Ylianst commented 4 years ago

Interesting. Can you stop the MeshAgent, rename "meshagent.db" to "meshagent.db.bak", start the agent again and let me know if it fixes it? I want to know if the meshagent.db got corrupt. A new device should show up on MeshCentral after you do this.

Thanks, Ylian

vitko-bg commented 4 years ago

@Ylianst - all done as per your advise, but same - no new device appeared on MC.... Anything else I can try? Is there any difference for Windows and Linux agents to connect to MC if MC is behind reverse proxy? On MC errors are more: image

Ylianst commented 4 years ago

Thanks for testing this. What server version are you on? I will setup my systems and do the same testing.

vitko-bg commented 4 years ago

It is a Debian 8, and MC is the latest. My full config files are in #1524 Have not tested this problem on additional domains, that is on my main domain.... Thanks for help. Distros I tried the agent are Debian 8, Ubuntu.... I think 18, and private server of my mate - no idea what linix on it :)

Ylianst commented 4 years ago

I just noticed in the code, you can add the following line in the settings section of config.json:

"ignoreagenthashcheck" : true

Let me know if that makes it work. I am looking at the code now, this problem is likely that the agent is seeing a TLS certificate that is different from what the server expects it to see.

vitko-bg commented 4 years ago

Hi @Ylianst - you are right, as soon as I added "ignoreagenthashcheck" : true - and restarted MC, all 3 Linux agents appeared on MC panel. So looks like hashcheck is the problem for Linux connections..... Is there a way to fix it, so MC won't ignore hashchecks? Also I believe that with that option, not only Linux agents are ignoring hashckecks, but Windows Agent also..... Isn't it?

Ylianst commented 4 years ago

Your correct that this option is not secure and hash check is not performed for all agents. Are you using a reverse proxy in front of MeshCentral? If so, can you anonymize and send over your config.json? - Thanks.

vitko-bg commented 4 years ago

Hi. It is already done - see full config in #1524 - MC, nginx, remote.domain.com.conf and sub1.domain.conf - all is there

Ylianst commented 4 years ago

Is it possible that all devices that are failing to connect are all part of the same domain? Your config.json has the "certurl" for the default domain but not for the two sub-domains. Go ahead and put "certurl":"https://127.0.0.1:443" in all sub-domains too.

Right now, devices connecting to the sub-domains will be assumed to have seen externally the default MeshCentral certificate.

Does all domains use the same TLS certificate externally? If not, there could be some extra considerations.

vitko-bg commented 4 years ago

Hi @Ylianst - those problematic linux machines are exactly in the main domain, so certurl for subdomains did not helped. In regards to certificates - for main domain and subdomans all are different ... BTW - just guessing now - why Windows machines are OK with current configuration? Possibly problem is somewhere in Agents - for Win and Linux machines, or the way how connection is established from those Agents.......

vitko-bg commented 4 years ago

If any additional information required - will provide all requested info if you need any.

vitko-bg commented 4 years ago

@Ylianst - any update on this issue? Any additional info/config will be provided... Please help to fix this. Thanks.

Ylianst commented 4 years ago

So, since your setup has different certificates in the reverse proxy for each subdomain, this is an interesting situation that I have not tested much. I want to check that for each "certurl", I get the right certificates for that given domain. I need to send the HTTPS request with the right details. I will make a few improvements and mail you back.

Ylianst commented 4 years ago

One thing I am adding is that you can now add the following line in the domain section:

"ignoreagenthashcheck":true

So, the plan is to remove the line from the settings section and add it to just the main domain. I will also add more debug options so that you can see the certificate common name that is loaded for each domain. That will make is easy to see if the incorrect certificate is loaded.

Ylianst commented 4 years ago

I have been tracing the code and I don't understand how it's possible to get a Agent connected with bad signature error that is caused by the server certificate being incorrect.

If your server is public and if it's ok, could you send me privately the agent.msh/agent.db files of a Linux machine that is not connecting? I am going to need to trace this in the agent and server code to see what is going on. My contact information is here.

In addition, I am adding server side helper commands to help debug this now.

vitko-bg commented 4 years ago

Hi @Ylianst - I have sent you my .msh and .db files over e-mail. Did you get them? Please advise if that is fixable - "ignoreagenthashcheck":true is working, but I prefer to stay on secure side and to switch it off ..... Thanks.

vitko-bg commented 4 years ago

@Ylianst I know you are very busy, sorry for asking again - did you managed by any chance to find what is wrong on configuration that I sent you? Thanks.

vitko-bg commented 4 years ago

I just now spotted something in installation link .... (wget "https://remote.domain.com/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh || wget "https://remote.domain.com/meshagents?script=1" --no-proxy --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://remote.domain.com 'xxxxxxxxxxxxxxxxxxxxxx'

Is that --no-proxy can somehow affect the installation and connection to MC server, is MC Server is behing NGINX reverse proxy?

vitko-bg commented 4 years ago

A quick update - did not found solution yet........... Cannot understand where is the problem based ............

vitko-bg commented 3 years ago

Still digging in with no result ... Small update, result in MC2 Trace with "ignoreagenthashcheck" : true 14:40:53 - AGENT: Verified agent connection to NGij273imUY (XXX.XXX.XXX.XXX:59917). 14:40:53 - AGENT: New agent at XXX.XXX.XXX.XXX:59917 14:40:44 - AGENT: Agent disconnect NGij273imUY (XXX.XXX.XXX.XXX:59193) id=6 14:39:44 - AGENT: Verified agent connection to NGij273imUY (XXX.XXX.XXX.XXX:59193). 14:39:44 - AGENT: New agent at XXX.XXX.XXX.XXX:59193 14:39:34 - AGENT: Agent disconnect NGij273imUY (XXX.XXX.XXX.XXX:40131) id=6

As soon as I change "ignoreagenthashcheck" : true to "false" the result is

15:21:13 - AGENT: Agent connected with bad signature, holding connection (XXX.XXX.XXX.XXX:42097). 15:21:13 - AGENT: New agent at XXX.XXX.XXX.XXX:42097

vitko-bg commented 3 years ago

Now I am on v0.7.31 and moved from NeDB to MongoDB - but the issue still persist.... @krayon007 - could you advise please what more I can do in order to investigate and resolve the issue? Thanks!

vitko-bg commented 3 years ago

Thanks to @Ylianst I have a workaround currently for this issue. The workaround solution is #2125 However - will appreciate if the root of the problem can be found and fixed.

vitko-bg commented 3 years ago

Managed to replicate issue on different server, different OS and using a different domain name. In short - I just got rid of "CertUrl": "https://127.0.0.1:443", for each domain in config.json and all Windows, Linux, Android, MAC based agents age connecting fine now. I am really happy that problem is finally resolved now on my server. Any information required to replicate the issue or any info about my config files - feel free to contact me.

vitko-bg commented 3 years ago

This issue is now completely fixed by Ylian - v0.7.69 resolves the problem with bad signature.