Closed SemoTech closed 2 years ago
I'm very unfamiliar with name resolution at LAN level but have you tried explicitly adding the port to the mesh.server.com in your Meshcentral config?
Thanks for the reply @Typhonragewind.
Name resolution on LAN is working just fine and is properly resolved to the IP address of the Docker server (Ubuntu server). The only container on the server using mapped port 443 is MeshCentral. Currently port 443 is mapped to 8086 in order to access the web interface of MeshCentral.
Not sure what your mean by “add the port to mesh.server.com”. The agent was created by the MeshCentral server, and is attempting to connect back to the server (after properly resolving the LAN IP) on port 443. Is there a different port it should use, and if so, how can that be specified and where exactly?
Thank you kindly.
No, what I meant is, in your meshcentral config (either the file or the HOSTNAME variable in the docker-compose), do you have https://mesh.server.com or https://mesh.server.com:8086 ?
Oh, sorry @Typhonragewind, I misunderstood.
The HOSTNAME (in Docker-compose) is set to just: mesh.server.com The separate port variable is set to the default of: 443 and was never changed.
It's ok, I'm not feeling completely eloquent today (monday vibes)
Try changing the HOSTNAME to https://mesh.server.com:8086 on a fresh install (without reusing the currently written meshcentral config.json) and check if that works
LOL, no worries @Typhonragewind, Mondays get everyone :-)
OK, I will deploy a new instance as you specified and see if that works.
What should I set the separate port option in the Docker-compose config? Leave it as 443?
Thank you.
What should I set the separate port option in the Docker-compose config? Leave it as 443?
yeah, leave it as 8086:443
Ok. Sounds good. Will try that and post back here with results. Thanks again.
Hi @Typhonragewind,
OK, I tried twice with both the standard deployment, and the MongoDB version. However after using "https://mesh.server.com:8086" for HOSTNAME in docker-compose I cannot access the web interface in my browser anymore! At least I used to be able to when I just had "mesh.server.com" as the HOSTNAME.
Also, connecting to the Docker server via SSH and issuing the command: docker port meshcentral
used to show the 8086 -> 443 port mapping, but now shows nothing... Please advise. Thank you.
Hi @Typhonragewind, I figured out why the ports are not mapped and I am unable to connect. Changing the HOSTNAME variable as you broke the self certificate and the container has been in a self restart cycle since creation. Here is an excerpt from the logs:
Is there another way to fix allowing the Agent to reach the server?
thanks.
Honestly I'm kinda stumped on this one. Mind sharing your docker-compose.yml and config.json (sanitized ofc)?
Hey @Typhonragewind, apparently I had a dns resolution issue, and that is why the certificate error occurred. I corrected it.
Here is the info you wanted to review:
docker-compose.yml
version: '3'
services:
mongodb:
container_name: meshcentral_db
restart: always
image: mongo:latest
expose:
- 27017
volumes:
- '/opt/meshcentral/database:/data/db'
meshcentral:
restart: always
container_name: meshcentral
depends_on:
- 'mongodb'
image: typhonragewind/meshcentral:mongodb
ports:
- 8086:443
- 8087:800
environment:
- HOSTNAME=mesh.server.com #your hostname - sanitized
- REVERSE_PROXY=false #set to your reverse proxy IP if you want to put meshcentral behind a reverse proxy
- REVERSE_PROXY_TLS_PORT=443
- IFRAME=false #set to true if you wish to enable iframe support
- ALLOW_NEW_ACCOUNTS=false #set to false if you want disable self-service creation of new accounts besides the first (admin)
- WEBRTC=true #set to true to enable WebRTC - per documentation it is not officially released with meshcentral, but is solid enough to work with. Use with caution
- NODE_ENV=production
volumes:
- ./meshcentral/data:/opt/meshcentral/meshcentral-data
- ./meshcentral/user_files:/opt/meshcentral/meshcentral-files
config.json from the meshcentral container:
root@18c617d5db39:/opt/meshcentral/meshcentral-data# cat config.json
{
"$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
"settings": {
"cert": "mesh.server.com",
"_WANonly": true,
"_LANonly": true,
"_sessionKey": "MyReallySecretPassword1",
"port": 443,
"_aliasPort": 443,
"redirPort": 80,
"_redirAliasPort": 80,
"AgentPong": 300,
"TLSOffload": false,
"SelfUpdate": false,
"AllowFraming": "false",
"mongodb": "mongodb://mongodb:27017/mesh",
"mongodbcol": "mesh",
"WebRTC": "false"
},
"domains": {
"": {
"_title": "MyServer",
"_title2": "Servername",
"_minify": true,
"NewAccounts": "false",
"_userNameIsEmail": true,
"_certUrl": "my.reverse.proxy"
}
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "myemail@mydomain.com",
"_names": "myserver.mydomain.com",
"production": false
}
}
Log from meshcentral container when using "mesh.server.com" as the HOSTNAME:
Installing archiver@4.0.2...
Installing mongodb@4.1.0...
Installing saslprep...
Installing otplib@10.2.3...
MeshCentral HTTP redirection server running on port 80.
Generating certificates, may take a few minutes...
Generating root certificate...
Generating HTTPS certificate...
Generating MeshAgent certificate...
Generating Intel AMT MPS certificate...
MeshCentral v1.0.10, Hybrid (LAN + WAN) mode, Production mode.
MeshCentral Intel(R) AMT server running on mesh.server.com:4433.
MeshCentral HTTPS server running on mesh.server.com:443.
Log from meshcentral container if using "https://mesh.server.com:8086" as the HOSTNAME:
Installing archiver@4.0.2...
Installing mongodb@4.1.0...
Installing saslprep...
Installing otplib@10.2.3...
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.0.10, Hybrid (LAN + WAN) mode, Production mode.
MeshCentral Intel(R) AMT server running on mesh.server.com:4433.
MeshCentral HTTPS server running on mesh.server.com:443.
Got the ports to work again:
root@server:/# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e9f97bd86004 typhonragewind/meshcentral:mongodb "bash /opt/meshcentr…" 9 minutes ago Up 9 minutes 80/tcp, 0.0.0.0:8086->443/tcp, 0.0.0.0:8087->800/tcp meshcentral
5ff9c32a325b mongo:latest "docker-entrypoint.s…" 26 minutes ago Up 26 minutes 27017/tcp meshcentral_db
root@server:/# docker port meshcentral
800/tcp -> 0.0.0.0:8087
443/tcp -> 0.0.0.0:8086
Regardless if I use the plain domain name for the server HOSTNAME, or "https://mesh.server.com:8086", I still cannot get the agent to download via wget:
Admin@Mac local $ sudo wget -O meshagent "https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005"
Password:
--2022-05-04 09:22:57-- https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005
Resolving mesh.server.com (mesh.server.com)... 192.168.2.50
Connecting to mesh.server.com (mesh.server.com)|192.168.2.50|:443... failed: Connection refused.
So still getting "443... failed: Connection refused." Any ideas?
@SemoTech Hey, sorry for the delay, been having some quite busy days. That said, your problem has me a bit stumped. Can you confirm if your firewall or other security software is not blocking the connection?
@Typhonragewind no worries and thanks for the reply. I assumed you were busy so did not want to bother you. I just waited.
Both the Docker server (ubuntu) and the Mac I attempted to connect and download the agent from are on the same wired LAN. The Mac has an inbound firewall but I even tried to disable it with no luck. For reference, several other systems including PiHole, Homebridge, HomeAssistant, Portainer, etc... run in docker on the same Ubuntu server and have no issues. I even tried to shut them all down and leave just MeshCentral. Anyway, I am totally stumped as well.
@SemoTech I think I may have figured it out. Try changing this line in your config: "_certUrl": "my.reverse.proxy"
to "certUrl": "https://your.reverseproxyIP:port"
(note, not the domain)
Edit: Oh, and you'll probably need to reinstall the agent
@SemoTech I think I may have figured it out. Try changing this line in your config:
"_certUrl": "my.reverse.proxy"
to"certUrl": "https://your.reverseproxyIP:port"
(note, not the domain)Edit: Oh, and you'll probably need to reinstall the agent
Thanks @Typhonragewind I will try.
For reference, the agent was never installed as it would not download via wget, since the connection could not be established. See here:
Admin@Mac local $ sudo wget -O meshagent "https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005"
Password:
--2022-05-04 09:22:57-- https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005
Resolving mesh.server.com (mesh.server.com)... 192.168.2.50
Connecting to mesh.server.com (mesh.server.com)|192.168.2.50|:443... failed: Connection refused.
So, since the IP of the MeshCentral Docker server is: 192.168.2.50, would "certUrl" be: "https://192.168.2.50:443" OR "https://192.168.2.50:8086" ???
Thanks.
@SemoTech I keep thinking you're on a reverse-proxy, don't know why 😅
This does shed light in the issue, you're trying to use https without providing a certificate. You can either create your own and point to it in the configuration file, or let meshcentral handle it with its Let's Encrypt feature. Check out Section 6.8 (page 27) of the installers manual on how to do it - https://info.meshcentral.com/downloads/MeshCentral2/MeshCentral2InstallGuide.pdf
Thanks @Typhonragewind Since this is running on a LAN only, Let’sEncrypt would not work, but there seems to be a LAN setting and it also seems the certificate is auto-generated.
See here:
Think this will work without me needing to create a cert? And if so, how would the generated cert be trusted?
@SemoTech In theory yes, that should allow you to work in a LAN-only environment by making use of self-signed certificates. As to how it would be trusted, self-signed certificates are not the best practice when exposing services to the internet, but they should be more than enough for a internal network.
Hi @Typhonragewind Sadly that did not work either.
I made the LAN only and FastCert changes, and the internal log shows that MeshCentral went from LAN/WAN to just LAN on port 443, yet I get the same "443... failed: Connection refused." error when attempting to download the agent via wget!
Here is the bottom of the container logs (last 2 are after the changes in config.json):
MeshCentral v1.0.10, Hybrid (LAN + WAN) mode, Production mode.
MeshCentral Intel(R) AMT server running on mesh.server.com:4433.
MeshCentral HTTPS server running on mesh.server.com:443.
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.0.10, LAN mode, Production mode.
MeshCentral HTTPS server running on port 443.
@SemoTech Is there any way that you can test it with a machine of a different OS?
Hi @Typhonragewind, thanks for trying.
Sadly I only have one Ubuntu Docker server (Mac Mini) that I can deploy this on. Ubuntu is running native.
Here is the SSH login summary:
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-110-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Fri 20 May 2022 08:06:23 AM PDT
System load: 0.27
Usage of /: 11.5% of 218.57GB
Memory usage: 10%
Swap usage: 0%
Temperature: 53.0 C
Processes: 243
Users logged in: 0
@SemoTech Hm, that complicates testing. How about you use an USB live boot of a linux distro on your mac mini, just so we can test if it is some iOS stuff affecting the client?
Sure @Typhonragewind but that will take some doing, and also take down my fully functional (aside from MeshCentral) Docker server and all containers. Also, even if this works I still need to make it work on the Ubuntu Docker server (MacMini) for it to be usable to me....
I was hoping since the error seems very specific to "port 443 being blocked or unreachable" there was something to test in the Ubuntu/Docker firewall ports or the like...
Thank you.
@SemoTech Oh, i misread it then, i thought you had you ubuntu server and a MacMini. So you're trying to use meshcentral server, linked to the meshcentral client, in the same host ?
Hi @Typhonragewind no worries, lots of variables.
I am using the MacMini as a bare-metal server, hardwired to the LAN. It is running Ubuntu Server 20.04.4 LTS and latest Docker. MeshCentral (your Docker version) was installed on it through Portainer, and seems to be running just fine (after I fixed the DNS resolution issue) :-) I am also using a MacPro hardwired on the same LAN, running Monterey 12.3.1 as the "client" to have the agent run on, and connect to Mesh Central on the MacMini.
The problem is my MacPro is unable to download the agent in order to being the install.
Originally I downloaded the Agent installer manually from the MacMini MeshCentral server web interface, and installed it on the MacPro. However when it did not show up in the server as a machine to control, I tried to use the wget method to grab the agent (thus verifying connectivity between the 2 systems) and that's when the "443... failed: Connection refused." issue was discovered and remains unsolved.
Thank you.
@SemoTech Ah, i get it now, thanks for summarizing, i was getting lost.
Try changing the "port" value in the settings to 8086 and restart Meshcentral
Glad it's clearer now @Typhonragewind
So are you saying change this:
root@18c617d5db39:/opt/meshcentral/meshcentral-data# cat config.json
{
"$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
"settings": {
"cert": "mesh.server.com",
"_WANonly": true,
"_LANonly": true,
"_sessionKey": "MyReallySecretPassword1",
"port": 443,
"_aliasPort": 443,
"redirPort": 80,
"_redirAliasPort": 80,
"AgentPong": 300,
"TLSOffload": false,
"SelfUpdate": false,
"AllowFraming": "false",
"mongodb": "mongodb://mongodb:27017/mesh",
"mongodbcol": "mesh",
"WebRTC": "false"
},
"domains": {
"": {
"_title": "MyServer",
"_title2": "Servername",
"_minify": true,
"NewAccounts": "false",
"_userNameIsEmail": true,
"_certUrl": "my.reverse.proxy"
}
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "myemail@mydomain.com",
"_names": "myserver.mydomain.com",
"production": false
}
}
To This:
root@18c617d5db39:/opt/meshcentral/meshcentral-data# cat config.json
{
"$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
"settings": {
"cert": "mesh.server.com",
"_WANonly": true,
"_LANonly": true,
"_sessionKey": "MyReallySecretPassword1",
"port": 8086,
"_aliasPort": 443,
"redirPort": 80,
"_redirAliasPort": 80,
"AgentPong": 300,
"TLSOffload": false,
"SelfUpdate": false,
"AllowFraming": "false",
"mongodb": "mongodb://mongodb:27017/mesh",
"mongodbcol": "mesh",
"WebRTC": "false"
},
"domains": {
"": {
"_title": "MyServer",
"_title2": "Servername",
"_minify": true,
"NewAccounts": "false",
"_userNameIsEmail": true,
"_certUrl": "my.reverse.proxy"
}
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "myemail@mydomain.com",
"_names": "myserver.mydomain.com",
"production": false
}
}
@SemoTech yup, that's it. But i forgot, in addition to that you also have to change the 443 port in the docker compose to 8086, so it would be like 8086:8086
Got it @Typhonragewind,
So both
version: '3'
services:
mongodb:
container_name: meshcentral_db
restart: always
image: mongo:latest
expose:
- 27017
volumes:
- '/opt/meshcentral/database:/data/db'
meshcentral:
restart: always
container_name: meshcentral
depends_on:
- 'mongodb'
image: typhonragewind/meshcentral:mongodb
ports:
- 8086:8086
- 8087:800
environment:
- HOSTNAME=mesh.server.com #your hostname - sanitized
- REVERSE_PROXY=false #set to your reverse proxy IP if you want to put meshcentral behind a reverse proxy
- REVERSE_PROXY_TLS_PORT=443
- IFRAME=false #set to true if you wish to enable iframe support
- ALLOW_NEW_ACCOUNTS=false #set to false if you want disable self-service creation of new accounts besides the first (admin)
- WEBRTC=true #set to true to enable WebRTC - per documentation it is not officially released with meshcentral, but is solid enough to work with. Use with caution
- NODE_ENV=production
volumes:
- ./meshcentral/data:/opt/meshcentral/meshcentral-data
- ./meshcentral/user_files:/opt/meshcentral/meshcentral-files
AND
root@1f175bf8b8ed:/opt/meshcentral/meshcentral-data# cat config.json
{
"$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
"settings": {
"FastCert": true,
"_WANonly": true,
"LANonly": true,
"_sessionKey": "MyReallySecretPassword1",
"port": 8086,
"_aliasPort": 443,
"redirPort": 80,
"_redirAliasPort": 80,
"AgentPong": 300,
"TLSOffload": false,
"SelfUpdate": false,
"AllowFraming": "false",
"mongodb": "mongodb://mongodb:27017/mesh",
"mongodbcol": "mesh",
"WebRTC": "false"
},
"domains": {
"": {
"_title": "MyServer",
"_title2": "Servername",
"_minify": true,
"NewAccounts": "false",
"_userNameIsEmail": true,
"_certUrl": "my.reverse.proxy"
}
},
"_letsencrypt": {
"__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
"_email": "myemail@mydomain.com",
"_names": "myserver.mydomain.com",
"production": false
}
}
Correct?
Hi @Typhonragewind both port changes listed above have been done!
meshcentral container log shows server listening to new 8086 instead of 443:
Installing archiver@4.0.2...
Installing mongodb@4.1.0...
Installing saslprep...
Installing otplib@10.2.3...
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.0.10, LAN mode, Production mode.
MeshCentral HTTPS server running on port 8086.
Running wget again from the MacPro "client", shows SAME 443 error!
Admin@Mac local $ sudo wget -O meshagent "https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005"
Password:
--2022-05-20 10:19:31-- https://mesh.server.com/meshagents?id=LrbUtkrDQTSUL%40OVmWkLPPekWNnOyDhN5tT4WdjrvJWqh3rrBTs&installflags=0&meshinstall=10005
Resolving mesh.server.com (mesh.server.com)... 192.168.2.50
Connecting to mesh.server.com (mesh.server.com)|192.168.2.50|:443... failed: Connection refused.
I give up!
@SemoTech I must admit defeat as well, as i'm truly out of ideas xD
But do try your luck on MeshCentral repo, They may help you better there - https://github.com/Ylianst/MeshCentral
Sorry I couldn't help.
Thanks @Typhonragewind appreciate all your efforts. I hope they don't tell me its a docker issue :-)
Have you ever solved this? I have the exact problem and tried every combination like you have...
I can access the web ui fine, it's just that the agents can connect.
@klibansky You'll have to check SemoTech's issue opened on the Meshcentral repo - https://github.com/Ylianst/MeshCentral/issues/4027
Have you ever solved this? I have the exact problem and tried every combination like you have...
I can access the web ui fine, it's just that the agents can connect.
Hey @klibansky, yes, it is resolved but it was a multitude of issues, from lack of port 80 from my ISP preventing a proxy & Let's Encrypt Certificates, to IP blocks in the CSF firewall on the cPanel server, to Docker incompatibilities with cPanel / WHM. The easiest functional deployment was using Vultr, but I did not want to have to pay for it when I knew there had to be a way for me to host it myself.
In the end I managed to get it all working using a local instance of Docker on Ubuntu Linux, with caddy as the Proxy server to get around the blocked port 80 limitation and make Let's Encrypt work. Took forever but it is working now...
Is there anything particular about your config file and settings?
When I leave all the meshcentral settings as default but just expose the 443 (internal) port to 4433 (external) and have mesh.server.com:443 > 4433 (which in turn translates that to the meshcentral internal 443 port) via Nginx reverse proxy manager, I can reach the web UI but the agents are not connecting. Let encrypt works automatically from within the NginX Proxy Manager docker image.
The docker compose file is not completely clear. Therese an environment variable REVERSE_PROXY= #set to your reverse proxy IP if you want to put meshcentral behind a reverse proxy
Like you should, I have a outward facing network connecting the reverse proxy to the internet, and then there is an internal network connecting the proxy with all in containers. So which IP to use?
So which IP to use?
In this case it would be the internal one, as in IP:PORT
Ok And do I still have to manually edit the config.json file and set tlsoffload?
So which IP to use?
In this case it would be the internal one, as in IP:PORT
So the internal docker ip of the reverse proxy + the exposed port? And do I use the ip of the external network or the internal network, since the reverse proxy container is connected to 2 networks.
Assuming you have your reverse proxy also in a docker container, that would be the IP of your host machine where the proxy resides and the outer IP (the exposed one) from the docker container. If you already tried to create the container you'll have to edit the config.json directly with these changes. I'm not sure what you want the tlsoffload for.
Hello @Typhonragewind let me start by saying thanks for all your hard work to get the docker container for MeshCentral.
I followed your instructions and using Portainer I attempted to setup both a simple install, and a MongoDB install. In both separate instances, no issues with the server interface using https://102.168.2.50:8086 (or by name at https://mesh.server.com:8086) on my LAN. Also no problem downloading an agent for MacOS, but I am unable to reach the server from the agent on the Mac. Both Mac and Docker Server are on the same LAN and can ping each other.
I had originally installed the Mac Agent after doing the MongoDB deployment, and was unsure why it did not appear in the MeshCentral console, and then I deleted it and installed the simple version and had the same issue. Finally I tried to download the agent directly from the server with "wget" and that's when I stumbled on the error.
I used:
sudo wget -O meshagent "https://mesh.server.com/meshagents?id=wnxIzXIRTjxOaBOFb%24nIv0LgM1sDsYV0029TtEj9Rr53%24FG9oJH8lLQSlnzoWngM&installflags=0&meshinstall=10005
and I got:
--2022-04-29 19:40:35-- https://mesh.server.com/meshagents?id=wnxIzXIRTjxOaBOFb%24nIv0LgM1sDsYV0029TtEj9Rr53%24FG9oJH8lLQSlnzoWngM&installflags=0&meshinstall=10005 Resolving mesh.server.com (mesh.server.com)... 192.168.2.50 Connecting to mesh.server.com (mesh.server.com)|192.168.2.50|:443... failed: Connection refused.
I checked in docker with "docker port and not other container is using port 443 except meshcentral.
Any idea why it is not working and how to fix?
Thanks in advance.