Agent bad web cert hash - after migrating to a different VM

tomsik-radek commented 9 months ago

Describe the bug After migrating MeshCentral to a new VM and redirecting my reverse proxy, I get the Agent bad web cert hash error. If I disable certificate checking agents connect.

Using typhonragewind/meshcentral (sqlite version) and Nginx Proxy Manager

To Reproduce Steps to reproduce the behavior:

Deploy the same docker-compose.yml
Move content of meshcentral-data from one VM to another to the exact same path
Change Nginx Proxy Manger configuration from meshcentral.domain.tld/http://vm1:7876 (external port) to meshcentral.domain.tld/http://meshcentral:443 (LetsEncrypt certificate, nginx proxy_set_header are the same)

Portainer docker log keeps saying

Agent bad web cert hash (Agent:e6985209e7 != Server:7bbe059e90 or 03f0a25da8), holding connection (172.18.0.5:58264).
Agent reported web cert hash:e6985209e7648f835345f3839735b10bbc8a979ece72fdaed9bf83a98f74ce604b9f93cd2b1534978ca5e27915fd66bd

Eventually ends up on

Failed to load web certificate at: "https://meshcentral.domain.tld:443", host: "meshcentral.domain.tld"

Expected behavior Agents connect

Server Software (please complete the following information):

OS: Debian Bookworm
Virtualization: Docker
Network: WAN mode, Nginx Proxy Manager, docker bridge network
Version: typhonragewind/meshcentral:1.1.13

Client Device (please complete the following information): All of them. Windows, Linux. LAN local and external remote (an example is a rented server in a datacenter)

Additional context

Nginx Proxy Manager has been moved to VM2, so a different LetsEncrypt certificate can't be a problem since it works with VM1 install of MeshCentral
172.18.0.5 in this case is the internal docker IP address of nginx-proxy-manager
config.json is identical between both VMs, as is content of meshcentral-data folders (I also tried copying just webserver, root and agentserver certificates, same behavior)
VM1 is the old VM, Mesh there is on a different docker network than NPM (so accessing outside port, that works)

Your config.json file Redacted anything that could contain my name

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "_maintenanceMode": true,
    "cert": "meshcentral.domain.tld",
    "SQLite3": true,
    "WANonly": true,
    "sessionKey": "null",
    "port": 443,
    "aliasPort": 443,
    "redirPort": 80,
    "redirAliasPort": 80,
    "AgentPong": 300,
    "TLSOffload": "127.0.0.1",
    "_ignoreAgentHashCheck": true,
    "SelfUpdate": false,
    "AllowFraming": true,
    "WebRTC": false,
    "nice404": true,
    "allowHighQualityDesktop": true,
    "publicPushNotifications": false,
    "trustedProxy": "nginx-proxy-manager"
  },
  "domains": {
    "": {
      "_siteStyle": 2,
      "title": "MeshCentral",
      "title2": "[redacted]",
      "_titlePicture": "title-sample.png",
      "_loginPicture": "title-sample.png",
      "mobileSite": true,
      "maxDeviceView": 200,
      "_unknownUserRootRedirect": "https://www.youtube.com/watch?v=2Q_ZzBGPdqE",
      "nightMode": 0,
      "ipkvm": false,
      "minify": true,
      "newAccounts": false,
      "_welcomeText": "Sample Welcome Test.",
      "_welcomePicture": "mainwelcome.jpg",
      "_welcomePictureFullScreen": false,
      "meshMessengerTitle": "MeshMessenger",
      "_meshMessengerPicture": "messenger.png",
      "___hide__": "Sum of: 1 = Hide header, 2 = Hide tab, 4 = Hide footer, 8 = Hide title, 16 = Hide left bar, 32 = Hide back buttons",
      "hide": 4,
      "footer": "<a href='https://homepage.domain.tld'>Homepage</a>",
      "loginfooter": "This is not a public service.",
      "allowSavingDeviceCredentials": true,
      "guestDeviceSharing": true,
      "_AutoRemoveInactiveDevices": 37,
      "_DeviceSearchBarServerAndClientName": false,
      "_agentSelfGuestSharing": {
        "expire": 120
      },
      "certUrl": "https://meshcentral.domain.tld:443",
      "deviceMeshRouterLinks": {
        "rdp": true,
        "ssh": true,
        "scp": true
      },
      "_PreconfiguredScripts": [
        {
          "name": "Run NotePad as user",
          "file": "scripts/notepad.bat",
          "type": "bat",
          "runas": "user"
        },
        {
          "name": "Run NotePad as agent",
          "cmd": "notepad.exe",
          "type": "bat",
          "runas": "agent"
        },
        {
          "name": "Run echo",
          "cmd": "echo \"hello world\"",
          "type": "sh",
          "runas": "agent"
        },
        {
          "name": "Agent Update",
          "cmd": "agentupdate",
          "type": "agent"
        }
      ],
      "myServer": {
        "Backup": false,
        "Restore": false,
        "Upgrade": false,
        "ErrorLog": false,
        "Console": false,
        "Trace": false
      },
      "passwordRequirements": {
        "min": 8,
        "max": 128,
        "upper": 1,
        "lower": 1,
        "numeric": 1,
        "nonalpha": 1,
        "reset": 3650,
        "force2factor": false,
        "skip2factor": "127.0.0.1",
        "oldPasswordBan": 5,
        "banCommonPasswords": false,
        "twoFactorTimeout": 30
      },
      "twoFactorCookieDurationDays": 180,
      "agentInviteCodes": false,
      "_agentNoProxy": true,
      "geoLocation": false,
      "novnc": true,
      "mstsc": true,
      "ssh": true,
      "_WebEmailsPath": "/myserver/email-templates",
      "consentMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} requesting remote desktop access. Grant access?",
        "terminal": "{0} requesting remote terminal access. Grant access?",
        "files": "{0} requesting remote files access. Grant access?",
        "consentTimeout": 60,
        "autoAcceptOnTimeout": false
      },
      "notificationMessages": {
        "title": "MeshCentral Remote Access",
        "desktop": "{0} started a remote desktop session.",
        "terminal": "{0} started a remote terminal session.",
        "files": "{0} started a remote files session."
      },
      "agentCustomization": {
        "displayName": "MeshCentral Agent",
        "description": "MeshCentral agent for remote monitoring, management and assistance.",
        "_companyName": "[redacted]",
        "serviceName": "MeshAgent",
        "_image": "agent-logo.png",
        "fileName": "MeshAgent"
      },
      "_agentFileInfo": {
        "_icon": "agent.ico",
        "_filedescription": "sample_filedescription",
        "fileversion": "1.0",
        "_internalname": "sample_internalname",
        "_legalcopyright": "sample_legalcopyright",
        "_originalfilename": "sample_originalfilename",
        "productname": "sample_productname",
        "productversion": "v0.1.2.3"
      },
      "assistantCustomization": {
        "title": "Mesh Assistant",
        "_image": "assistant-logo.png",
        "fileName": "MeshAssist"
      },
      "androidCustomization": {
        "title": "Mesh Android™",
        "_subtitle": "Product Subtitle™",
        "_image": "assistant-logo.png"
      },
      "_userAllowedIP": "127.0.0.1,192.168.1.0/24",
      "_userBlockedIP": "127.0.0.1,::1,192.168.0.100",
      "_agentAllowedIP": "192.168.0.100/24",
      "_agentBlockedIP": "127.0.0.1,::1",
      "___userSessionIdleTimeout__": "Number of user idle minutes before auto-disconnect",
      "_userSessionIdleTimeout": 60,
      "_userConsentFlags": {
        "desktopnotify": true,
        "terminalnotify": true,
        "filenotify": true,
        "desktopprompt": true,
        "terminalprompt": true,
        "fileprompt": true,
        "desktopprivacybar": true
      },
      "urlSwitching": true,
      "_desktopPrivacyBarText": "Privacy bar: {0}, {1}",
      "_limits": {
        "maxDevices": 3000,
        "maxUserAccounts": 100,
        "maxUserSessions": 100,
        "maxAgentSessions": 100,
        "maxSingleUserSessions": 10
      }
    }
  }
}

docker-compose.yml

version: '3'

networks:
  reverse_proxy:
    external: true
volumes:
  data:
  files:
  backups:

services:

    meshcentral:
        restart: always
        container_name: meshcentral
        image: typhonragewind/meshcentral:1.1.13
        ports:
            - 8786:443 #MeshCentral will moan and try everything not to use port 80, but you can also>
        environment:
            - TZ=Europe/Prague
        volumes:
            - /docker/containers/meshcentral/meshcentral-data:/opt/meshcentral/meshcentral-data
            - files:/opt/meshcentral/meshcentral-files
            - backups:/opt/meshcentral/meshcentral-backups
            - /docker/containers/meshcentral/meshcentral-web:/opt/meshcentral/meshcentral-web
            - /docker/containers/meshcentral/config.json:/opt/meshcentral/meshcentral-data/config.json       
        networks:
          - reverse_proxy

Nginx config for old server - that works

# ------------------------------------------------------------
# meshcentral.domain.tld
# ------------------------------------------------------------

server {
  set $forward_scheme http;
  set $server         "vm1.local.domain.tld";
  set $port           8786;

  listen 80;
listen [::]:80;

listen 443 ssl http2;
listen [::]:443 ssl http2;
  server_name meshcentral.domain.tld;
  # Let's Encrypt SSL
  include conf.d/include/letsencrypt-acme-challenge.conf;
  include conf.d/include/ssl-ciphers.conf;
  ssl_certificate /etc/letsencrypt/live/npm-2/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/npm-2/privkey.pem;
  # HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
    # Force SSL
    include conf.d/include/force-ssl.conf;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
  access_log /data/logs/proxy-host-3_access.log proxy;
  error_log /data/logs/proxy-host-3_error.log warn;

proxy_set_header CF-Connecting-IP $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

  location / {
  # HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
    proxy_http_version 1.1;
    # Proxy!
    include conf.d/include/proxy.conf;
  }
  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}

Nginx config for new server - that doesn't work

# ------------------------------------------------------------
# meshcentral.domain.tld
# ------------------------------------------------------------

server {
  set $forward_scheme http;
  set $server         "meshcentral";
  set $port           443;

  listen 80;
listen [::]:80;

listen 443 ssl http2;
listen [::]:443 ssl http2;
  server_name meshcentral.domain.tld;
  # Let's Encrypt SSL
  include conf.d/include/letsencrypt-acme-challenge.conf;
  include conf.d/include/ssl-ciphers.conf;
  ssl_certificate /etc/letsencrypt/live/npm-2/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/npm-2/privkey.pem;
  # HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
    # Force SSL
    include conf.d/include/force-ssl.conf;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
  access_log /data/logs/proxy-host-3_access.log proxy;
  error_log /data/logs/proxy-host-3_error.log warn;

proxy_set_header CF-Connecting-IP $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

  location / {
  # HSTS (ngx_http_headers_module is required) (63072000 seconds = 2 years)
  add_header Strict-Transport-Security "max-age=63072000;includeSubDomains; preload" always;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
    proxy_http_version 1.1;
    # Proxy!
    include conf.d/include/proxy.conf;
  }
  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}

si458 commented 9 months ago

This looks like a VM networking issue rather a meshcentral issue

If u bash into the new meshcentral container, can u ping meshcentral.domain.tld?

Can u curl https://meshcentral.domain.tld and get an output?

Meshcentral works by curling the url u give it for it to get the hash and use that

You could also check you have set the tlsoffload correct to the ip of your nginx?

Edit: I also don't think u should set :443 in the certurl and tlsoffload should be the ip of nginx container

tomsik-radek commented 9 months ago

This looks like a VM networking issue rather a meshcentral issue

If u bash into the new meshcentral container, can u ping meshcentral.domain.tld?

Can u curl https://meshcentral.domain.tld and get an output?

Meshcentral works by curling the url u give it for it to get the hash and use that

You could also check you have set the tlsoffload correct to the ip of your nginx?

It's not DNS It's not DNS It can't be DNS it was DNS

Okay, it seems like my local (meshcentral.domain.tld -> 10.0.1.202 (where NPM is)) DNS record doesn't work with the new VM. When the record is enabled curl fails on new server and succeeds on old server. The million dollar question is why that happens.

si458 commented 9 months ago

Sounds like a docker dns issue? does it return the wrong ip completely or does it return empty ip? Can u ping say Google.co.uk from inside the container ok?

tomsik-radek commented 9 months ago

Well that was embarrassing. I didn't allow 443 and 80 in my VM's firewall. I just kinda assumed that since Docker can bypass iptables (and NPM listens fine on 443/80 without ALLOW rules for those two ports), curl would work too. But for some reason it didn't. After ufw allow 443 and ufw allow 80 it works. Either way appreciate the help!

Ylianst / MeshCentral

Agent bad web cert hash - after migrating to a different VM #5570