Using Meshcentral behind Traefik reverse proxy

routerino commented 5 years ago

Hi, I'll preface this with saying that I haven't determined if this is a Traefik issue or a Meshcentral issue. I've requested a response on the Traefik slack as well, but no response from them as of yet.

Traefik is a docker and cluster aware reverse proxy, with the ability to dynamically create and monitor let's encrypt certificates on container load without needing to change the config files.

It appears that Traefik forwards traffic to the MeshCentral web interface just fine, but when agents connect via websockets it does not get forwarded. Traefik provides the following error code whenever an agent tries connecting: 2019-02-10T00:50:54Z" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: websocket: incorrect mask flag

Traefik does support websockets from other products, does this indicate that MeshCentral is providing an incorrect flag when initiating a websockets connection?

krayon007 commented 5 years ago

As an optimization, I think MeshCentral doesn't mask websockets when it's carried over TLS. I wonder if that's causing an issue with your reverse proxy. In the morning I can force it to always mask websockets to see if that resolves your issue.

routerino commented 5 years ago

It sounds like doing so will resolve the problem, I'll give it a test when you update and see how it goes.

Ylianst commented 5 years ago

It would be nice if Traefik supported web sockets that do not use masks. Marks are useful with HTTP, but with HTTPS I don't see any upside and it just cuts down on a little on server performance. This said, I guess we would set a mask of all zero bits, that should get around the issue.

routerino commented 5 years ago

Hi Ylianst. Would you be willing to submit an issue on traefik's github? If not I'll be happy to do it, however my knowledge of websockets is quite basic and I might not correctly describe the issue.

Ylianst commented 5 years ago

I can do it. Doing it now, hold on...

Ylianst commented 5 years ago

Done: https://github.com/containous/traefik/issues/4487

Ylianst commented 5 years ago

Progress report on this one. Bryan (krayon007) added support in the latest agent to perform full websocket masking, but it's not enabled by default. I will be adding a server-side flag to enable this. Once enabled, when you download the agent, the server will give you an agent that has full masking enabled. So, I will report back when it's done.

Ylianst commented 5 years ago

Just published MeshCentral v0.2.8-v with support for adding extra configuration parameters that will be inserted into the MeshAgent and .msh file when someone downloads it. So you can now do this:

{
  "settings": {
    "port": 443
  },
  "domains": {
    "": {
      "Title": "MyServer",
      "AgentConfig": [ "webSocketMaskOverride=1" ]
    }
  }
}

Note that the string "webSocketMaskOverride=1" is case sensitive, so type it exactly. You need to add "AgentConfig" to the domain you want it to take effect (often the default "" domain). After that, reset the server and each agent downloaded with have TLS masking enabled.

Hopefully that will fix the Traefik problem. Hope it helps - Ylian

routerino commented 5 years ago

Hi Yilan, Thanks for the update. Good News: I can now route through traefik, and the agent gets detected properly. Hooray!

Bad news: I don't seem to actually gain control. Remote desktop/remote console do not appear as options, and commands sent to the remote computer (wake up, shutdown, reset) do not seem to respond. It correctly detects when the computer is connected, but that's about as far as it goes. Neither the traefik or meshcentral console (via docker) are saying there are issues.

That being said, this may just be a configuration issue on my end. I'll see if I can set up a similar config using nginx later and see if it has the same issues with masking on.

ghost commented 5 years ago

@Ylianst I think your issue on the Traefik GitHub has gone unanswered because of the missing issue template. I just created a new issue that follows their template: https://github.com/containous/traefik/issues/4513

Ylianst commented 5 years ago

Apologies for the delay, I will get on this in the next few days. We just have to give you a sample web socket client that does not do masking.

Ylianst commented 5 years ago

The issue is now discussed at: https://github.com/containous/traefik/issues/4487. The "webSocketMaskOverride=1" workaround does fix it, however, once the agent is installed without masking, there is no way for IT to switch to using Traefik without reinstalling all agents.

JMyrng commented 5 years ago

Got the same problem as @routerino . Clients are showing up but Remote Desktop does not appear. When installing a new client I got the following error in my Traefik logs:

vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF

So I think it's not an configuration problem.

Ylianst commented 5 years ago

If you are using Traefik, did you put this line in your config.json domain section?

"AgentConfig": [ "webSocketMaskOverride=1" ]

Also, you may want to try again with the latest server, I made a fix there user notification on Linux and MacOS would cause a black screen.

JMyrng commented 5 years ago

Yes, at first I got no connetction at all between my server and clients. After putting this line in my config.json there is a connection, but console, remote desktop etc. aren´t showing up.

routerino commented 5 years ago

Yes, at first I got no connetction at all between my server and clients. After putting this line in my config.json there is a connection, but console, remote desktop etc. aren´t showing up.

Can confirm, I am still experiencing the same issue (issue didn't go away, just had to put meshcentral testing on hiatus for a while). Config.json can be found here:

https://pastebin.com/FSC7G8Mh

Error within traefik is as follows: time="2019-05-13T01:21:46Z" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF"

How it is viewed in the interface: https://imgur.com/YLC1Rnz

Note how the remote viewing tab doesn't exist at all.

Ylianst commented 5 years ago

Arg. This is not good. This may not be related to the web socket masking at all and would explain why the config.json line has no effect. The reported error does not look like the masking problem. I will need to install Traefik and try it myself.

routerino commented 5 years ago

Arg. This is not good. This may not be related to the web socket masking at all and would explain why the config.json line has no effect. The reported error does not look like the masking problem. I will need to install Traefik and try it myself.

Hi ylianst.

You'll have to set up for your own instance, but you can find my docker-compose for traefik here: https://pastebin.com/YSmyRYAU

Here is a corresponding toml config file for traefik https://pastebin.com/v3HH3SDe

Finally, docker-compose for meshconnect that I'm using (you already have the config file) https://pastebin.com/j0bHLC3B

depending on what distro you are running docker on, cockpit with cockpit-docker is great for checking errors in docker-containers and current status. For ubuntu server, you install the whole stack with "sudo apt-get install cockpit cockpit-docker docker docker-compose"

JMyrng commented 5 years ago

I can put my configs online as well if you like. I´m checking the logs of the containers with Portainer.

Ylianst commented 5 years ago

What version of Traefik are you using?

I am not familiar with Traefik at all... I just downloaded the Windows version and trying to make a small config file that routes port localhost:444 to localhost:443. So far, no luck. I have no idea what configuration is ok with that version of Traefik. If I could make this work, I could easily add this to my regular test runs. Do you know what I am doing wrong below? I am trying on both 1.7 and 2.0.0-alpha4.

When I hit "https://localhost:444" I currently see "404 page not found".

[entryPoints]
  [entryPoints.http]
  address = ":81"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":444"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "webserver-cert-public.crt"
      keyFile = "webserver-cert-private.key"

[frontends]
  [frontends.frontend1]
  backend = "backend1"
    [frontends.frontend1.routes.test_1]
    rule = "Host:localhost"

[backends]
  [backends.backend1]
    [backends.backend1.servers.server1]
    url = "http://127.0.0.1:443"
    weight = 1

Note: I played around with "healtcheck" and added "/health" in the next version of MeshCentral that returns "200 OK". Traefik seems to be polling this well.

Ylianst commented 5 years ago

Nevermind, I got it working!! I can now start testing...

[entryPoints]
  [entryPoints.http]
  address = ":81"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":444"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "webserver-cert-public.crt"
      keyFile = "webserver-cert-private.key"

[file]

[backends]
  [backends.backend1]
    [backends.backend1.healthcheck]
      path = "/health"
      interval = "30s"

    [backends.backend1.servers.server1]
    url = "http://127.0.0.1:443"
    weight = 1

[frontends]
  [frontends.frontend1]
  entryPoints = ["https"]
  backend = "backend1"
  passHostHeader = true
  [frontends.frontend1.routes]
    [frontends.frontend1.routes.main]
    rule = "Host:devbox.mesh.meshcentral.com"
    #rule = "Path:/"

[api]
  entryPoint = "traefik"
  dashboard = true
  #address = "localhost:8089"

Ylianst commented 5 years ago

So, I did some work with Traefik 1.7 and it worked perfectly for me. I also wrote a new section in the MeshCentral User's Guide 0.2.3 on how to setup Traefik.

I now realize that if you setup a MeshAgent and see "websocket: close 1006 (abnormal closure)". It's probably because Traefik is presenting a TLS certificate that MeshCentral is not aware of and so, MeshCentral and the MeshAgent think there is a man-in-the-middle attack.

You can debug this by running "node ./node_modules/meshcentral --debug". You can fix this by adding this to the domain section of config.json:

"certUrl": "1.2.3.4"

There 1.2.3.4 is the IP address where MeshCentral can do a HTTPS connection and load the TLS certificate that is presented externally to the agents that are connecting. You can also do add the following in the settings section:

"ignoreagenthashcheck": true

This is not recommended, but will tell the server not to check for TLS man-in-the-middle attacks. This is a good way to see if this is the problem since everything will work immidiatly. Hope it helps.

MC2-Traefik-Docs

routerino commented 5 years ago

Hi again,

I agree, the issues I've been having related to traefik were cert related. I've since rebuilt again from scratch and the EOF/socket mask issues have gone away. It's safe to say that traefik isn't the cause of the below issue.

I apologise about misdiagnosing the problem, though it's good that it's involved in traefik being investigated as a proxy.

That being said, I'm still encountering this issue of no remote desktop tab. Something very interesting about this fault though: If I go into the console, and upload the recovery core, I gain the "files" and "terminal" tabs. These appear to be functioning. Still no remote desktop.

So maybe this is an independent issue. It appears to be agent related. Tried on both windows 1809 and windows 1803.

Ylianst commented 5 years ago

I added in "dupagents" command in the server console a few days back. You go in "My Server" tab, go in "Console" sub-tab and type "dupagents". It gives you a list of what agents cause the duplicates and from what IP address. I am noticing a lot of these "duplicates" on my server and working on this now. To my surprise, it's all over the place but I got good data for investigating this now.

routerino commented 5 years ago

As an update, I've since set up a new instance completely separate from docker, inside of a virtual machine. This way I could access the instance directly via LAN, or via WAN in traefik.

From my testing, the lack of a remote desktop/files/terminal tab still appears to be a traefik related problem. It may be indirectly related to the socket mask filtering setting (since you get no connectivity at all via traefik without it).

However, LAN connectivity works flawlessly.

I've dug around some open issues in traefik related to web sockets, and it appears there may be further issues with their implementation. For now, I'll probably switch to nginx or haproxy (which is unfortunate because traefik's docker integration makes it very useful).

routerino commented 5 years ago

Interesting update to this problem. I've been using meshcentral directly for a while (though in LAN only since I can't provision its own IP for it yet). I updated meshcentral and traefik and put it back in the old configuration, just in case things have improved.

All of the existing agents can connect just fine, even through the proxy (with the certurl set up on meshcentral). Any new agents I try to add have the issue described above (works without remote desktop in recovery core mode, does not respond in normal core mode).

So, what's special about the handshaking that only affects newly added clients?

EDIT: Mask socket problem still needs an overrride, but the fact about the existing clients working is certainly odd.

routerino commented 5 years ago

I've created a short video of the problem. I created a temporary client, and connected it directly, no proxy on. Works fine. I then set up the proxy, changed DNS, reconnected the agent (this is where the video starts). Again, no problem. I can disconnect and reconnect fine. I then (in the video) deleted the db file and the mesh file on the client side, and attempted to reconnect. I then get the above described problem. It seems that if the DB file is already there, I'm not having an issue. If the DB file needs creating, there's something missing out of the DB file that's stopping all the action.

I can also confirm identical results on ubuntu linux.

Video in Question

routerino commented 4 years ago

Hi @Ylianst ,

I suspect this is not going to be solved easily (or ever), but I have found a workaround.

What I have done is exposed meshcentral directly on a different port, and set up the whitelist within the settings to only allow meshcentral agents through the exposed port. I then whitelist the traefik server IP. This allows users to log in through traefik (and the benefits that provides), while allowing the agents to connect to the server directly, bypassing any problems with reverse proxies completely.

The end result is the agents connect to www.contosso.com:1234 fine, while users will get blocked if accessing on that port. However, users can connect via reverse proxy on mesh.cotosso.com.

MailYouLater commented 4 years ago

@routerino: That's a good workaround, but really is a workaround. Perhaps you could compare the two meshagent.db files to see what the difference between them is. @Ylianst / @krayon007: So that @routerino can compare the files properly, what format is the meshagent.db file in?

uniuuu commented 3 years ago

Got the same problem as @routerino . Clients are showing up but Remote Desktop does not appear. When installing a new client I got the following error in my Traefik logs:

vulcand/oxy/forward/websocket: Error when copying from backend to client: websocket: close 1006 (abnormal closure): unexpected EOF

So I think it's not an configuration problem.

Hi @Ylianst

I confirm it's reproducible.

I use chain traefik < - > nginx < - > meshcentral

traefik and nginx in docker swarm meshcentral in docker via docker-compose

My docker-compose for meshcentral: I use next repository https://github.com/techno-express/MeshCentral.git

version: '3.9'
services:
  meshcentral:
    build:
      context: .
      dockerfile: Dockerfile
    extra_hosts:
      - "mywebsite.org:127.0.0.1"
    hostname: mywebsite.org
    container_name: meshcentral
    working_dir: /home/meshserver
    ports:
    - "4430:4430"
    - "27017:27017"
    - "27018:27018"
    restart: always
    environment:
       EMAIL: info@somemail.org
       HOST: mywebsite.org
       SMTP: mail.somemail.org
       USER: info@somemail.org
       PASS: somepassword
    volumes:
     - meshcentral_data:/home/meshserver
     - "/etc/localtime:/etc/localtime:ro"
volumes:
  meshcentral_data:
            driver: local

meshcentral config.json

{
    "settings": {
        "Cert": "mywebsite.org",
        "Port": 4430,
        "AliasPort":443,
        "RedirPort": 800,
        "_MpsPort": 4443,
        "TlsOffload": "10.21.9.5",
        "AllowLoginToken": true,
        "AllowFraming": true,
        "WebRTC": true,
        "WANonly": true,
        "ignoreagenthashcheck": true
        },
    "domains": {
    "": {
      "certUrl": "https://10.21.9.5",
      "agentConfig": [ "webSocketMaskOverride=1"]
        }
        },
    "smtp": {
        "host": "mail.somemail.org",
        "port": 25,
        "from": "info@somemail.org",
        "user": "info@somemail.org",
        "pass": "somepassword",
        "tls": false
        }
}

My nginx proxy yaml

version: "3.9"
services:
  nginx:
    image: nginx:latest
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.frontend.rule=Host:mywebsite.org
        - traefik.docker.network=traefik_public
        - traefik.port=80
        - traefik.frontend.passHostHeader=true
      placement:
        constraints:
           - node.hostname == host002
    volumes:
      - /var/swarm/meshcentral/meshcentral.conf:/etc/nginx/conf.d/default.conf:ro
      - "/etc/localtime:/etc/localtime:ro"
networks:
  traefik_public:
    external: true

Nginx conf

upstream meshcentral-upstream {
    server 10.21.9.5:4430;
}
server {
    listen 80;
    server_name remotesupport.techfount.com;
    client_max_body_size 25M;
    location / {
        proxy_pass http://meshcentral-upstream;
        proxy_set_header Host $http_host;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_send_timeout 999999s;
        proxy_read_timeout 999999s;

# Allows websockets over HTTPS.
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Traefik two stacks

version: "3.9"
# What is this?
# This stack exists solely to deploy the traefik_public overlay network, so that
# other stacks (including traefik-app) can attach to it
services:
  scratch:
    image: scratch
    deploy:
      replicas: 0
    networks:
      - public
networks:
  public:
    driver: overlay
    attachable: true
    ipam:
      config:
        - subnet: 172.16.200.0/24

version: "3.9"
services:
  traefik:
    image: traefik:v1.7.1
    command: --web --docker --docker.swarmmode --docker.watch --docker.domain=host002 --logLevel=DEBUG --acme.keytype=EC256
    # Note below that we use host mode to avoid source nat being applied to our ingress HTTP/HTTPS sessions
    # Without host mode, all inbound sessions would have the source IP of the swarm nodes, rather than the
    # original source IP, which would impact logging. If you don't care about this, you can expose ports the
    # "minimal" way instead
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8080
        published: 8080
        protocol: tcp
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /swarm/traefik/traefikv1:/etc/traefik
      - /swarm/traefik/traefikv1/traefik.log:/traefik.log
      - /swarm/traefik/traefikv1/acme.json:/acme.json
      - "/etc/localtime:/etc/localtime:ro"
    networks:
      - traefik_public
    # Global mode makes an instance of traefik listen on _every_ node, so that regardless of which
    # node the request arrives on, it'll be forwarded to the correct backend service.
    deploy:
      labels:
        - "traefik.enable=false"
      mode: global
      placement:
        constraints: [node.role == manager]
      restart_policy:
        condition: on-failure
networks:
  traefik_public:
    external: true

Traefik toml

checkNewVersion = true
defaultEntryPoints = ["http", "https"]
# This section enable LetsEncrypt automatic certificate generation / renewal
[acme]
email = "adm@somemail.org"
storage = "acme.json" # or "traefik/acme/account" if using KV store
entryPoint = "https"
acmeLogging = true
onDemand = true
OnHostRule = true
  [acme.httpChallenge]
  entryPoint = "http"

# Redirect all HTTP to HTTPS (why wouldn't you?)
[entryPoints]
  [entryPoints.http]
    address = ":80"
     [entryPoints.http.redirect]
       entryPoint = "https"
  [entryPoints.https]
    address = ":443"
     [entryPoints.https.tls]
  [entryPoints.dashboard]
    address = ":8080"
    [entryPoints.dashboard.auth]
      [entryPoints.dashboard.auth.basic]
        users = ["admin:$apr1$SqL73kcedrgergerecyNlUETHXeeF1"]
[api]
entrypoint="dashboard"

[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "host002"
watch = true
swarmmode = true

Starting meshcentral container I can notice that it loded certificate from Traefik:

meshcentral    | Loaded web certificate from "https://10.21.9.5", host: "mywebsite.org"
meshcentral    |   SHA384 cert hash: ffd4e947f9fc8900e9e34b2b2f2ec62a4781bbb285ef726b80fa9acafef67f2ed233ef0ac7ed5f8d7ed5605db4de3d0b

Then some intermittent issue appearing when first time loading mywebsite.org page in webbrowser and it shows empty window

Traefik throws error at the same time I believe connected to this.

time="2021-06-06T16:48:56+08:00" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"HEAD\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"viewmode=10\\u0026gotonode=@xTtHjrET0jnl0QlD1tdcoWBtPJMHql8Dq5zsZr19mebvloJ2vsUIvnUpdJuAeyP\",\"Fragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-GB,en;q=0.5\"],\"Cookie\":[\"_ga=GA1.2.1702699465.1613630621; _fbp=fb.1.1613630621494.150199519; xid=eyJ1c2VyaWQiOiJ1c2VyLy9hZG1pbmlzdHJhdG9yIiwiaXAiOiIxOTIuMTY4LjE2LjEiLCJ0IjoyNzA0OTQ2NH0=; xid.sig=uU6CpiBOXQa7NlDrbggWerYw2XI\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"remotesupport.techfount.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"58.172.68.170:25827\",\"RequestURI\":\"/?viewmode=10\\u0026gotonode=@xTtHjrET0jnl0QlD1tdcoWBtPJMHql8Dq5zsZr19mebvloJ2vsUIvnUpdJuAeyP\",\"TLS\":null}" ForwardURL="http://172.16.200.88:80"

time="2021-06-06T16:49:39+08:00" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF"

time="2021-06-06T16:49:39+08:00" level=debug msg="vulcand/oxy/forward/websocket: completed ServeHttp on request"
Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.16.200.88:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Key\":[\"HKrXVE6ndmw95qHoWAo300==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"remotesupport.techfount.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"58.182.68.120:25833\",\"RequestURI\":\"/agent.ashx\",\"TLS\":null}"
time="2021-06-06T16:49:39+08:00" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.16.200.88:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Connection\":[\"Upgrade\"],\"Sec-Websocket-Key\":[\"HKrXVE6ndmw95qHoWAo300==\"],\"Sec-Websocket-Version\":[\"13\"],\"Upgrade\":[\"websocket\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"remotesupport.techfount.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"58.182.68.120:25833\",\"RequestURI\":\"/agent.ashx\",\"TLS\":null}"

After clicking on My devices the frame refreshes and all details are appearing:

Going further when downloading agent one more error is encountered.

meshcentral    | COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
meshcentral    | COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

I beleive it's separate issue and not connected to the subject.

When click on device there is no tab Desktop

No more errors generated by traefik or meshcentral

Please help how to debug it ?

Update: The error also is being generated every time disconnecting agent: time="2021-06-06T17:28:19+08:00" level=error msg="vulcand/oxy/forward/websocket: Error when copying from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF"

Command info in console retunrs nothing.

PrplHaz4 commented 3 years ago

Has anyone made progress on this one yet? I've spent some time digging in over the last few days and have come to similar conclusions as @inzi from #3047 and #106.

Symptoms

Agent appears connected
"Desktop" menu is not available
No console or network errors on agent or on server
Agent console commands do not get any responses with the "default" core
Upload Tiny Core works successfully
Tiny Core console commands are successful (versions, eval, help...etc)
Upload Default and Recovery Core does not throw any errors, but db file sizes don't change and once selected it's back to not responding to any commands

Possible Cause(s)?

Either the size of the core or something in it is causing the websocket transfer to stop without error
Per this open issue Traefik handles websockets according to RFC, but some Web Socket servers do not (appears that ws and express-ws versions have bounced around a few times in this repo - so maybe there is a magic combo?)

Workarounds

Bypass proxy for initial load of Default Agent Core
@Ylianst Is there a way to force upload of a core without going through the server/websocket?

PrplHaz4 commented 2 years ago

* @Ylianst Is there a way to force upload of a core without going through the server/websocket?

To answer this question - you can overwrite the agent core in MeshAgent.db using dbTool.js

Download meschore.js and rename to CoreModule.js and put it alongside MeshAgent.exe
Download dbTool.js (from modules dir) and put it alongside MeshAgent.exe
Stop MeshAgent service
Run $> MeshAgent.exe dbTool.js import CoreModule
Restart MeshAgent service

More troubleshooting on the Traefik issue:

Setting "AgentUpdateBlockSize": 2048 causes even TinyCore update/upload to fail with no new errors
Removing AgentUpdateBlockSize returns to the previous behavior or TinyCore loading successfully, but the Default core will still not load
This points to Traefik having a problem with whatever chunking mechanism is being used by the agent updater (wherever that is...) - anything more than one "block" seems to fail

Log Output on failed Core update:

[2021-11-26 06:42:19 PM] [80645B8D669A2A74] ..\microstack\ILibParsers.c:10746 (0,0) ProcessCommand(11)...
[2021-11-26 06:42:19 PM] [80645B8D669A2A74] ..\microstack\ILibParsers.c:10746 (0,0) BinaryCommand(11, 0)...
[2021-11-26 06:42:52 PM] [80645B8D669A2A74] ..\microstack\ILibParsers.c:10746 (0,0) ProcessCommand(31522)...
[2021-11-26 06:44:52 PM] [80645B8D669A2A74] ..\microstack\ILibParsers.c:10746 (0,0) AgentCore/MeshServer_ControlChannel_IdleTimeout(): Sending Ping
[2021-11-26 06:44:52 PM] [80645B8D669A2A74] ..\microstack\ILibParsers.c:10746 (0,0) AgentCore/MeshServer_ControlChannel_IdleTimeout(): Pong Received

Authentication Complete...
ProcessCommand(11)...
BinaryCommand(11, 0)...
Timeout waiting for Server, launching cached meshcore...

Pretty much at the end of my skills - I'm not sure how to debug into the actual update method. Maybe I can get more out of the Traefik logs - the most curious thing to me is that at some point @Ylianst had traefik working without issues - any idea if that is still the case with the current meshcentral code?

Edit: Added MeshAgent startup log that includes Timeout waiting for Server, launching cached meshcore... - this may point to it having nothing to do with the block/chunking method...

Ylianst / MeshCentral

Using Meshcentral behind Traefik reverse proxy #89