search

jhuckaby / Cronicle

A simple, distributed task scheduler and runner with a web based UI.
http://cronicle.net
Other
3.95k stars 391 forks source link

I found a work around for websocket errors on newer builds with the "waiting for master server" but I'm not sure why? #417

Open jwjhdev opened 3 years ago

jwjhdev commented 3 years ago

Summary

Firstly, thank you community and @jhuckaby for this terrific tool! I have been using Cronicle for the past 2 years and it is excellent.

Unfortunately, however, with the new builds from the past year I have been having issues getting past the dreaded blank screen of death: "Waiting for master server...".

I have been reverting back to older revisions as a work around and told myself this weekend I would deep dive into why this happening.

I first started having this issue when installing newer versions (past 12 months) with Docker containers and this weekend I attempted to install out of the container environment and was still bumping into the problem on a test server.

Installing in a localhost environment without a reverse proxy or SSL certificates, this package installs without issue.

It isn't until you start navigating https and hosting on dedicated servers with domains that the issue presents itself.

Please keep in mind that it is likely it could be user error on my behalf! Hopefully with your help we can get the bottom of the problem, and hopefully help others with the same issue that might be viewing this ticket.

The problem

Websocket errors when resolving a reverse proxy subdomain pointing the Cronicle running on localhost in single server mode.

The error

Was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint

I am able to see the Cronicle web interface and a message "Waiting for master server..." which disappears after 30 seconds and doesn't proceed.

The assumption of the problem

Somewhere there is a http cross websocket attempt which is being rejected as force https is enabled. This is ultimately what I would love to find out.

I was able to solve my issue by using polling instead

I was able to solve my issue by adding a line from the configuration file on a on version 0.8.24:

"socket_io_transports": ["polling", "websocket"],

This should now force Cronicle to use polling as a preference and disregard using websockets.

The socket_io_transports configuration setting is missing in the newer revision configuration files, so I didn't immediately realise this option was available.

Polling is quite archaic, so I would certainly prefer to use websocks. Polling in general is more resource hungry, has issues with ordering of requests and websockets is built on the tcp stack so it has those inbuilt security mechanisms as well.

I worked around this by allocating more memory for Cronicle and setting up a .htpasswd for the entire subdomain it is hosted on for security.

My question

How would I go about hunting down where the rogue http sock request is coming from? Is it perhaps my configuration that is the problem? Welcome any advice :)

My configuration

Here are the custom entries in the config file (the rest being default):

config.json

"base_app_url": "https://subdomain.example.com"
...
"https": true,
"https_port": 4000,
"https_cert_file": "/etc/letsencrypt/live/subdomain.example.com/fullchain.pem",
"https_key_file": " /etc/letsencrypt/live/subdomain.example.com/privkey.pem",
"https_force": true,

Added a new line:

"socket_io_transports": ["polling", "websocket"],

Here is my server block for Nginx subdomain:

subdomain.example.com

server {
        root /var/www/subdomain.example.com/html;
        index index.html index.htm index.nginx-debian.html;

        server_name subdomain.example.com;

        location / {
                proxy_pass https://localhost:4000;
        }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/subdomain.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/subdomain.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

server {
    if ($host = subdomain.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen 80;
        listen [::]:80;

        server_name subdomain.example.com;
        return 404; # managed by Certbot
}

Thanks for reading, sorry about the wall-of-text!

Steps to reproduce the problem

  1. Install the latest stable version of Cronicle
  2. Edit the configuration file with the above settings
  3. Run setup (single server): $ /opt/cronicle/bin/control.sh setup
  4. Start the service: /opt/cronicle/bin/control.sh start

Your Setup

Operating system and version?

Node.js version?

Cronicle software version?

Are you using a multi-server setup, or just a single server?

Are you using the filesystem as back-end storage, or S3/Couchbase?

Can you reproduce the crash consistently?

Log Excerpts

Note: I have substituted my domain with subdomain.example.com

universalModuleDefinition:3 WebSocket connection to 'wss://subdomain.example.com/socket.io/?EIO=3&transport=websocket&sid=Rggi9sYGV25F9ODZAAAC' failed:

was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint

mikeTWC1984 commented 3 years ago

Strange. Actually "Waiting for master" means that your node doesn't have access to storage or hostname doesnt match regex for master group. Check the logs when you start cronicle (first 50 lines or so). On a single node the proxy would only cause issues with loading UI.

Regarding reverse https proxy - it should work, I used it with nginx (as rout) and now using with traefik (as subdomain). The only thing that would not work is livelog, because of downgrading websocket from https to http is blocked

stubclan commented 3 years ago

@jwjhdev Figured out a reason/ solution yet? Running into the exact same error.

image

Behind a nginx reverse proxy and tried setting up Websocket but still getting same error

  location ~* /(terminals/websocket)/? {

    proxy_pass http://127.0.0.1:3012;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # WebSocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade "websocket";
    proxy_set_header Connection "Upgrade";
    proxy_read_timeout 86400;
  }
jwjhdev commented 3 years ago

@stubclan my solution was to use 'polling' instead of websockets.

I haven't been able to get websockets to work on a reverse proxy using https.

mikeTWC1984 commented 3 years ago

Below is the ngnix config I've used to use for reverse https (to rout localhost:3012 to https://myhost.com and https/myhost.com/cronicle). The only trick was to handle route for socket.io. It worked without any extra cronicle config tweaking.

https://github.com/cronicle-edge/cronicle-edge/blob/main/Docker/nginx.conf

I'm currently using traefik, as subdomain https proxy, and have no any issues (well besides livelog). I haven't set it up myself though. So there is no some "by design" issue, keep tweaking your configs

stubclan commented 3 years ago

@mikeTWC1984 THANK YOU very much. That worked like a charm.

Now I'm getting websocket error to print logs. I'm trying to find solution but please let me know if you have seen this before and found a solve: image

mikeTWC1984 commented 3 years ago

Yes, there will be problems with livelog in this case. Check that issue: https://github.com/jhuckaby/Cronicle/issues/426