search

ElasticHQ / elasticsearch-HQ

Monitoring and Management Web Application for ElasticSearch instances and clusters.
http://www.elastichq.org
Other
4.96k stars 532 forks source link

JS error on Metrics UI with Nginx reverse proxy setup #455

Open mocobeta opened 5 years ago

mocobeta commented 5 years ago

General information

Issue Description

We are trying to set up a Nginx reverse proxy for ElasticHQ. Nginx and ElasticHQ is deployed on an Amazon EC2 instance and the Elasticsearch cluster to be monitored is deployed separate EC2 instances. HTTP accesses to ElasticHQ are restricted by Basic authentication.

Our nginx configuration is here:

server {
    ...
    location /elastichq/ {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;

        rewrite ^/elastichq/(.*) /$1 break;
        proxy_pass http://localhost:5000;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  Host $http_host;
    }

    location /socket.io/ {
        #rewrite ^/socket.io/(.*) /$1 break;
        proxy_pass http://localhost:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        error_page  405     =200 $uri;
    }
    ...
}

ElasticHQ works fine and we can connect it through the nginx proxy, except for Metrics UI. When we access to the Metrics UI, the graphs are not shown and an error message is continually printed in the Chrome's developer tools console.

Error: <path> attribute d: Expected number, "MNaN,190 NaN,0".

The Metrics UI works fine when I run ElasticHQ on the local machine, so I think this is caused by nginx or network setup. I found a similar issue #340, but the problem discussed in the issue seems slightly different with ours.

Are there any workarounds or solutions for this? I welcome any advice from users having similar setups to ours.

Thanks again for maintaining this excellent tool!

djlambert commented 5 years ago

I'm having the same issue. It looks like the websocket connection is upgrading okay, but no frames are sent.

djlambert commented 5 years ago

I think this being caused by the lack of data. Looks like it's trying to get the data point under the mouse cursor, but there's no data there. After clicking the Metrics button the errors don't start logging until the cursor is moved over a graph.

djlambert commented 5 years ago

The problem arises when the clusterNodesController instantiates the SocketIO object. It's including any extra path components in the URL, and SocketIO is including this in the message data. Presumably the UI isn't expecting this extra path component, and discarding the data. Changing the argument value to the origin got the data flowing. Not an elegant solution, but now I understand what's happening.

diff -Naur elasticsearch-HQ.orig/ui/src/containers/cluster-nodes/cluster-nodes.controller.js elasticsearch-HQ/ui/src/containers/cluster-nodes/cluster-nodes.controller.js
--- elasticsearch-HQ.orig/ui/src/containers/cluster-nodes/cluster-nodes.controller.js   2019-02-01 14:56:49.000000000 -0600
+++ elasticsearch-HQ/ui/src/containers/cluster-nodes/cluster-nodes.controller.js    2019-02-01 14:49:43.000000000 -0600
@@ -27,7 +27,7 @@
         baseUrl = baseUrl += `/ws`;
         // console.log('---- baseUrl: ', baseUrl)

-        this.socket = io(baseUrl);
+        this.socket = io(location.origin + '/ws');
         this.socket.on('connect', () => {
             this.connected = true;
             this.socket.emit('join', {"room_name": this.clusterName + "::nodes"});
royrusso commented 5 years ago

Interesting. This may be tricky to add as a pull request, as it will affect non-proxy users. I can talk to @pcasa and see if he can add a context switch in case we get blank data. We will have to set things up locally with nginx as a reverse proxy. @djlambert do you have a sample nginx config, we can use to test with?

djlambert commented 5 years ago

@royrusso I had a feeling that might be the case, but had to throw in the towel for the day. Was starting to have flashbacks of battles with SocketIO and Flask-SocketIO in my own projects.

I think this is the relevant nginx.conf:

http {
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    server {
        location /elastichq {
            proxy_read_timeout 90;
            proxy_http_version 1.1;

            proxy_set_header Connection "Keep-Alive";
            proxy_set_header Proxy-Connection "Keep-Alive";
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Script-Name /elastichq;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

            proxy_redirect   off;
            proxy_buffering  off;
            proxy_pass       http://elastichq:5000;
        }
    }
}

I'm also wrapping the WSGI application to handle the URI instead of using nginx to rewrite, to avoid this exact issue :) I didn't consider WebSocket though, or SocketIO anyways. It might work okay with native HTML5 WebSocket.

from flask import Flask
from application import application

class ReverseProxied(object):
    def __init__(self, app: Flask):
        self.app = app

    def __call__(self, environ, start_response):
        script_name = environ.get('HTTP_X_SCRIPT_NAME', '')

        if script_name:
            environ['SCRIPT_NAME'] = script_name
            path_info              = environ['PATH_INFO']

            if path_info.startswith(script_name):
                environ['PATH_INFO'] = path_info[len(script_name):]

        return self.app(environ, start_response)

app = ReverseProxied(application)

In my use case I'm creating a containerized management server, with each application in its own path (i.e http://server/elastichq, http://server/pgadmin, etc.). nginx is proxying to the applications running under gunicorn.

pcasa commented 5 years ago

@djlambert can you post what the output is for window.location object that is logged to the console? Need to know what the output is for href & origin.

djlambert commented 5 years ago 
---- window.location Location {replace: ƒ, assign: ƒ, href: "https://192.168.42.168/elastichq/#!/clusters/mail-services", ancestorOrigins: DOMStringList, origin: "https://192.168.42.168", …}
        ancestorOrigins: DOMStringList {length: 0}
        assign: ƒ ()
        hash: "#!/clusters/mail-services/nodes"
        host: "192.168.42.168"
        hostname: "192.168.42.168"
        href: "https://192.168.42.168/elastichq/#!/clusters/mail-services/nodes"
        origin: "https://192.168.42.168"
        pathname: "/elastichq/"
        port: ""
        protocol: "https:"
        reload: ƒ reload()
        replace: ƒ ()
        search: ""
        toString: ƒ toString()
        valueOf: ƒ valueOf()
        Symbol(Symbol.toPrimitive): undefined
        __proto__: Location
pcasa commented 5 years ago

Believe this is a configuration issue due to proxy, but I'm no NGINX / APACHE expert.

What happens if you add the following to your nginx.conf above the location /elastichq { line? NOTE: you might have to undo your ReverseProxied in the flask app.

location /elastichq/ws {

    proxy_pass       http://elastichq:5000/ws;
    proxy_http_version 1.1;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Script-Name /elastichq;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_redirect default;

}
djlambert commented 5 years ago

If I add that to the nginx config and remove the patch and the ReversedProxied, I'm not able to bring up elastichq at all. The ui tries to load the bundles from the site root.

GET https://192.168.253.40/static/app.bundle.css?110e62edca51cd300982 net::ERR_ABORTED 404 (Not Found)
GET https://192.168.253.40/static/commons.110e62edca51cd300982.js?110e62edca51cd300982 net::ERR_ABORTED 404 (Not Found)

If I add the ReversedProxied back I am able to load the ui, but the SocketIO library attempts to connect to https://192.168.253.40/socket.io/?EIO=3&transport=polling&t=MZ3-nvT.

If I change the nginx config to proxy `/socket.io'

        location /socket.io {
            proxy_pass       http://elastichq:5000/socket.io;
            proxy_http_version 1.1;

            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;

            proxy_redirect default;
        }

Then socket.io does make the connection and upgrade to websocket, but no data is displayed.

engineio logs the websocket messages:

engineio socket.receive:52  e344e10938b04235a41244a119b2405e: Received packet MESSAGE data 0/elastichq/ws,
engineio socket.send:91     e344e10938b04235a41244a119b2405e: Sending packet MESSAGE data 0/elastichq/ws

I think the problem is the message data is being send with 0/elastichq/ws, but the ui is looking for 0/ws

pcasa commented 5 years ago

@djlambert sent you an email.

can you replace this.socket = io(location.origin + '/ws'); with

const path = window.location.pathname === "/" ? "/socket.io" : `${window.location.pathname}/socket.io`
this.socket = io(baseUrl, { path: path });

And report the new errors?

p0wertiger commented 5 years ago

Hello

I just wanted to say I am currently experiencing the same problem with "/elastichq" prefix. I followed your discussion, config looks the same (except for nginx rewrite in mine), checked out feature/455 but it didn't change anything, metrics not showing. The only meaningful console error I can see is numerous Error: <path> attribute d: Expected number, "MNaN,190 NaN,0".

My application.log says:

2019-03-22 17:46:59,600          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet OPEN data {'sid': 'e44e4b27172e4c468d09b0ca30d87cca', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
2019-03-22 17:46:59,602          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet MESSAGE data 0
2019-03-22 17:46:59,769          INFO    engineio        socket.receive:52       e44e4b27172e4c468d09b0ca30d87cca: Received packet MESSAGE data 0/elastichq/ws,
2019-03-22 17:46:59,769          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet MESSAGE data 0/elastichq/ws
2019-03-22 17:46:59,776          INFO    engineio        socket.handle_get_request:101   e44e4b27172e4c468d09b0ca30d87cca: Received request to upgrade to websocket
2019-03-22 17:46:59,789          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet NOOP data None
2019-03-22 17:46:59,799          INFO    engineio        socket.receive:52       e44e4b27172e4c468d09b0ca30d87cca: Received packet MESSAGE data 2/elastichq/ws,["join",{"room_name":"xxx-log::nodes"}]
2019-03-22 17:46:59,800          INFO    socketio        server._handle_event:453        received event "join" from e44e4b27172e4c468d09b0ca30d87cca [/elastichq/ws]
2019-03-22 17:46:59,814          INFO    engineio        socket._websocket_handler:203   e44e4b27172e4c468d09b0ca30d87cca: Upgrade to websocket successful
2019-03-22 17:47:24,771          INFO    engineio        socket.receive:52       e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:47:24,773          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None
2019-03-22 17:47:50,024          INFO    engineio        socket.receive:52       e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:47:50,024          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None
2019-03-22 17:48:16,037          INFO    engineio        socket.receive:52       e44e4b27172e4c468d09b0ca30d87cca: Received packet PING data None
2019-03-22 17:48:16,039          INFO    engineio        socket.send:91          e44e4b27172e4c468d09b0ca30d87cca: Sending packet PONG data None

window.location:

ancestorOrigins: DOMStringList {length: 0}
assign: ƒ ()
hash: "#!/clusters/xxx-log/nodes"
host: "logger.xxx.pl"
hostname: "logger.xxx.pl"
href: "https://logger.xxx.pl/elastichq/#!/clusters/xxx-log/nodes"
origin: "https://logger.xxx.pl"
pathname: "/elastichq/"
port: ""
protocol: "https:"
reload: ƒ reload()
replace: ƒ ()
search: ""
toString: ƒ toString()
valueOf: ƒ valueOf()
Symbol(Symbol.toPrimitive): undefined
__proto__: Location
royrusso commented 5 years ago

Moving to 3.6.0. Depends on having #485 working, so we can test nginx + HQ and close all related issues with reverse proxies.

chrysalisloyalty commented 4 years ago

@royrusso just to comment on this one. We've implemented your great tool on our prod stack but are battling with a couple of security concerns. Nginx gives us IP restriction as well as basic HTTP authentication so this issue would be good to be resolved. We are just going to open the firewall port to restricted IP addresses but there is still a low threat vector in that you could snoop and use the query functionality to access Live data and potentially Personally Identifiable Information. Another way around that is to make the query functionality configurable so we can just turn it off. Will leave it to the experts whether any of this is feasible or you have any alternative ideas