Load balancing across multiple kibana nodes fail

[mohamed-mfarag]

Hi,

We have 2 kibana nodes and we have installed logtrail on both nodes and we are using apache reverse proxy to balance the requests between the 2 nodes.

However once we start up both nodes in the same time we receive the attached error

there is no error reported in kibana logs and when we checked back with elastic support, they came back pointing toward the logtrail plugin as causing this issue, i have tried to use disable the plugin on both nodes and try again and it did resolve the issue.

Is there any possibility to resolve this issue from your side, i have very limited experience in this area so i cant locate the issue on my side.

Please let me know what kind of info you need to help you trace down the issue.

ELK stack version 5.5.1

Reagrds Mohamed Farag

[sivasamyk]

@mohamed-mfarag I have not tested logtrail plugin in this set up. Did you get any info from Elastic support regarding the issue cause by logtrail plugin? Can you also attach the errors if any, from the Chrome developer console?

[mohamed-mfarag]

@sivasamyk Sadly Elastic didnt offer much as logtrail is not an officially supported plugin, Chrome does not show any errors, that was the strange thing and why i was not able to locate the error in the beginning.

[sivasamyk]

Can you share the apache reverse proxy configuration for Kibana? I will try to reproduce the issue locally.

[mohamed-mfarag]

Please find it below :

---

<Proxy "balancer://balancer-kibana"> BalancerMember "http://kibana-app01:5601" BalancerMember "http://kibana-app02:5601"

ProxyPass /kibana balancer://balancer-kibana ProxyPassReverse /kibana balancer://balancer-kibana

ProxyPass /bundles balancer://balancer-kibana/bundles ProxyPassReverse /bundles balancer://balancer-kibana/bundles

ProxyPass /logtrail balancer://balancer-kibana/logtrail ProxyPassReverse /logtrail balancer://balancer-kibana/logtrail

ProxyPass /status balancer://balancer-kibana/status ProxyPassReverse /status balancer://balancer-kibana/status

ProxyPass /api balancer://balancer-kibana/api ProxyPassReverse /api balancer://balancer-kibana/api

ProxyPass /api/status balancer://balancer-kibana/api/status ProxyPassReverse /api/status balancer://balancer-kibana/api/status

ProxyPass /plugins balancer://balancer-kibana/plugins ProxyPassReverse /plugins balancer://balancer-kibana/plugins

ProxyPass /app balancer://balancer-kibana/app ProxyPassReverse /app balancer://balancer-kibana/app

ProxyPass /elasticsearch balancer://balancer-kibana/elasticsearch ProxyPassReverse /elasticsearch balancer://balancer-kibana/elasticsearch

ProxyPass /login balancer://balancer-kibana/login ProxyPassReverse /login balancer://balancer-kibana/login

ProxyPass /logout balancer://balancer-kibana/logout ProxyPassReverse /logout balancer://balancer-kibana/logout

ProxyPass /app balancer://balancer-kibana/app ProxyPassReverse /app balancer://balancer-kibana/app

ProxyPass /es_admin balancer://balancer-kibana/es_admin ProxyPassReverse / balancer://balancer-kibana/es_admin

ProxyPass /ui balancer://balancer-kibana/ui ProxyPassReverse /ui balancer://balancer-kibana/ui

ProxyPass / balancer://balancer-kibana/app/kibana ProxyPassReverse / balancer://balancer-kibana/app/kibana

ProxyPreserveHost on

[sivasamyk]

@mohamed-mfarag I tried with attached conf (000-default.conf) and it works. Kibana version is 5.5.2

apache2 -v

Server version: Apache/2.4.18 (Ubuntu)
Server built:   2017-07-27T14:34:01

reverse-proxy.txt

[mohamed-mfarag]

Do you have xpack also enabled ?

---

From: Sivasamy Kaliappan notifications@github.com Sent: Monday, September 18, 2017 10:01:25 AM To: sivasamyk/logtrail Cc: mohamed-mfarag; Mention Subject: Re: [sivasamyk/logtrail] Load balancing across multiple kibana nodes fail (#180)

@mohamed-mfaraghttps://github.com/mohamed-mfarag I tried with attached conf (000-default.conf) and it works. Kibana version is 5.5.2

apache2 -v

Server version: Apache/2.4.18 (Ubuntu) Server built: 2017-07-27T14:34:01

reverse-proxy.txthttps://github.com/sivasamyk/logtrail/files/1309998/reverse-proxy.txt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/sivasamyk/logtrail/issues/180#issuecomment-330151783, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AOK3TSYOpNzeABd_VEQCOcJ_vw9uL_EKks5sjiNVgaJpZM4PY4MS.

[sivasamyk]

No. Mine is plain Kibana installation with logtrail. I will try with x-pack installed.

[sivasamyk]

Oops. Looks like xpack is licensed. I do not have license to run x-pack.

[mohamed-mfarag]

Sorry for the late reply, i was not able to update for the last 2 weeks, however i know there is a trail license for x-pack, would you be able to test using this one ?

[sivasamyk]

@mohamed-mfarag There is a solution suggested in this link https://discuss.elastic.co/t/kibana-uncaught-typeerror/82107 for same Kibana error.

[mohamed-mfarag]

This issue is for only single node as far as i can see from the post and if its related to the user shell it should happen on both node but the issue we have happen only if we use both nodes and balance them using Apache - but if we stop one every thing work fine, have you managed to test with X-pack , please let me know if i can help ?

[johnseekins]

@mohamed-mfarag You have sticky sessions enabled for Kibana in your proxy? Something like https://cwiki.apache.org/confluence/display/OFBIZ/Sticky+session+load+balancing+with+Apache+and+mod_balancer for Apache2?

Kibana's docs say sticky sessions should be enabled.

[mohamed-mfarag]

Hi @johnseekins

Thanks for your suggestion, that did solve the issue !