Kibana UI is not showing any data from ES

[Eternity-RS]

Hi,

I have setup as logs >> logstash -->> Redis server >> (logstash ) Elasticsearch >> Kibana

I am able to recieve logs on Redis and then also to ES but nothing gets visible on Kibana UI.

Have also configured KibanaConfig.rb for ES ip:port as Elasticsearch = "X.X.X.X:9200" What could be the issue?

BR //

[Eternity-RS]

Forgot to mention , even predefined filters are also not visible on left side of Kibana UI.

BR //

[ghubnik]

I am experiencing the same issue. Confirmed that the built-in ES web UI is functioning, not too overloaded - i.e. it's reasonably responsive and new logs are showing up all the time.

I'm running Kibana on the same host as the elasticsearch node, so am using Elasticsearch = "localhost:9200" in KibanaConfig.rb.

I'm not 100% confident I've got all the ruby gems set up correctly because the system is behind a firewall and I had to install Kibana on a host that had access to rubygems repository and then copy the gems over to the actual system using the technique outlined here.

When I fire up "ruby --debug kibana.rb" I see this:

[04:25]:[root@myhost:Kibana-0.2.0]# ruby --debug kibana.rb 
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems.rb:1113 - no such file to load -- rubygems/defaults/operating_system
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/config_file.rb:50 - no such file to load -- Win32API
Exception `NoMethodError' at /usr/lib/ruby/1.8/rational.rb:78 - undefined method `gcd' for Rational(1, 2):Rational
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- sinatra
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- Win32API
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- Win32API
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- json
Using Ext extension for JSON.
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- xml/parser
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- xml/parser
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- xmlparser
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- xmlparser
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- xmlparser
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- xmlscan/scanner
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- xmlscan/scanner
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- xmlscan/scanner
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- tzinfo
./lib/query.rb:23: warning: method redefined; discarding old initialize
./lib/query.rb:66: warning: method redefined; discarding old to_s
./lib/query.rb:81: warning: method redefined; discarding old initialize
./lib/query.rb:102: warning: method redefined; discarding old initialize
./lib/query.rb:126: warning: method redefined; discarding old initialize
./lib/query.rb:150: warning: method redefined; discarding old initialize
./lib/query.rb:191: warning: method redefined; discarding old initialize
./lib/query.rb:214: warning: method redefined; discarding old initialize
./lib/client_request.rb:20: warning: method redefined; discarding old initialize
./lib/client_request.rb:54: warning: method redefined; discarding old to_s
./lib/client_request.rb:62: warning: method redefined; discarding old hash
./lib/compat.rb:3: warning: method redefined; discarding old ruby_18
./lib/compat.rb:6: warning: method redefined; discarding old ruby_19
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- fastercsv
Exception `LoadError' at /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- thin
Exception `NameError' at /usr/lib/ruby/gems/1.8/gems/eventmachine-1.0.3/lib/eventmachine.rb:1 - uninitialized constant EventMachine
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:14: warning: instance variable @silent not initialized
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:14: warning: instance variable @debug not initialized
== Sinatra/1.4.2 has taken the stage on 5601 for development with backup from Thin
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:15: warning: instance variable @silent not initialized
>> Thin web server (v1.5.1 codename Straight Razor)
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:14: warning: instance variable @silent not initialized
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:14: warning: instance variable @debug not initialized
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:13: warning: instance variable @silent not initialized
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:13: warning: instance variable @trace not initialized
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:15: warning: instance variable @silent not initialized
>> Maximum connections set to 1024
/usr/lib/ruby/gems/1.8/gems/thin-1.5.1/lib/thin/logging.rb:15: warning: instance variable @silent not initialized
>> Listening on 0.0.0.0:5601, CTRL+C to stop

As it loads up, while running top, I hardly see any load at all from the ruby process. All the while, redis is bumping along at one or 2% CPU usage and the logstash monolithic jar JVM is consuming around 10% to 20% CPU. It looks like Kibana is doing no work at all.

The weird thing is that it looks like Kibana is not making any connections to port 9200 on the localhost. I've confirmed this by running tcpdump. When I do

[04:36]:[root@myhost:~]# telnet localhost 9200
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
blah
Connection closed by foreign host.

I see this

[04:36]:[root@myhost:~]# tcpdump -nn -i lo port 9200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
04:36:30.196429 IP 127.0.0.1.45701 > 127.0.0.1.9200: Flags [S], seq 2012529634, win 32792, options [mss 16396,nop,wscale 8], length 0
04:36:30.196448 IP 127.0.0.1.9200 > 127.0.0.1.45701: Flags [S.], seq 2011457799, ack 2012529635, win 32792, options [mss 16396,nop,wscale 8], length 0
04:36:30.196460 IP 127.0.0.1.45701 > 127.0.0.1.9200: Flags [.], ack 1, win 129, length 0
04:36:33.330710 IP 127.0.0.1.45701 > 127.0.0.1.9200: Flags [P.], seq 1:7, ack 1, win 129, length 6
04:36:33.330731 IP 127.0.0.1.9200 > 127.0.0.1.45701: Flags [.], ack 7, win 129, length 0
04:36:33.333667 IP 127.0.0.1.9200 > 127.0.0.1.45701: Flags [F.], seq 1, ack 7, win 129, length 0
04:36:33.333790 IP 127.0.0.1.45701 > 127.0.0.1.9200: Flags [F.], seq 7, ack 2, win 129, length 0
04:36:33.333813 IP 127.0.0.1.9200 > 127.0.0.1.45701: Flags [.], ack 8, win 129, length 0

However, when I start up Kibana it never gets beyond the message that it's "fetching some interesting data", and entering a search string also causes no output in the tcpdump.

And the strangest thing of all, is that when I originally set up Kibana on the host that had access to rubygems, just to get started, I used an ssh tunnel to connect the ES server (I left Elasticsearch = "localhost:9200" in KibanaConfig.rb). It was working fine after I first set it up. Now that I try the same system, using ssh tunnelling, it's no longer working. Just like the instance I'm colocating with ES, when I'm not seeing any TCP packets from Kibana, but when I telnet the tunnelled port, I do see the packets.

Do the exceptions in the debug output look like they're implicated in the symptoms I'm seeing? If not, is there further debugging I can try?

[ghubnik]

Update - it looks like the technique "installing gems with no network" that I used to try to satisfy dependencies was a bit too crude.

Running "bundle check" shows the following problems

[22:18]:[kibana@myhost]# bundle check
Resolving dependencies...
The following gems are missing
 * diff-lcs (1.1.3)
 * eventmachine (1.0.0)
 * json (1.7.5)
 * rack (1.4.1)
 * rack-protection (1.2.0)
 * tilt (1.3.3)
 * sinatra (1.3.3)
 * thin (1.5.0)
 * tzinfo (0.3.35)
 * rspec-core (2.11.1)
 * rspec-expectations (2.11.3)
 * rspec-mocks (2.11.3)
 * rspec (2.11.0)
Install missing gems with `bundle install`

For comparison, "gem list" shows me I have slightly different versions of these gems installed:

[22:37]:[kibana@myhost]# gem list
*** LOCAL GEMS ***
bundler (1.3.5)
daemons (1.1.9)
diff-lcs (1.2.3)
eventmachine (1.0.3)
fastercsv (1.5.5)
json (1.7.7)
rack (1.5.2)
rack-protection (1.5.0)
rake (10.0.4)
rspec (2.13.0)
rspec-core (2.13.1)
rspec-expectations (2.13.0)
rspec-mocks (2.13.1)
sinatra (1.4.2)
thin (1.5.1)
tilt (1.3.7)
tzinfo (0.3.37)

The versions of these missing dependencies are different to what was installed.

As an extra data point, here's what Jordan provided as the correct list of dependencies.

[dkincaid]

I'm having the same issue here with a new install. Anyone have any ideas? My "bundle check" says that all the dependencies are satisfied. When I point my browser to "http://localhost:5601" it redirects to "http://localhost:5601/#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6OTAwLCJncmFwaG1vZGUiOiJjb3VudCJ9" then says

Oops! Something went terribly wrong. I'm not totally sure what happened, but maybe refreshing, or hitting Reset will help. If that doesn't work, you can try restarting your browser. If all else fails, it is possible your configuation has something funky going on.

If it helps, I received a 0 error from: api/search/eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6OTAwLCJncmFwaG1vZGUiOiJjb3VudCJ9?_=1377282138169

[david-caro]

I have the same problem using the non-service version, and I've found out that if I specify http://localhost:9200 as the ES url in the configuration it works... (I created a tunnel from my machine to the actual ES server).

It seems that when trying to connect to non-localhost servers it requests the available methods with HTTP OPTIONS request and ES answers with PUT and DELETE only, maybe that's what it's stopping it from doing the usual POST request:

Request:

OPTIONS /logstash-2013.09.05/_search HTTP/1.1
Host: log-server.local:9200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Origin: http://localhost
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type,x-requested-with
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

Response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 1728000
Access-Control-Allow-Methods: PUT, DELETE
Access-Control-Allow-Headers: X-Requested-With
Content-Type: text/plain; charset=UTF-8
Content-Length: 0

[david-caro]

I've just tested putting an apache server in the middle that overwrite the Access-Control-Allow-Methods header to appends the GET and POST methods, but no luck.

It's really strange that the OPTIONS request is only done when the server is not localhost... I'll continue investigating.

[david-caro]

Finally I was right, I forgot to add the Access-Control-Allow-Headers header too, with both headers it works. Here's the http config I used to test adding those headers:

Listen *:9201

<VirtualHost *:9201>
  RewriteEngine On
  Header set Access-Control-Allow-Methods "PUT, DELETE, POST, GET"
  Header set Access-Control-Allow-Headers "Content-Type,X-Requested-With"
  RewriteRule ^(.*)$ http://localhost:9200$1 [P,L]
</VirtualHost>

Then using http://server-name:9201 as ES server in the configuration of kibana, it works (opening the ports too, of course). I think that it's more a problem of ES than kibana itself, ES should say that it supports those methods and those headers imho.