search

rashidkpc / kibana2

Kibana was acquired by Elastic in 2013. See elastic/kibana. More info at http://kibana.org
Other
1.19k stars 246 forks source link

Build as deployable blob #324

Open jordansissel opened 11 years ago

jordansissel commented 11 years ago

To avoid new users getting lost fighting a losing battle against their distro's missteps in packaging, a standalone runnable blob could help.

Java jar or the way Vagrant releases work are fine with me. Either way, a single deployable thing could help new users and reduce support requests related to installation problems.

Telmo commented 11 years ago

it is already a gem, you just need to build it.

jordansissel commented 11 years ago

'gem install kibana' requires a number of things to have been done correctly, none of which are (and should not necessarily be) documented on the kibana site - you need ruby, ruby development packages, a C compiler, a C++ compiler, and many many other things - all of which are different on different OS platforms and distributions.

New users often stumble while trying to setup Kibana because they use OS distributions that punish them (for example, by not making it obvious they need a compiler or the ruby development packages instealled). As a result, new users will confuse a poor kibana installation experience (due to OS badness) with Kibana being a poor project.

Happier users == more users == stronger community :)

Logstash went through the same problems years ago, which is why you don't see logstash available from rubygems anymore, for exmaple ;)

Telmo commented 11 years ago

@jordansissel you will have the same issue with many of the gems out there, as a matter of fact any gem that has a c extension or depends on external libraries will have this issue. But Kibana does not suffer from this.

Based on the kibana.gemspec file this is the list of gems it requires and their external dependencies:

The only required gem that needs to be compiled is "thin" and that's because think requires eventmachine which needs to be compiled (debian/ubuntu provides the ruby-eventmachine deb, RH/CentOS/etc provide rubygem-eventmachine, OSX ... well if you are installing this in OSX its probably development.. you should probably know what you are doing and already using brew)

Gems are a cross-platform packaging system, only 1% of the gems need to be compiled, on top of that most modern OS already ship with a version of ruby packaged, and they offer this gems that need to be compiled as a native package, the mysql and mysql2 gems are infamous for this.

New users should just be able to do gem install kibana edit a config file and go. I agree that the current state of packaging of the project is not finalized but 95% of the work is already there and the gem does build without issues.

The KibanaConfig.rb should be a json or yml file or the configuration should be done via a block passed to a config class within Kibana. I may work on that and create a pull request.

I don't think logstash is an application to mirror after. I understand the reasoning behing everything together as a shippable package for agents. In a blackbox situation this may be a good thing to do, but then I am forced to install java to get it up and running. What if my blackbox doesn't have java, what if I want logstash to use my already installed and perfectly functioning ruby instance. What if every agent decides to follow this pattern and you end up with 5 ruby instances individually installed in your system and java to run them all. Taking it to the extreme, can you imagine if whenever you install chrome it wanted to install ChromeOS along with it, it you want to install IE it would install its own version of windows with it, etc, etc.

Jar files are bad, they force me to install java. Vagrant is brilliant, but I don't want to spin a VM to run Kibana, and Vagrant installs its components using chef or puppet... which will do a gem install.

Also, one of the strengths that I've found with Kibana and that should be really highlighted in the documentation is that you do not need logstash to use Kibana, kibana talks to elasticsearch and as long as the data is in the expected format in elasticsearch what put it there or how doesn't matter. I am currently running fluentd instad of logstash and Kibana is working perfectly, it may have started as a logstash frontend but it has grown beyond that

ludwigb3 commented 11 years ago

I do not have a C-compiler available on the Linux box made available to me [non-root]. Do I need at least a pre-processor? At the moment I do not see the ruby.h files, except in the logstash directory. Can I use that one?

appadm@fldevlxapplog001 ~/Kibana/ $ bundle install --path /opt/app/.gem/ruby/1.8/bin/ Fetching gem metadata from http://rubygems.org/......... Fetching gem metadata from http://rubygems.org/.. Resolving dependencies... Installing rake (10.0.4) Installing daemons (1.1.9) Installing diff-lcs (1.1.3) Installing eventmachine (1.0.0) Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/usr/bin/ruby extconf.rb mkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h

Gem files will remain installed in /opt/app/.gem/ruby/1.8/bin/ruby/1.8/gems/eventmachine-1.0.0 for inspection. Results logged to /opt/app/.gem/ruby/1.8/bin/ruby/1.8/gems/eventmachine-1.0.0/ext/gem_make.out

An error occurred while installing eventmachine (1.0.0), and Bundler cannot continue. Make sure that gem install eventmachine -v '1.0.0' succeeds before bundling.

Telmo commented 11 years ago

@ludwigb3 your issue is not kibana but eventmachine (which is required by thin which in turn is required by kibana) depending of your Linux flavor there are packages available with it already compile (see my previous post)

The only required gem that needs to be compiled is "thin" and that's because think requires eventmachine which needs to be compiled (debian/ubuntu provides the ruby-eventmachine deb, RH/CentOS/etc provide rubygem-eventmachine, OSX ... well if you are installing this in OSX its probably development.. you should probably know what you are doing and already using brew)

another solution (although I don't recommend it) would be to break the thin dependency and have it use a different webserver to serve the app.

ludwigb3 commented 11 years ago

Got the sysadm to install gcc & g++, and a fuller ruby install [including ruby.h]. Compiled & running. Thanks.

grayaii commented 11 years ago

dang. just got bit by this too. I'm trying to to install Kibana, but I got stuck on eventmachine, same as ludwigb3. I kinda agree with Jordan to make it easier to install this tool. Now I have to put my beer down, hunt down the sys admins to trouble shoot this, wait for the them to install everything that is needed, and by that time, my beer will be too warm.

madAndroid commented 11 years ago

aren't you glad that you don't have to do any of this compilation bulls**t with Kibana3?

cmera commented 11 years ago

Are you guys able to use this effectively as a tool to displace Splunk?

On Wednesday, July 17, 2013, madAndroid wrote:

aren't you glad that you don't have to do any of this compilation bulls**t with Kibana3?

— Reply to this email directly or view it on GitHubhttps://github.com/rashidkpc/Kibana/issues/324#issuecomment-21105704 .

christopher l mera camusny@gmail.com 917.731.2081

grayaii commented 11 years ago

OK. I got the sys admin to figure this out. I think he ended up yum installing various ruby packages until it worked. @madAndroid I'm a complete noob at this. On the elasticsearch machine I only have redis/logstash/elasticsearch installed, and I have like 25 other machine running logstash, shipping logs to the elasticsearch machine. It works. I have no standalone web server on the elasticsearch machine, but elasticsearch and logstash has it's own webserver running, which is how I do searches, so I'm not sure how to install Kibana3. (I guess I can try creating a "_site" directory under elasticsearch/plugin and dump the files from Kibana 3 there to see if that works. In fact, I've never even heard of Kibana 3. I originally did a google search for "kibana" which took me to http://kibana.org/intro.html, and I followed the instructions there (ie... bundle install).

ludwigb3 commented 11 years ago

@camunsy. We had selected Splunk, but then balked on the price. I have Logstash/ElasticSearch/Kibana2 working well in a proof of concept VM environment, using rsyslog and Nxlog as logshippers on Linux/Windows boxes. Looks good, though hard core evaluation of its scalability & wide-spread usability with big data, applications, etc. will part of a formal project that is still gearing up. And I have not implemented authentication yet [will use a web proxy, which is also needed for Kibana 3 & to protect ElasticSearch].

For queueing, I was planning to use Redis, but do not see how that would work w/ Nxlog, which might be able to do the queuing itself. Actually what I really would like to know, is how I can set the Logstash 'type' in Nxlog agent rather than doing it on the Logstash server.