sanikolaev commented 2 years ago

Manticore can work with Kibana like this:

# download manticore beta version with support for Kibana, check https://repo.manticoresearch.com/repository/kibana_beta/ for different OS versions  
wget https://repo.manticoresearch.com/repository/kibana_beta/ubuntu/jammy.zip  

# unarchive it  
unzip jammy.zip  

# install the packages  
dpkg -i build/*  

# switch Manticore to the mode supporting Kibana  
mysql -P9306 -h0 -e "set global log_management = 0; set global log_management = 1;"  

# start Kibana pointing it to Manticore Search instance listening on port 9308  
docker run -d --name kibana --rm -e ELASTICSEARCH_HOSTS=http://127.0.0.1:9308 -p 5601:5601 --network=host docker.elastic.co/kibana/kibana:7.4.2  

# install php and composer, download loading script and put into Manticore 1 million docs of fake users  
apt install php composer php8.1-mysql  
wget https://gist.githubusercontent.com/sanikolaev/13bf61bbe6c39350bded7c577216435f/raw/8d8029c0d99998c901973fd9ac66a6fb920deda7/load_manticore_sharded.php  
composer require fakerphp/faker  
php load_manticore_sharded.php 10000 16 1000000 16 1  

# don't forget to create an index patter in Kibana (user*)  

# run `docker stop kibana` to stop the Kibana server

If all went well you should see:

But it has some limitations. This task is to complete this integration.

unixfox commented 2 years ago

What are the current limitations?

tomatolog commented 2 years ago

user can not manage indexes \ cluster from Kibana UI as searchd provide only dummy replies.

Search with nested aggregations does not work.

There could be some more requests that daemon failed to parse or returns wrong reply

nickchomey commented 1 year ago

I just discovered manticore and am very curious about it. But I need Kibana. Has any progress been made on this integration/any estimate of when it might be done?

sanikolaev commented 1 year ago

Has any progress been made on this integration/any estimate of when it might be done?

Yes, we've made great progress on this integration, but unfortunately it's still in an active development stage. In particular we've been working actively on adapting Elasticsearch aggregation capabilities to Manticore which is essential for the integration with Kibana. Unfortunately, we won't make it to the upcoming release we are doing soon, but hopefully we'll have something to check out in a month or so.

unixfox commented 1 year ago

thank you for the feedback @sanikolaev. against which version of kibana are you working on? last time I tried manticore search on a recent kibana version 7.17.10 and 8.8.2 it was complaining about manticore search not being a compatible elasticsearch product and thus failing to launch completely.

sanikolaev commented 1 year ago

We are testing against 7.6.0.

nickchomey commented 1 year ago

That's great that you're working on es aggregations!

May I suggest using the latest version of kibana for testing? 7.6 is both 3.5 years old and not even the latest major version (8.x).

The changes to the elastic license probably dont really affect most manticore users - my understanding is it just stops you from starting something that competes with elastic cloud (like aws elasticsearch service did).

sc0rp10 commented 1 year ago

Hello! Is there any progress with integration with Kibana? We're looking for a replacement for ES as a log storage, but any other dashboards otheer than Kibana are pretty unusable.

sanikolaev commented 1 year ago

There's progress, but the work is not publicly available yet due to dependencies on some other things that are being implemented. We hope to make it all available for testing soon.

b17k0 commented 9 months ago

Any news?

sanikolaev commented 9 months ago

Any news?

Yes. This task has been blocked by the work related with improving date/time functions in Manticore - https://github.com/manticoresoftware/manticoresearch/commit/514d35b497f4bdb20b6473ff963752a978a4bb8d

That's now done and we are getting back to this task soon.

I remember @nickchomey and a few others in the Public Slack were interested in beta-testing the new functionality. We'll ping you guys as soon as we have something to give you. Hope it will happen soon.

deathalt commented 9 months ago

I'd like to join kibana integration beta-testing :slightly_smiling_face:

tomatolog commented 9 months ago

I've built the package for testing. You could grab it from here https://github.com/manticoresoftware/manticoresearch/actions/runs/7957643370 and test integration.

sanikolaev commented 9 months ago

I've built the package for testing. You could grab it from here https://github.com/manticoresoftware/manticoresearch/actions/runs/7957643370 and test integration.

Thanks. Let's prepare a better instruction for beta-testers, so it's clear how to install the beta version.

tomatolog commented 9 months ago

seems @timestamp field handling got changed that breaks the Logstash integration

curl localhost:9308/cli -d 'create table test(@timestamp text)'

curl localhost:9308/_bulk -H "Content-Type: application/x-ndjson" -d '
{ "index" : {"_index":"test"} }
{"@timestamp": "a"}
'

[{"total":0,"warning":"","error":"row 1, column 1: string expected"}]

tomatolog commented 9 months ago

should be fixed at the master head https://github.com/manticoresoftware/manticoresearch/commit/15975cf3e6471d3a3ca4c2f19e4e930359a14622

sanikolaev commented 8 months ago

@Nick-S-2018 As discussed, reassining to you to continue working on the project from the Buddy's standpoint. Here's an updated instruction how to set up Kibana + Manticore + Elasticsearch + HAproxy + Mitmproxy:

Testing Kibana

# Schema

kibana @5601 -> haproxy @9201 -> mitm @19308 (UI @29308) -> manticore @9308
                              -> mitm @19200 (UI @29200) -> elastic @9200

apt -y update
apt -y install docker.io mysql-client php php-mysql jq

# Init .screenrc
cat << EOF > ~/.screenrc
caption always "%{= 45}%{+b w}Screen: %n | %h %=%t"
hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<"
shell -/bin/bash
EOF

# Install and start Manticore normally

# Install mitmproxy
wget https://downloads.mitmproxy.org/10.2.2/mitmproxy-10.2.2-linux-x86_64.tar.gz
tar -xvf mitmproxy-10.2.2-linux-x86_64.tar.gz

# Run Elasticsearch
mkdir es_data && chmod 777 es_data
docker run --name elasticsearch --rm -e discovery.type=single-node -v $(pwd)/es_data:/usr/share/elasticsearch/data --network=host docker.elastic.co/elasticsearch/elasticsearch:7.6.0

# Run haproxy

cat << EOF > haproxy_kibana.cfg
global
   log stdout format raw local0
   daemon

defaults
   log global
   mode http

frontend esms
   bind *:9201
   default_backend manticore
   acl idx path_beg /. /_
   use_backend elasticsearch if idx

backend manticore
   server manti 127.0.0.1:19308

backend elasticsearch
   server es 127.0.0.1:19200
EOF

docker run -v $(pwd)/haproxy_kibana.cfg:/usr/local/etc/haproxy/haproxy.cfg --network=host haproxy:2.6.5

# Run mitmproxy for Elasticsearch 
./mitmproxy --mode reverse:http://localhost:9200 -p 19200

  # Alternatively, run mitmweb
  # ./mitmweb --web-host 0.0.0.0 --web-port 29200 --mode reverse:http://localhost:9200 -p 19200
  # Open http://<host>:29200

# Run mitmproxy for Manticore
./mitmproxy --mode reverse:http://localhost:9308 -p 19308

  # Alternatively, run mitmweb
  # ./mitmweb --web-host 0.0.0.0 --web-port 29308 --mode reverse:http://localhost:9308 -p 19308
  # Open http://<host>:29308

# Install and start Manticore normally
wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && \
sudo dpkg -i manticore-dev-repo.noarch.deb && \
sudo apt -y update && \
sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l manticore-galera manticore-galera-dbgsym 

# Run Kibana through the proxy
docker run --name kibana --rm -e SERVER_NAME=kibana -e ELASTICSEARCH_HOSTS=http://127.0.0.1:9201 -e ELASTICSEARCH_USERNAME=elastic -e ELASTICSEARCH_PASSWORD=pass -e SERVER_PORT=5601 -e KIBANA_AUTOCOMPLETETIMEOUT=10000 -e KIBANA_AUTOCOMPLETETERMINATEAFTER=100000000 -e ELASTICSEARCH_REQUESTTIMEOUT=600000 -e ELASTICSEARCH_SHARDTIMEOUT=0 --network=host docker.elastic.co/kibana/kibana:7.6.0

gaby commented 8 months ago

@sanikolaev Why such a complicated setup? Can I just run manticore / kibana in compose without all this?

Can Kibana v8.x be used?

sanikolaev commented 8 months ago

@gaby That's a temporary setup for development. Of course when we are done with the integration you'll be able to just point Kibana to Manticore. But we are not there yet. Still some work left until we can announce that.

gaby commented 8 months ago

@sanikolaev Thanks for the explanation, looking forward to this! 💪