Closed michaeleino closed 6 years ago
lazyfrosch
commented
7 years ago Just noticed your issue...
Never saw that error before, but haven't worked on the module in a while. Its still a kind of preview for Elasticsearch integration.
Did you search something specific?
michaeleino
commented
7 years ago the module is a genius integration & a must
but i can't figure out what the problem to work around.. might i just can't install/use properly.
or it might be missed package dependacy or php module...
can u please provide what is the dependences required? & a quick start guide?
michaeleino
commented
7 years ago Any help here ?
lazyfrosch
commented
7 years ago Just rebuild my development environment for the future.
So I'm running on Elasticsearch 5.4.1 without a problem. I'm able to search, create event types, and browse through them.
Where do you get that error? I need:
michaeleino
commented
7 years ago Hello @lazyfrosch I have just upgraded my packages to be: Ubuntu 16.04.2 LTS PHP 7.0.18-0ubuntu0.16.04.1 (cli) ( NTS ) icinga2 2.6.3-1 icingaweb2 2.4.1 Elasticsearch 5.4.1
hitting the same problem :/
ERROR
Filter column "message" not found
#0 /usr/share/php/Icinga/Repository/Repository.php(1003): Icinga\Repository\Repository->requireFilterColumn('_all', 'message', Object(Icinga\Repository\RepositoryQuery), Object(Icinga\Data\Filter\FilterMatch))
#1 /usr/share/php/Icinga/Repository/Repository.php(1007): Icinga\Repository\Repository->requireFilter('_all', Object(Icinga\Data\Filter\FilterMatch), Object(Icinga\Repository\RepositoryQuery), false)
#2 /usr/share/php/Icinga/Repository/RepositoryQuery.php(276): Icinga\Repository\Repository->requireFilter('_all', Object(Icinga\Data\Filter\FilterAnd), Object(Icinga\Repository\RepositoryQuery))
#3 /usr/share/php/Icinga/Repository/RepositoryQuery.php(247): Icinga\Repository\RepositoryQuery->addFilter(Object(Icinga\Data\Filter\FilterAnd))
#4 /usr/share/php/Icinga/Web/Widget/FilterEditor.php(341): Icinga\Repository\RepositoryQuery->applyFilter(Object(Icinga\Data\Filter\FilterAnd))
#5 /usr/share/php/Icinga/Web/Controller.php(223): Icinga\Web\Widget\FilterEditor->handleRequest(Object(Icinga\Web\Request))
#6 /usr/share/icingaweb2/modules/elasticsearch/application/controllers/SearchController.php(30): Icinga\Web\Controller->setupFilterControl(Object(Icinga\Repository\RepositoryQuery), NULL, NULL, Array)
#7 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Elasticsearch\Controllers\SearchController->indexAction()
#8 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch('indexAction')
#9 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#10 /usr/share/php/Icinga/Application/Web.php(389): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#11 /usr/share/php/Icinga/Application/webrouter.php(109): Icinga\Application\Web->dispatch()
#12 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...')
#13 {main}Probably I do a wrong installation and/or there is missing dependency!! Installation performed exactly as follows:
#wget https://github.com/Icinga/icingaweb2-module-elasticsearch/archive/master.zip
#unzip master.zip
#mkdir /usr/share/icingaweb2/modules/elasticsearch
#cp -rv icingaweb2-module-elasticsearch-master/* /usr/share/icingaweb2/modules/elasticsearch/From icingaweb2 > Configuration > Modules > elasticsearch click the enable button From the same page I go to the Elasticsearch tab & defined
Elasticsearch URL: http://x.x.x.x:9200
Logstash index pattern: myindex-*where x.x.x.x is ELK IP my ELK doesn't have username/password/Certificate so I left these fields empty, and I can access the ELK through "elasticsearch toolbox, kibi, kibana, and other apps"
from Elasticsearch > Event search > I typed test in the search field & hit enter.
so the URL through search is http://x.x.x.x/icingaweb2/elasticsearch/search?(message=%2Atest%2A)
Webserver: nginx version: nginx/1.10.0 (Ubuntu)
PHP -m:
[PHP Modules]
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
imagick
intl
json
ldap
libxml
mbstring
mcrypt
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zlib
[Zend Modules]
Zend OPcacheN.B icinga & ELK on the same server & IP
What missing here !? it will be very helpful to create a quick start guide... as I managed this module as just others.
lazyfrosch
commented
7 years ago Ok, I finally understand what happens. I have the same error when my Elasticsearch is empty
The modules tries to get any columns that exist in your Elasticsearch indizes, so it can offer you a list for searching.
However, by default when you enter something into the search box, "message" is a default column to search for.
You should be able to see that the module can't find any columns by opening the filterEditor: http://localhost/icingaweb2/elasticsearch/search?modifyFilter=1
The dropdown in the editor should be empty.
Questions:
We query the mapping API of Elasticsearch to get a list:
curl "http://localhost:9200/logstash-*/_mappings"Empty result: {}
Example from logstash syslog data:
curl "http://localhost:9200/logstash-*/_mappings?pretty=true"{
"logstash-2017.06.21" : {
"mappings" : {
"_default_" : {
"_all" : {
"enabled" : true,
"norms" : false
},
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"keyword" : {
"type" : "keyword"
}
},
"norms" : false,
"type" : "text"
}
}
}
],
// ...that's good to catch the problem, it is the most hard step....
Yes the module can't get can't find any columns by opening the filter Editor:
http://x.x.x.x/icingaweb2/elasticsearch/search?modifyFilter=1
Answers:
curl "http://x.x.x.x:9200/myindex-2017/_mappings?pretty=true""message" : {
"type" : "text"
},
"received_at" : {
"type" : "date"
},
"sentdate" : {
"type" : "text",
"index" : false
},
"senttime" : {
"type" : "date"
},
"source" : {
"type" : "text"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"targetid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"targetindex" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "date"
},
"timezone" : {
"type" : "text"
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}What should I check, is there any log anywhere? or how to check the request initiated from the module to the ELK ?
michaeleino
commented
7 years ago in the syslog I can find below, I can't figure out but it may help!
Jun 21 18:06:42 localhost icingaweb2[14317]: Cannot order by column "@timestamp" in repository "Icinga\Repository\Repository"
Jun 21 18:06:42 localhost icingaweb2[14317]: Rendered elasticsearch filter: {"bool":{"must":[{"match_all":{}}]}}
Jun 21 18:06:42 localhost icingaweb2[14317]: Rendered elasticsearch filter: {"bool":{"must":[{"match_all":{}}]}}
Jun 21 18:06:50 localhost icingaweb2[31240]: Cannot order by column "@timestamp" in repository "Icinga\Repository\Repository"
Jun 21 18:06:50 localhost icingaweb2[31240]: Rendered elasticsearch filter: {"bool":{"must":[{"match_all":{}}]}}
Jun 21 18:06:50 localhost icingaweb2[31240]: Rendered elasticsearch filter: {"bool":{"must":[{"match_all":{}}]}}
Jun 21 18:06:54 localhost icingaweb2[5545]: Icinga\Exception\QueryException in /usr/share/php/Icinga/Repository/Repository.php:1194 with message: Filter column "message" not found #0 /usr/share/php/Icinga/Repository/Repository.php(1003): Icinga\Repository\Repository->requireFilterColumn('_all', 'message', Object(Icinga\Repository\RepositoryQuery), Object(Icinga\Data\Filter\FilterMatch)) #1 /usr/share/php/Icinga/Repository/Repository.php(1007): Icinga\Repository\Repository->requireFilter('_all', Object(Icinga\Data\Filter\FilterMatch), Object(Icinga\Repository\RepositoryQuery), false) #2 /usr/share/php/Icinga/Repository/RepositoryQuery.php(276): Icinga\Repository\Repository->requireFilter('_all', Object(Icinga\Data\Filter\FilterAnd), Object(Icinga\Repository\RepositoryQuery)) #3 /usr/share/php/Icinga/Repository/RepositoryQuery.php(247): Icinga\Repository\RepositoryQuery->addFilter(Object(Icinga\Data\Filter\FilterAnd)) #4 /usr/share/php/Icinga/Web/Widget/FilterEditor.php(341): Icinga\Repository\RepositoryQuery->applyFilter(Object(Icinga\Data\Filter\FilterAnd)) #5 /usr/share/php/Icinga/Web/Controller.php(223): Icinga\Web\Widget\FilterEditor->handleRequest(Object(Icinga\Web\Request)) #6 /usr/share/icingaweb2/modules/elasticsearch/application/controllers/SearchController.php(30): Icinga\Web\Controller->setupFilterControl(Object(Icinga\Repository\RepositoryQuery), NULL, NULL, Array) #7 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Elasticsearch\Controllers\SearchController->indexAction() #8 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch('indexAction') #9 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response)) #10 /usr/share/php/Icinga/Application/Web.php(389): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response)) #11 /usr/share/php/Icinga/Application/webrouter.php(109): Icinga\Application\Web->dispatch() #12 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...') #13 {main}
@lazyfrosch I think I found the root cause I have stopped everything accessing elasticsearch & captured a tcpdump stream while accessing from icingaweb2 by this module...
here is a sample!
GET /logstash-%2A/_mappings HTTP/1.1
Host: localhost:9200
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 2
{}GET /logstash-%2A//_count HTTP/1.1
Host: localhost:9200
Accept: */*
Content-Length: 46
Content-Type: application/json
{"query":{"bool":{"must":[{"match_all":{}}]}}}HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 59
{"count":0,"_shards":{"total":0,"successful":0,"failed":0}}GET /logstash-%2A//_search HTTP/1.1
Host: localhost:9200
Accept: */*
Content-Length: 82
Content-Type: application/json
{"from":0,"size":100,"query":{"bool":{"must":[{"match_all":{}}]}},"_source":false}HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 121
{"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}I can see the plugin searching for the "logstash-" pattern, however I changed it from the config page ,
and reflected on /etc/icingaweb2/modules/elasticsearch/config.ini :
[elasticsearch]
index_pattern = "myindex-2017"
url = "http://localhost:9200"Please help
lazyfrosch
commented
7 years ago Oh small but evil bug, thanks for all the info.
I pushed a fix to master, please check if it works now. It should.
Please leave this issue open, I want to add some proper error messaging.
michaeleino
commented
7 years ago @lazyfrosch Amazing to see my ELK events on icingaweb thanks a lot, works fine
I'm leaving this issue open :)
lippserd
commented
6 years ago Closed because we changed the module. Search over events with the new implementation is not yet possible.
I have just installed/configured this module & tried to search, but i hit this :
Ubuntu 16.04 Php v7.0.15-0ubuntu0.16.04.4 icinga2 2.6.3 icingaweb2 2.4.1 Elasticsearch 5.3.1