sirensolutions / kibi

PLEASE READ: Kibi is now "Siren Investigate", part of the Siren Platform. This code repository is only provided to facilitate code review and integration by Siren customers and cannot be successfully compiled. The code also requires Siren Federate to be installed in Elasticsearch.
http://siren.io
Other
494 stars 111 forks source link

Query on Oracle database fails #60

Closed jd8935 closed 7 years ago

jd8935 commented 8 years ago

Versions

Kibi: kibi-4.5.3-3-windows64-demo-full

Operating System: Windows 7

Behavior

Create a simple query in "Edit Queries over External Datasource" select x from dual

Then press Save. Kibi immediately exits without any error messages. Browser screen reports "Courier Fetch Error: unhandled courier request error: unknown error"

Expected:

Query should return 1 column, containing 'X'

Actual:

kibi.bat immediately exits with no messages of any kind. Verbose logging is enabled.

Steps to reproduce the problem

I created an Oracle datasource:

jdbc:oracle:thin:@d1tvt1d4.vci.att.com:1522:D1TVT libraries path : c:\jdemmon\elastic driver name oracle.jdbc.OracleDriver libraries : ojdbc6.jar

kibi.yml:

kibi_core: load_jdbc: true datasource_encryption_algorithm: 'AES-GCM' datasource_encryption_key: 'iSxvZRYisyUW33FreTBSyJJ34KpEquWznUPDvn+ka14=' datasource_cache_size: 501 default_dashboard_id: Articles

other observations:

The above driver config works good using other tools. When deliberately misspelling the jar name, Kibi does not report any error that that jar is missing. I can find no way to verify if Datasource is working. There is no button to test the connection, as found on most other tools. Kibi log never reports datasource errors, nor does it confirm if datasources are valid.

jd8935 commented 8 years ago

Update: I was able to get some log info from Kibi. The following is output by kibi.bat between executing "select x from dual" and when kibi.bat exits. No explanation for why Kibi exits.

qtest2 is the Oracle query.

{"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["debug","kibi_proxy","translated elasticsearch query"],"pid":11784,"message":"\n-------------------------\n{\"docs\":[{\"_index\":\".kibi\",\"_type\":\"query\",\"_id\":\"qtest2\"}]}\n\n-------------------------"} {"type":"response","@timestamp":"2016-09-29T18:01:03+00:00","tags":[],"pid":11784,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_mget?timeout=0&ignoreunavailable=true&preference=1475171966603","method":"post","headers":{"host":"localhost:5606","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0","accept":"application/json, text/plain, /_","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","kbn-version":"4.5.3","referer":"http://localhost:5606/app/kibana","content-length":"60","connection":"keep-alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5606/app/kibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"POST /elasticsearch/_mget?timeout=0&ignoreunavailable=true&preference=1475171966603 200 5ms - 9.0B"} {"type":"response","@timestamp":"2016-09-29T18:01:03+00:00","tags":[],"pid":11784,"method":"get","statusCode":200,"req":{"url":"/clearCache","method":"get","headers":{"host":"localhost:5606","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0","accept":"application/json, text/plain, /_","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"4.5.3","referer":"http://localhost:5606/app/kibana","connection":"keep-alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5606/app/kibana"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /clearCache 200 4ms - 9.0B"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading 10 queries into memory:"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Companies-in-the-same-domain]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Companies-\"With-competitors\"]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Top-investors-by-count-of-investments]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [qtest2]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Top-investors-of-a-given-type-by-count-of-investments]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Top-investors-from-given-country-by-investment-count]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Top-50-companies-(HR-count)]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Company-Info]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Top-500-companies-(HR-count)]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["info","query_engine"],"pid":11784,"message":"Reloading [Company-Competitors]"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["debug","kibi_proxy","raw kibi query"],"pid":11784,"message":"\n-------------------------\n/elasticsearch/.kibi/template/_search?size=100\n{\"query\":{\"match_all\":{}}}\n-------------------------"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["debug","kibi_proxy","translated elasticsearch query"],"pid":11784,"message":"\n-------------------------\n{\"query\":{\"match_all\":{}}}\n\n-------------------------"} {"type":"log","@timestamp":"2016-09-29T18:01:03+00:00","tags":["debug","kibi_proxy","raw kibi query"],"pid":11784,"message":"\n-------------------------\n/elasticsearch/.kibi/datasource/_search?size=100\n{\"query\":{\"match_all\":{}}}\n-------------------------"}

scampi commented 8 years ago

This was tested in another issue https://github.com/sirensolutions/kibi/issues/56 but I see you have the latest release. We'll have a look at what goes wrong. We are in the middle of putting integration tests so hopefully this kind of error will be prevented in the future.

Regarding your other observations, this is a good idea.

scampi commented 7 years ago

should be fine since 4.5.4, though configuration validation is not there yet. Please reopen if you have an issue with the latest release that is 4.6.4