Question regarding createSearchProperties

[danbev]

We are using this plugin as port of origin-aggregated-logging and hitting an issue which I think is related to createSearchProperties and the content it adds to the .kibana/search index. What we are seeing that it is not possible to save searches in Kibana. Trying to do so gives the following error message:

Discover: Request to Elasticsearch failed: "MapperParsingException[object mapping for [search] tried to parse field [title] as object, but got EOF, has a concrete value been provided to it?]"

Error: Request to Elasticsearch failed: "MapperParsingException[object mapping for [search] tried to parse field [title] as object, but got EOF, has a concrete value been provided to it?]"
    at https://kibana.local.feedhenry.io/index.js?_b=7562:88457:15
    at wrappedErrback (https://kibana.local.feedhenry.io/index.js?_b=7562:20902:78)
    at wrappedErrback (https://kibana.local.feedhenry.io/index.js?_b=7562:20902:78)
    at https://kibana.local.feedhenry.io/index.js?_b=7562:21035:76
    at Scope.$eval (https://kibana.local.feedhenry.io/index.js?_b=7562:22022:28)
    at Scope.$digest (https://kibana.local.feedhenry.io/index.js?_b=7562:21834:31)
    at Scope.$apply (https://kibana.local.feedhenry.io/index.js?_b=7562:22126:24)
    at done (https://kibana.local.feedhenry.io/index.js?_b=7562:17661:45)
    at completeRequest (https://kibana.local.feedhenry.io/index.js?_b=7562:17875:7)
    at XMLHttpRequest.xhr.onreadystatechange (https://kibana.local.feedhenry.io/index.js?_b=7562:17814:11)

After enabling logging we can see the following in the ElasticSearch log:

[2016-02-15 12:00:46,069][DEBUG][io.fabric8.elasticsearch.plugin.kibana.KibanaSeed] CREATE: '.kibana.a94a8fe5ccb19ba61c4c0873d391e987982fbbd3/search/properties' source: '{"title":{"type":"string"},"description":{"type":"string"},"hits":{"type":"integer"},"columns":{"type":"string"},"sort":{"type":"string"},"version":{"type":"in
teger"},"kibanaSavedObjectMeta":{"properties":{"searchSourceJSON":{"type":"string"}}}}'

And shortly afterwards this log entry:

[2016-02-15 12:00:46,095][DEBUG][cluster.metadata         ] [Astronomer] [.kibana.a94a8fe5ccb19ba61c4c0873d391e987982fbbd3] update_mapping [search] (dynamic) with source [{"search":{"properties":{"columns":{"properties":{"type":{"type":"string"}}},"description":{"properties":{"type":{"type":"string"}}},"hits":{"properties":{"ty
pe":{"type":"string"}}},"kibanaSavedObjectMeta":{"properties":{"properties":{"properties":{"searchSourceJSON":{"properties":{"type":{"type":"string"}}}}}}},"sort":{"properties":{"type":{"type":"string"}}},"title":{"properties":{"type":{"type":"string"}}},"version":{"properties":{"type":{"type":"string"}}}}}}]

I'm not very familiar with ElasticSearch or this plugin, but from looking at the code for createSearchProperties, it look like it is inserting a document into the .kibana index with a type of search, and the id is properties. My understanding is that the search type is used by Kibana to store searches. The content inserted by createSearchProperties looks more like a mapping and not a search. What I think is happening is that ElasticSearch is dynamically updating the search type mapping with the fields from this data. Later when Kibana tries to save a search the mapping is does not match the search data that Kibana is trying to store.

If I comment out the call to createSearchProperties I'm able to save searches in Kibana and things seem to work fine.

Basically I'm wondering what createSearchProperties is for?

Thanks,

[jcantrill]

@ewolinetz I'm assigning to you as I think you worked on the seeding logic for Kibana. @danbev part of the logic this plugin accomplishes is to rewrite requests and responses from Kibana. This version of Kibana has no understanding of multi-users which is a feature we are trying to support with OpenShift Origin aggregated logging.

[ewolinetz]

@danbev we would expect ES to update its mappings when data is pushed to it. If I remember correctly, we originally had createSearchProperties to mimic Kibana's initial calls when connecting to ES and one action was to populate this /search/properties type/id. Out of curiosity, what version of Kibana are you using, is it also 4.1.2?

If everything is able to start up correctly without the createSearchProperties call, we should probably remove it. I'll try to recreate what you're seeing and confirm that removing this call resolves the issue and causes no negative impacts otherwise.

[danbev]

Thanks for your quick replies!

Out of curiosity, what version of Kibana are you using, is it also 4.1.2?

I've looked into this and if I look at the version specified in package.json on the running container the version is:

$ oc rsh logging-kibana-1-2ie7s
bash-4.2$ cat /usr/share/kibana/src/package.json |grep version
  "version": "4.1.2",

Inspecting Settings -> About in the Kibana console shows:

Version 4.1.2
Build   7562
Commit SHA  34af16e

Looking at the Dockerfile is says 4.1.1. I didlook into this previously as I though it might have been the issue but since the version on the running container seems to be 4.1.2 I did pursue that any further.

Not sure if it matters or not but this is the OpenShift version we are running:

[vagrant@local ~]$ openshift version
openshift v3.1.0.4-5-gebe80f5
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

[danbev]

@jcantrill @ewolinetz I added a PR just to show the update that we have been using the last couple of days. I've been saving searches and using them in dashboards, exporting them and a colleague imported them successfully (also using the same changes). Let me know if there is anything specific you'd like me to try out or test.

Would it be possible to get a patch release for version 0.7 to maven central for this if you do decide this is a valid issue?

[jcantrill]

@danbev once merged its a matter of pushing the release button :)

[danbev]

@jcantrill excellent! :)

[ewolinetz]

@danbev sorry for the delay, I'll be able to get around to verifying this today. If I do find it to be valid I'll make the changes as part of our next release

[danbev]

@ewolinetz No problems, I really appreciate you taking a look at this so quickly.

If you do find it valid I would like to ask if it would be possible to patch the 0.7 version? The reason for asking is that 0.7 is the version we are currently using and our product is going to be released in the near future. If we had say a 0.7.1 version, we are going to see if is possible to get a tag/branch of origin-aggregated-logging which would use 0.7.1 in install.sh.

That way we would only have to update our docs using something like:

$ oc process -f https://raw.githubusercontent.com/openshift/origin-aggregated-logging/enterprise-1.1/deployment/deployer.yaml

Just wanted to give you some background about this and why I'm a little eager to get a fix in for this.

[ewolinetz]

@jcantrill thoughts on the above?

I was able to recreate this issue. However I'm also seeing that we aren't setting the default index right away for users, so I'm going to see if the proposed change will resolve what I'm seeing as well...

[ewolinetz]

The issue I was experiencing was due to a missing ES configuration; the removal of the search type seeding seems to cause no regression. I'll get this ready for our next release with a planned tech-debt task and discuss cutting a .7.1 release with this change as well

[ewolinetz]

@danbev would you be opposed to using a .9 release? To release a .7.1 we would have to change the current strategy for maintaining branches.

@jimmidyson thoughts?

[ewolinetz]

.9 proposed changes are in #22

[jcantrill]

@jimmidyson we have a CI process that magically deploys, etc. Can you advise how we might manually publish a 0.7.1 without needing to set up an additional CI chain

[jimmidyson]

Setting up a release build is simple enough - duplicate release build, change source branch (I hope you have a maintenance branch for this release?). I'm on vacation right now but @rawlingsj or @jstrachan might be able to help out with that hopefully?

[danbev]

would you be opposed to using a .9 release?

That should not be a problem. I'll try out #22 and just double check that are things work for us.

[danbev]

@ewolinetz @jcantrill I've tried out #22 and everything works as expected for us. I was able to save searches, import searches, visualisation, and dashboards saved with the previous version with out any issues :)

[jcantrill]

@ewolinetz We will need to branch if we are going to create a maintenance release. @jstrachan is there a way to manually publish over creating a CI chain. I don't see the value in maintaining a chain for this point release, but that would be fine if that's what you guys would prefer; its sits on the fabric8 infrastructure.

[ewolinetz]

@jcantrill I think at this time we won't need to do a maintenance release since @danbev is good with using a .9 release.

[ewolinetz]

This was fixed with #22

[danbev]

@jcantrill @ewolinetz Thanks guys!