Automated validation of Console autocompletion overrides

[cjcenizal]

Our current process for defining autocomplete suggestions for request bodies is to go through all of the generated files, read the corresponding docs, and create/update override files for any endpoints which support bodies.

Beyond the manual nature of this process, this is problematic because it means we have no automated means of being notified of changes made to APIs' bodies. Those changes won't result in new generated autocompletion files, so they won't be surfaced by this process. As a result, we risk accumulating outdated body specs.

As long as the breaking changes documentation is kept up-to-date, one manual solution is to just check these breaking changes (e.g. 7.0) and make sure the console override specs are updated accordingly.

CC @jloleysens

[jloleysens]

@cjcenizal Thanks a lot for creating this issue!

Just to clarify (and make sure I understand correctly) here is my understanding of the current process; we have a spec_to_console script which generates/updates JSON files. This only solves part of the autocompletion problem because on top of those we've created JS files for further completion preparation and overrides in JSON files. Neither of these is updated when the script is run and so manual updates of files is required. These JS and JSON we create and maintain patch the generated JSON so that we get improved autocompletion.

I have the following questions around the current process:

1. How frequently does the ES API change such that we need to create new overrides and how much time ~ does it take to update them by hand?

2. It looks like derived/generated ES specs should have update url_params/patterns/methods included:

   2.1. Is it possible for us to, at runtime, discover what ES version we are talking to?

   2.2. If 1.1. is true, can we load only that ES's set of overrides over "base" ES version spec? (this would mean potentially a lot more folders/files)

   2.3. Also if 1.1. is true perhaps we can beef up the spec_to_console script to automatically generate (and maintain) a set of "diffed" overrides for ES versions? This would remove a large chunk of manual updates (I don't see how we'll get around manually updating body spec :( ).

   2.4. How are we currently dealing with different ES versions' autocomplete if we are updating a single set of overrides as we go?

3. Is it possible for ES REST request spec to include specifications of body? Currently the body values look like "body": { "required": true, "description": "foobar" }. It would help a lot if we could get more information on the expected shape of the body for a given ES version. Without something like this I'm not sure we'll even be able to do automated validation. Alternatively, if we can get this it could mean we wouldn't have to manually touch any JSON overrides - the JS is still problematic.

Let me know if I've got anything wrong and of course if we can clarify any these points that would be great!

[cjcenizal]

Just to clarify (and make sure I understand correctly) here is my understanding of the current process...

This description is accurate. Nicely done!

1. How frequently does the ES API change such that we need to create new overrides and how much time ~ does it take to update them by hand?

I don't know, as I've never written any override files. I would ask Bill McConaghy as he's the one who's most familiar with this process. In terms of how frequently the ES API changes, I'm not sure how to quantify that. I would ask Gordon Brown if he's aware of any metrics we have for quantifying these changes and gauging their frequency.

2. It looks like derived/generated ES specs should have update url_params/patterns/methods included: 2.1. Is it possible for us to, at runtime, discover what ES version we are talking to?

Hitting GET / will return a response which contains version.number, which is the version of ES.

{
  "name" : "CJs-MacBook-Pro.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "zE72pjYeTc-exeXq1cF2wA",
  "version" : {
    "number" : "8.0.0-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5436581f696ce27513266090965afc824df76175",
    "build_date" : "2019-08-13T09:22:14.583005Z",
    "build_snapshot" : true,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "7.4.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

2.2. If 1.1. is true, can we load only that ES's set of overrides over "base" ES version spec? (this would mean potentially a lot more folders/files)

You wrote 1.1 but I assume you meant 2.1? The Elastic Stack is comprised of Elasticsearch, Kibana, Beats, and Logstash, and as a rule we only allow users to run these products together when they all share the same version number. So we can assume that a user using ES version 7.6 will be on Kibana version 7.6. So I think we only ever need to ship a single version's overrides, e.g. if you're on Kibana 7.6 you won't need access to autocompletions for version 6.5 of Elasticsearch. Does this answer your question?

2.3. Also if 1.1. is true perhaps we can beef up the spec_to_console script to automatically generate (and maintain) a set of "diffed" overrides for ES versions? This would remove a large chunk of manual updates (I don't see how we'll get around manually updating body spec :( ).

This sounds like an interesting idea. Do you want to flesh out a proposal?

2.4. How are we currently dealing with different ES versions' autocomplete if we are updating a single set of overrides as we go?

I think my comment about only supporting a single version of ES with each version of Kibana addresses this.

3. Is it possible for ES REST request spec to include specifications of body? Currently the body values look like "body": { "required": true, "description": "foobar" }. It would help a lot if we could get more information on the expected shape of the body for a given ES version. Without something like this I'm not sure we'll even be able to do automated validation. Alternatively, if we can get this it could mean we wouldn't have to manually touch any JSON overrides - the JS is still problematic.

I chatted with Jason Tedor about this over Slack. He feels like this is very high hanging fruit. The ES backend engineers have thought about having a deeper specification for many years but no one has come up with a viable approach. Part of the problem is the design of some of the ES APIs don’t lend themselves to be specified well, so they'd probably need to implement a new form of writing their APIs in order to enable more complete specification. Everyone agrees on the high value of a deeper spec but any solution they come up with would have a long dev timeline.

[jloleysens]

You wrote 1.1 but I assume you meant 2.1? ...

Yep I meant 2.1. 🙂

..for version 6.5 of Elasticsearch. Does this answer your question?

Yes thanks! And makes things somewhat simpler.

I don't know, as I've never written any override files. I would ask Bill McConaghy as he's the one who's most familiar with this process...

Ok, I'll check with Bill. I guess what I was trying to figure out is how much of our time does doing this by hand consume and whether it's worth putting in time to automate. It does seem like a pain to create/validate by hand (and is error prone).

This sounds like an interesting idea. Do you want to flesh out a proposal?

I guess this was under the assumption that Kibana and ES can be run against different versions of one another - given that this is not the case I don't think it would be worth doing auto generated diffs like what I was thinking.

I'd hoped that the JSON overrides we maintain only give information about body autocompletion. But this looks to not be the case (we're overriding information like patterns and methods for our autocomplete purposes - for instance, in indices.get_template.json override we override patterns so that we can match the URL pattern to the correct autocomplete set). I guess overall this hems in what we can automate i.t.o. validation quite a bit.

Were you thinking we could investigate some more unconventional means (scraping the docs site 😛ouchie). Alternatively, we could write script that runs through our JSON, creates requests against an actual ES instance and see what works/breaks. Would this be along the lines of what you were thinking?

[cjcenizal]

Alternatively, we could write script that runs through our JSON, creates requests against an actual ES instance and see what works/breaks

I hadn't thought of this, but this type of automated testing sounds like a reasonable solution. The Elasticsearch codebase already has these types of tests for their APIs of course, but that doesn't help us -- we need a way to verify our own assumptions about the ES APIs and your proposal would be one way of doing that.

[sebelga]

Alternatively, we could write script that runs through our JSON, creates requests against an actual ES instance and see what works/breaks

This is what I had in mind for the mappings definitions. Adding some contract tests. https://martinfowler.com/bliki/ContractTest.html

[cuff-links]

Alternatively, we could write script that runs through our JSON, creates requests against an actual ES instance and see what works/breaks

This is what I had in mind for the mappings definitions. Adding some contract tests. https://martinfowler.com/bliki/ContractTest.html

@cjcenizal @sebelga @jloleysens This seems like a pretty crackerjack idea. I also this this could pair well with the work @jloleysens did in https://github.com/elastic/kibana/pull/87331. We can use the contract tests to verify the back end (the overrides and generated json) and then perhaps do randomized spot checks via the front-end to ensure that the autocomplete works on the front end. Thoughts?

[yuliacech]

Related to https://github.com/elastic/kibana/issues/153451

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)