Add a query/field compatibility matrix

[markharwood]

We have a growing catalogue of query types and field types but not all queries are supported in all fields. Neither field docs nor query docs want to deal with the messy business of documenting compatibility - documentation for queries do not list the field types that support that query type and the documentation for fields do not list the supported queries.

A simple support matrix with a yes/no check in each cell would help fill in a lot of the detail for users. Ideally this document would be automatically tested or even generated by checking against the software itself - documentation can get out of synch quickly.

Beyond a simple yes/no assertion it may prove necessary for the support matrix to document some of the nuances of certain query settings or behaviours when used on certain fields e.g

• the wildcard query's rewrite parameter does not support values other than constant when used on wildcard fields.
• the keyword field supports term queries but, unlike other supporting fields, is case insensitive by default when the field uses a normalizer.
• the regexp query works on keyword fields but is slow when there are millions of unique values - consider using the wildcard field instead for faster regex searches

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[markharwood]

I have done some work to automate this and have produced the raw data for : 1) Several elasticsearch versions 2) All term-level query types 3) Many field types

A Python script runs all the required combinations and outputs reports on the matching behaviour.

The software and raw results are here . Checkout the readme.txt file for an overview. The results indicate if each combination of the above is supported or functioning correctly (some earlier versions had bugs and we still have known discrepancies).

I don't know what we want to do with the data - there's ~1000 cells in this matrix for just the 3 elasticsearch versions I've tested so far, before we consider adding all the "point" releases. It's far too much for a single HTML table but storing and analysing in elasticsearch/Kibana seems like an obvious way to allow slice and dice.

From poking around the data a high level summary is:

• I found no false negatives
• <= 7.10 had false positives on wildcard queries
• flattened field does not support regex or wildcard query types
• To this day (7.14 being current) we have false positives on normalized keyword fields which messes with query term case (unlike all other field types which don't). We live with this behaviour because changing it would probably break a lot of existing systems.
• I haven't yet tested query_string-parsed variants of the above query types (e.g. there were known bugs in the way Lucene query parser parsed wildcard queries).

[nik9000]

I think it'd be cool to have a test for each entry in the matrix. Just paranoia. I wonder if we can do them without hand generating them.

For what it's worth, its always bothered me that we don't document these together too. But we do try to make them fairly loosely coupled. Sort of. We don't try super hard.

[markharwood]

I think it'd be cool to have a test for each entry in the matrix. Just paranoia. I wonder if we can do them without hand generating them

Full test coverage would be good. The requirement differs a little compared to conventional tests because we want to document: 1) Bugs in older versions where there were no tests 2) Known and accepted shortcomings: a) unsupported combos (e.g. flattened field doesn't support wildcard). b) exceptions to rules e.g. term-level queries don't have case changed, unless on a keyword field with a normalizer.

[ywelsch]

I like the idea of turning this into a test suite (with particular attention on having as much coverage as possible).

The requirement differs a little compared to conventional tests because we want to document:

I think there's a good reason to make them conventional tests.

• (1) could be addressed by backporting tests to older branches which we normally won't backport stuff to (similar to how sometimes docs clarifications are backported further back), or if we test using REST layer, we could have such BWC compat matrix tests live in our current release branch (e.g. 7.x) and have the test run against older version nodes (similar to YML REST tests).
• re (2) I think that known and accepted shortcomings could be just as well treated as conventional tests. For (2a), for example, I would expect ES to throw a proper exception (providing information that this is an unsupported combo), which would also be worth testing.

[markharwood]

@ywelsch I think there may be several concerns with back-porting conventional tests:

• As far as I know we don't run tests on older versions e.g. 7.10
• Unless we want red builds we'd sometimes have to backport versions of tests modified with the wrong expectations e.g. the issue linked above your comment where 7.10 was doing the wrong thing. We'd need a 7.10 test that asserted that wrong behaviour (normally we only test for correct behaviour).
• These new test assertions in code wouldn't be a very end-user friendly form of behaviour documentation. Ideally the docs would allow users to quickly see that 7.10 wildcard queries were giving false positives when it comes to case sensitivity and that was fixed in 7.13.