Switch doc_values=true to be the default for non-analyzed string fields

[gibrown]

We recently started doing a lot of sorting by dates and integer fields on subsets of our 5 billion documents. We are mostly running queries that filter to a few thousand docs and then sorting that subset. Needing to load 5 billion dates into memory for this seemingly common use case make ES feel broken. :)

Heap memory management can be very painful and I think represents a difficult point for new users. Search engines are all about preprocessing data (indexing) so that querying can be fast. In my opinion doc_values accomplishes this better than the current default even if it can be slower in some cases.

FWIW, doc_values is MUCH faster in our use case using 1.3.4.

[bobrik]

@gibrown have you seen increased segment memory usage after switching to doc_values? I've seen 2-8x increase (from 32mb to 70-270mb) per index with 120m events.

[clintongormley]

The JVM heap size is practically limited to 32GB - above this and the JVM can no longer use compressed pointers (resulting in more memory usage for the same data), and garbage collections become slower.

By far the biggest user of the heap for most users is in-memory fielddata, used for sorting, aggregations and scripts. In-memory fielddata is slow to load, as it has to read the whole inverted index and uninvert it. If the fielddata cache fills up, old data is evicted causing heap churn and bad performance (as fielddata is reloaded and evicted again.)

Doc values provide the same function as in-memory fielddata, but the datastructure is written to disk at index time. This results in more disk usage and somewhat slower indexing and mergging (because there is more I/O). Aggregations and sorting are about 20% slower than they are with in-memory fielddata.

The advantages are:

• less heap usage and faster garbage collections
• no longer limited by the amount of fielddata that can fit into 32GB of heap - instead the file system caches can make use of all the available RAM
• fewer latency spikes caused by reloading a large segment into memory

Unfortunately, there is no way to back-fill these values without reindexing. The proposal is to make doc values the default for all fields except analyzed string fields (which are not supported by doc values anyway).

Some users will end up writing much more data than they need, but to be clear: this is a default setting which can be changed as appropriate. It is similar to the fact that we index term positions by default on all analyzed string fields, so as to make phrase queries work out of the box.

Before making this decision, we need to understand the full impact of doc-values-by-default, by running the following tests:

• ssd vs spinning disk
• merge throttling enabled/disabled
• non-string fields vs all not_analyzed fields
• metric, logging, wikipedia
• aggs
• sorting asc and desc
• multiple aggs and sorting combined

And measuring the following:

• Node stats after indexing (and always starting from zero)
• number of segments before optimize
• index size before/after optimize
• indexing rate (docs/s)
• query rate (docs/s)
• query rate during heavy indexing
• query rate during heavy indexing with eager loading
• query rate during heavy indexing with eager global ordinals
• logs for GC messages and index throttling messages

We need to do this at large scale with a billion documents, on nodes that are properly tuned (eg ES_HEAP, mlockall etc)

Two further proposals:

• Turn off fielddata loading for analyzed string fields by default. It is too easy to blow up memory usage by sorting or aggregating on an analyzed string field, and it is seldom what the user intended. If the user really does want to run aggregations on an analyzed string field, then it is an easy option to enable on an existing index.
• Default not_analyzed string fields to have ignore_above set to (eg) 255 characters, as it seldom makes sense to index or write doc values for such large terms

[bharvidixit]

@clintongormley what if i enable doc_values=true for the metadata _id field? Will it have some effect? "_id": { "store": true, "index": "not_analyzed", "doc_values": true, "path": "id" }

[jpountz]

@bharvidixit Actually we are also thinking about having doc values enabled on _uid too. This way, random sorting (which mostly merges a seed with a hash of the _uid to be reproducible) would not need to load fielddata on the _uid field (which takes a lot of memory all the time since this field is unique by definition). And it could also help have consistent pagination by tie-breaking on the _uid instead of the internal lucene doc ids (since they are not the same on all copies of a shard).

[gibrown]

Awesome, Thanks!

[bobrik]

:+1:

[jknewman3]

Is it possible to set eager loading, or something like that, for doc_values? It would be helpful for us to always have the most recently added data in cache. And, yeah, warmers could do it for us. Just wondering if eager applies only to JVM.

[rjernst]

@jknewman3 See discussions on #8693

[jknewman3]

Ah, that helps a lot. Thanks!

[bolee]

how to set doc_value=true, via api to set every field?

[clintongormley]

@bolee please ask questions like that in the forum: https://discuss.elastic.co/