Support for BigInteger and BigDecimal

[clintongormley]

Lucene now has sandbox support for BigInteger (LUCENE-7043), and hopefully BigDecimal will follow soon. We should look at what needs to be done to support them in Elasticsearch.

I propose adding big_integer and big_decimal types which have to be specified explicitly - they shouldn't be a type which can be detected by dynamic mapping.

Many languages don't support big int/decimal. Javascript will convert to floats or throw an exception if a number is out of range. This can be worked around by always rendering these numbers in JSON as strings. We can possibly accept known bigints/bigdecimals as numbers but there are a few places where this could be a problem:

• indexing a known big field (do we know ahead of time to parse a floating point as a BigDecimal?)
• dynamic mapping (a floating point number could have lost precision before the field is defined as big_decimal)
• ingest pipeline (ingest doesn't know about field mappings)

The above could be worked around by telling Jackson to parse floats and ints as BIG* (USE_BIG_DECIMAL_FOR_FLOATS and USE_BIG_INTEGER_FOR_INTS) but this may well generate a lot of garbage for what is an infrequent use case.

Alternatively, we could just say that Big* should always be passed in as strings if they are to maintain their precision.

[fredgalvao]

Fair enough. I forgot to take into account the client progression when re-evaluating the issue. Thanks @jpountz !

[insukcho]

Thanks for handling this case @jpountz . I will keep watching #32434 for further support of 64-bit unsigned integers.

[fredgalvao]

I'm risking being out of scope by stretching this, but I think we still have an issue on 6.3.0 with BigDecimal at least in one place:

I'm using the LLRC (to talk to the cluster) in conjunction with the server artifact org.elasticsearch:elasticsearch (to build/manipulate queries), and using BigDecimal params on a Script object to be used in a bucketSelector, and it fails with the following:

cannot write xcontent for unknown value of type class java.math.BigDecimal
java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.math.BigDecimal
    at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:755)
    at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:810)
    at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:792)
    at org.elasticsearch.common.xcontent.XContentBuilder.field(XContentBuilder.java:788)
    at org.elasticsearch.script.Script.toXContent(Script.java:663)
    at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:779)
    at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:772)
    at org.elasticsearch.common.xcontent.XContentBuilder.field(XContentBuilder.java:764)
    at org.elasticsearch.search.aggregations.pipeline.bucketselector.BucketSelectorPipelineAggregationBuilder.internalXContent(BucketSelectorPipelineAggregationBuilder.java:120)
    at org.elasticsearch.search.aggregations.pipeline.AbstractPipelineAggregationBuilder.toXContent(AbstractPipelineAggregationBuilder.java:130)
    at org.elasticsearch.common.Strings.toString(Strings.java:778)
    at org.elasticsearch.search.aggregations.PipelineAggregationBuilder.toString(PipelineAggregationBuilder.java:92)

@jpountz Am I out of scope? Should I not expect the builders on the elasticsearch artifact to be as type-agnostic as the rest of the LLRC? Should I open a new issue?

[jpountz]

@fredgalvao This is a different bug. I would expect us to fix it when addressing #32395.

[tpmccallum]

Just pinging this conversation to point to a specific issue which requests that Elasticsearch commences support for Ethereum blockchain data types. Namely uint256 (2 ^ 256 -1) https://github.com/elastic/elasticsearch/issues/38242