Generate type-safe builders compatible with Kotlin DSL functionality

[MT-Jacobs]

🚀 Feature Proposal

The Elasticsearch Specification could be used to generate a Kotlin DSL that can be used with the Elasticsearch Java client's builders.

Alternatively, one could do the same thing directly in Kotlin using fully Kotlin-based builders.

Motivation

I'm writing a lot of Elasticsearch client code in Kotlin and was getting tired of how hard it was to read with complex queries, even with the builder -

    fun getMyValues() : Deferred<SearchResponse<MyDoc>> =
        client.search( { _0 ->
            _0.index("my_index")
                .size(0)
                .query { _1 ->
                    _1.bool { _2 ->
                        _2.filter { _3 ->
                            _3.term { _4 ->
                                _4.field("origin")
                                    .value { _5 ->
                                        _5.stringValue("Skywalker Ranch")
                                    }
                            }
                        }.filter { _3 ->
                            _3.term { _4 ->
                                _4.field("destination")
                                    .value { _5 ->
                                        _5.stringValue("Narnia")
                                    }
                            }
                        }.filter { _3 ->
                            _3.range { _4 ->
                                _4.field("startDate")
                                    .gte(JsonData.of("now-14d/d"))
                                    .lte(JsonData.of("now/d"))
                            }
                        }
                    }
                }.aggregations("avgCost") { _2 ->
                    _2.avg { _3 ->
                        _3.field("cost")
                    }
                }.aggregations("uniqueUsers") { _2 ->
                    _2.cardinality { _3 ->
                        _3.field("userId")
                    }
                }
        }, MyDoc::class.java).asDeferred()

Kotlin is great at this kind of thing, so I gave a try at manually implementing it and started to wonder if it could be automated. The easier it is to describe a query in Elasticsearch, the more I can do with the technology.

Example

The DSL could vastly improve ease of use in Kotlin projects. Here's one sample of some easy to work with DSL code that I successfully implemented as a proof of concept:

val searchResponse: CompletableFuture<SearchResponse<MyDoc> = client.search {
            index("my_index")
            size(0)
            query {
                bool {
                    filter {
                        term("origin", "Skywalker Ranch")
                        term("destination", "Narnia")
                        range("startDate") {
                            gte("now-14d/d")
                            lt("now/d")
                        }
                    }
                }
            }
            aggregations {
                "date_ranges".dateRange("startDate") {
                    range("1Wk") {
                        from("now-7d/d")
                        to("now/d")
                    }
                    range("2Wk") {
                        from("now-14d/d")
                        to("now-7d/d")
                    }
                    aggregations {
                        "avgCost".avg("cost")
                        "uniqueUsers".cardinality("userId")
                    }
                }
            }
        }

We can use an inline reified function for searches to 1) avoid having to indicate the class twice and 2) invoke the SearchKt object and build our SearchRequest.

inline fun <reified T> ElasticsearchAsyncClient.search(setup: SearchKt.() -> Unit): CompletableFuture<SearchResponse<T>> =
    this.search(SearchKt().apply(setup).build(), T::class.java)

All one needs to do to make the above functionality possible is for to auto-generate various Kotlin builders on top of the existing Java-based ones (or, alternatively, set them up without Java at all) - something that might look like this:

// See https://kotlinlang.org/docs/type-safe-builders.html#scope-control-dslmarker
@DslMarker
annotation class ElasticMarker

// Incomplete list of functionality that SearchRequest.Builder supports
@ElasticMarker
class SearchKt {
    private val builder = SearchRequest.Builder()

    fun build(): SearchRequest = builder.build()

    fun index(value: String, vararg values: String) {
        builder.index(value, *values)
    }
    fun size(value: Int) {
        builder.size(value)
    }
    fun query(setup: QueryKt.() -> Unit) {
        builder.query(QueryKt().apply(setup).build())
    }
    fun aggregations(setup: AggregationsKt.() -> Unit) {
        builder.aggregations(AggregationsKt().apply(setup).build())
    }
}

[MT-Jacobs]

Per https://github.com/elastic/elasticsearch-java/blob/main/CONTRIBUTING.md#project-structure:

The co.elastic.clients.elasticsearch package and its children are all entirely generated, and the generator is not part of this repository. Because of this, PRs will not work for this part of the code. If you want to suggest changes to the generated code, open an issue describing how the code should look like, so that we can discuss on updating the generator.

The code I want a DSL for lives in that package, and I'm assuming the generator is closed source(?) so this doesn't appear to be something I can build myself.

If someone knows where the generator is though, I'd love to take a look at it.

[MT-Jacobs]

Function literals with receiver make many parts of the DSL simple and elegant to construct:

fun SearchRequest.Builder.queryKt(commands: Query.Builder.() -> Unit): SearchRequest.Builder =
    query(Query.Builder().apply(commands).build())

fun Query.Builder.boolKt(commands: BoolQuery.Builder.() -> Unit): ObjectBuilder<Query> =
    bool(BoolQuery.Builder().apply(commands).build())

The problem is when you get to portions of functionality that use lists of maps such as SearchRequest.Builder#aggregations and BoolQuery.Builder#filter. That will require more complex code.

Again, I'm happy to help make this happen - it would make it way easier for my team to dive deep and iterate on Elasticsearch-driven code - I'm just not sure where to start with regards to the generator.

[MT-Jacobs]

For what it's worth I still would very much be interested in contributing to something like this.