Announcement: Future directions (updated on 2022-07-22)

[olivere]

Writing this in an issue is probably not the right way, but at least it's near the artifact that matters most to me—the code.

Disclaimer: Please consider this as my very own personal view. I'm a complete outsider, have no internal information and no investments in either Amazon or Elastic. I'm just a dev with an opinion.

I've started this project in 2012. First commit was on December 6, 2012. That was the same year that Searchworkings Global, the predecessor of the company we now know as Elastic, was founded, and way before Elasticsearch 1.0.0 was a thing (that was Feb 12, 2014). Since then this library supported all versions of Elasticsearch with the help of many contributors, but also because I ❤️ Elasticsearch and Go and I was using it extensively in my projects and at work. I've probably invested thousands of hours into building and supporting it. I've tried very hard to not break things. Elasticsearch has never disappointed me (and it made us faster—literally ;-)). My deepest respect and thanks to all the developers that made it such a success (and good companion). But in the end, I think, I wrote the library for myself—because I'm a techie at heart.

Over the recent months though, a strange controversy arose between Elastic and Amazon over the use of Elasticsearch, the technology.

First of all I understand both sides. But still I'm disappointed about the end result for us developers. On the one hand, there's Amazon offering Elasticsearch under their brand, now forking and rebranding it under the term OpenSearch. As a developer, I think Amazon shouldn't have done it the way they did. But it's Amazon. You and me have to decide if their way of doing business is having a positive impact on all of us in the long run. On the other hand, in a move to protect their stakes, Elastic introduced a new license and made technical changes to their API clients. Although it's quite easy to work around it, again—as a developer—I can't see this heading into the right direction.

But let's skip the legal issues. I've been using the official client for Go in my last project. It works well, it's code base is clean, and the GitHub repository is well maintained. So no offences whatsoever at the developers working on it. You're doing a good job. But. Maybe I'm just biased. But I don't like working with it. The most critical issue for me is it's lack of having a nice way of building queries, aggregates, and parsing responses. My codebase is filled with map[string]interface{}. It just doesn't feel right to me. I know the good people at Elastic know about this, and one day, they'll offer an API for requests and responses, so that one can generate these. But right now, I miss that. I miss it a lot.

So a lot of words for saying that I decided to continue working on the library, v7 and beyond, even if it's just for me. I want it to be usable and enjoyable to work with, for developers like me. I want it to keep out of any gorilla fights. And I sincerely hope that's possible. I want it to be my library of choice for working with Elasticsearch, the technology, regardless of the context in which you're using it.

EDIT: This announcement relates to my statement in this comment.

UPDATE 2022-07-13:

I've had a glimpse into the future direction of the official Elasticsearch client for Go. The client will very soon have a "typed API" which you can use to build your requests (and parse responses, soon) just like you can with my little client here. This resolves most (if not all) of my criticism of the client I stated above. And it actually looks and feels like a native Go client (see the integration tests). This is how you can create an index with a mapping, for example:

res, err := es.Indices.Create(indexName).
    Request(&create.Request{
        Mappings: &types.TypeMapping{
            Properties: map[types.PropertyName]types.Property{
                "price": types.NewIntegerNumberPropertyBuilder().Build(),
                "name":  types.NewKeywordPropertyBuilder().Build(),
            },
        },
    }).
    Do(context.Background())
if err != nil {
    panic(fmt.Sprintf("error creating index test-index: %v", err))
}
defer res.Body.Close()

Even better: All of the builders have been created from a specification, which is the way to go for such a huge surface area that is the Elasticsearch API with its hundreds of endpoints. A small caveat is that the client is supposed to be for 8.x or later. But who knows, maybe we'll get a 7.x version at one point. If not, this actually makes the switch from Elastic v7 to the official Elasticsearch client for Go 8.x the perfect time to rewrite your code.

I salute Elastic for taking the extra effort of working on clients that look and feel like they were written by an actual developer for the actual platform. The Go client looks and feels like Go, even the generated code. The .NET client, which I've had the pleasure to work with once, is just amazing. This is in stark contrast to what you see nowadays with many companies, just throwing out their OpenAPI generated codebase over us developers and call it a day. So thanks for that. It is also a stark contrast to my perceived and subjective view of the OpenSearch community. Yes, they have documentation. And yes, it's Amazon backing it. And yes, there's some kind of progress. But being a developer and looking at the overall experience, I'd choose Elasticsearch over OpenSearch every single time. Especially for me as a Go developer, it looks like Amazon is just not very interested.

Finally a word about this repository. I'm sorry about all the people still waiting for an Elastic v8 version, waiting for their PRs to be merged and issues to be answered. I had a hard time personally over the last two years, and when I had to decide whether working on Elastic or focusing on my health is the top priority, I always decide for the latter. Now that I've seen the future of the Go client for Elasticsearch, I couldn't be happier, and I'll be the first one to switch and use the typed API client. So, unfortunately, and sad to some extend, but I have to say: Elastic v8 won't happen. I will polish v7 if I have the time and energy, but v8 won't happen. I'm sorry.

Thanks to the people working on the official Go client and making it work for the community. And special thanks to @karmi, @Anaethelion and @philkra for your trust and proactive approach.

[shmilyoo]

Thanks to the developer of this library. I have used this library in depth on my production projects and it works great. Also, speaking of the official es go client library, this one has twice as many stars as it does, so what else needs to be said?

[tobinski]

Thanks for continue to work on it. I use the lib in production and I'm very happy with it

[aslng]

Moving forward, will this golang client be working only with elastic.co ES server or also working with AWS opensearch server?

[graysonchao]

@aslng we are currently using it in production with AWS OpenSearch and seeing no issues. Our use case is a small subset of the overall Elastic API that only deals with the core search and get APIs, so your mileage may vary.

[veqryn]

An idea is to support both future Elasticsearch and also future Opensearch releases. This shouldn't be too hard, because Opensearch is just a fork of Elasticsearch 7.10.x, and they won't be doing any breaking changes for a very long time for fear of losing their customer base. Elasticsearch could continue with the v7, v8, etc, releases, while Opensearch could do a v1-os or v1-opensearch release.

[chaudharisuresh997]

Really appreciate your work.I am also not happy with elastic official goclient and will not be using that will happily move to java elastic search client

[epelc]

I just took a look at the official client since contemplating a v7 -> v8 upgrade. The first thing I noticed besides the major reliance on maps is them promoting using string concatenation of potential user data in the indexing example.

Sadly doesn't give me a lot of hope for the rest of the client. At least submitted a pr to use proper json marshaling https://github.com/elastic/go-elasticsearch/pull/493

On the bright side v7 does have quite a while left for support: https://www.elastic.co/support/eol

The request/response structs in this driver are a killer feature especially for something as complicated as elasticsearch.

[olivere]

@epelc I agree wrt the request/response remark.

But there is hope on the horizon.

[epelc]

That’s very good news! Hopefully it will improve

Something similar happened when mongodb made their own official driver. It wasn’t as good as mgo for a while but at some point it really surpassed it.

[pruidong]

Really appreciate your work.I am also not happy with elastic official goclient and will not be using that will happily move to java elastic search client

If you want to use Java, it is recommended to use Spring data elasticsearch. It is better.

[bmfmsd]

Could you please remove the Deprecated statement from the home page? We've been using this software since 2016, and now we'll have to replace it with the official 7.x version because of the Deprecated statement, but as you said, the official 7.x version doesn't work well, and they took your advice. However, only the 8.x version is incorporated. Please help me. Thank you very much.