Upgrade to Elasticsearch 5.x

[niemyjski]

We need to reindex all data and update the client for Elasticsearch 2.0 as things have been deprecated. We should compile a list of these as well as look into performance gains that could be achieved via a reindex.

• [x] Update Foundatio.Repositories
• [x] Update Indexes
• [x] Decrease the shard size to 4 or 5
• [x] Setup new cluster to test
• [x] Test upgrading and redindexing

Performance gain:

• Look into precomputed hashes for cardinality lookups
• Correctly map extended data numeric types to the right elastic numeric type (for dynamic mapped fields). https://github.com/exceptionless/Exceptionless/issues/173 which is related to https://github.com/elastic/elasticsearch-net/issues/1643

[niemyjski]

https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing

[niemyjski]

https://www.elastic.co/blog/performance-indexing-2-0

[geffzhang]

When plans to support Elasticsearch 2.x

[niemyjski]

We will be skipping 2.x completely and going straight to 5.x by using external reindexing support built into 5.x. We are starting work on this process this week so we can be ready for the 5.x release and reap the many rewards of 2.x+5.x (80% faster indexing, update by query and much more).

[niemyjski]

https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/

[geffzhang]

Support Elasticsearch 5.x, also supports Elasticsearch 2.x?

[niemyjski]

We will be skipping 2.x and going straight to 5.x via external reindexing.

[niemyjski]

We have an /elastic5 branch that we are working out of :)

[geffzhang]

elastic5 release ,This release introduces a number of breaking changes. https://www.elastic.co/blog/elasticsearch-5-0-0-released

[niemyjski]

Yep, a whole lot of them, we just got foundatio.repositories completely upgraded and are working on exceptionless, we almost have the mappings done but we are blocked until they release a NEST-5.0-rc2 client

[geffzhang]

https://www.nuget.org/packages/NEST/5.0.0-rc2 NEST-5.0-rc2 release

[niemyjski]

https://docs.microsoft.com/en-us/azure/guidance/guidance-elasticsearch-tuning-data-ingestion-performance

[niemyjski]

We need to work on reindexing next (dev will be upgraded to elastic 5 today or tomorrow).. The reindex will need to have an ingest pipeline which choices the right daily index.

[ejsmith]

App is looking good. Tons of hard work paying off. :)

[niemyjski]

https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html

[niemyjski]

I have exceptionless up and running on elastic 5 (https://dev-app.exceptionless.io/)

Feel free to play around with it. I only have it running on a single node (haven't even optimized that node)

I just did find an issue where I wasn't able to invite users / get users list for an org, will look into that very soon. Other than that I'm aware of no issues but I'm sure there probably are

[niemyjski]

https://www.youtube.com/watch?v=cI-3mEUW_F4 https://sematext.com/blog/2015/11/30/running-elasticsearch-clusters-on-docker/ https://github.com/pires/kubernetes-elasticsearch-cluster/blob/master/README.md

[niemyjski]

We've deployed this into production, we'll create a release soon once we have more data