search

senacor / elasticsearch-evolution

A library to migrate elasticsearch mappings. Inspired by flyway.
Apache License 2.0
75 stars 13 forks source link
elasticsearch elasticsearch-evolution java migration spring-boot

readme

Elasticsearch-Evolution

A library to migrate Elasticsearch mappings. Inspired by flyway.

License LICENSE 996.icu Maven Central Javadocs Github build codebeat badge Coverage Status Lines of code Libraries.io dependency status for GitHub repo

1 Evolve your Elasticsearch mapping easily and reliable across all your instances

Elasticsearch-Evolution executes versioned migration scripts reliable and persists the execution state in an internal Elasticsearch/Opensearch index. Successful executed migration scripts will not be executed again!

2 Features

Compatibility Spring Boot Elasticsearch Opensearch
elasticsearch-evolution >= 0.6.0 3.x 7.5.x - 8.13.x 1.x - 2.x
elasticsearch-evolution >= 0.4.2 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1, 3.2 7.5.x - 8.13.x 1.x - 2.x
elasticsearch-evolution >= 0.4.0 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 7.5.x - 8.6.x 1.x - 2.x
elasticsearch-evolution 0.3.x 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7 7.5.x - 7.17.x
elasticsearch-evolution 0.2.x 1.5, 2.0, 2.1, 2.2, 2.3, 2.4 7.0.x - 7.4.x, 6.8.x

3 Quickstart

3.1 Quickstart with Spring-Boot starter

First add the latest version of Elasticsearch-Evolution spring boot starter as a dependency in your maven pom.xml:

<dependency>
    <groupId>com.senacor.elasticsearch.evolution</groupId>
    <artifactId>spring-boot-starter-elasticsearch-evolution</artifactId>
    <version>0.6.0</version>
</dependency>

Elasticsearch-Evolution uses internally Elastics RestClient and requires at minimum version 7.5.2. Spring boot could use an older version, depending on your Spring Boot version, so update it in your pom.xml:

<properties>
    <elasticsearch.version>7.5.2</elasticsearch.version>
</properties>

Place your migration scripts in your application classpath at es/migration

That's it. Elasticsearch-Evolution runs at application startup and expects your Elasticsearch/Opensearch at http://localhost:9200

3.2 Quickstart with core library

First add the latest version of Elasticsearch-Evolution core as a dependency:

<dependency>
    <groupId>com.senacor.elasticsearch.evolution</groupId>
    <artifactId>elasticsearch-evolution-core</artifactId>
    <version>0.6.0</version>
</dependency>

Place your migration scripts in your application classpath at es/migration

Create a ElasticsearchEvolution instance and execute the migration.

// first create a Elastic RestClient
RestClient restClient = RestClient.builder(HttpHost.create("http://localhost:9200")).build();
// then create a ElasticsearchEvolution configuration and create a instance of ElasticsearchEvolution with that configuration
ElasticsearchEvolution elasticsearchEvolution = ElasticsearchEvolution.configure()
        .load(restClient);
// execute the migration
elasticsearchEvolution.migrate();

4 Migration script format

4.1 Migration script file content

A Elasticsearch-Evolutions migration script represents just a rest call. Here is an Example:

PUT _template/my_template
Content-Type: application/json

{
  "index_patterns": [
    "my_index_*"
  ],
  "order": 1,
  "version": 1,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
    "version": {
      "type": "keyword",
      "ignore_above": 20,
      "similarity": "boolean"
    },
    "locked": {
      "type": "boolean"
    }
    }
  }
}

The first line defines the HTTP method PUT and the relative path to the Elasticsearch/Opensearch endpoint _template/my_template to create a new mapping template. Followed by a HTTP Header Content-Type: application/json. After a blank line the HTTP body is defined.

The pattern is strongly oriented in ordinary HTTP requests and consist of 4 parts:

  1. The HTTP method (required). Supported HTTP methods are GET, HEAD, POST, PUT, DELETE, OPTIONS and PATCH. The First non-comment line must always start with a HTTP method.
  2. The path to the Elasticsearch/Opensearch endpoint to call (required). The path is separated by a blank from the HTTP method. You can provide any query parameters like in a ordinary browser like this /my_index_1/_doc/1?refresh=true&op_type=create
  3. HTTP Header(s) (optional). All non-comment lines after the HTTP method line will be interpreted as HTTP headers. Header name and content are separated by :.
  4. HTTP Body (optional). The HTTP Body is separated by a blank line and can contain any content you want to sent to Elasticsearch/Opensearch.

4.1.1 Comments

Elasticsearch-Evolution supports line-comments in its migration scripts. Every line starting with # or // will be interpreted as a comment-line. Comment-lines are not send to Elasticsearch/Opensearch, they will be filtered by Elasticsearch-Evolution.

4.1.2 Placeholders

Elasticsearch-Evolution supports named placeholder substitution. Placeholders are marked in your migration script like this: ${my-placeholder}

4.2 Migration script file name

Here is an example filename: V1.0__my-description.http

The filename has to follow a pattern:

Elasticsearch-Evolution uses the version for ordering your scripts and enforces strict ordered execution of your scripts, by default. Out-of-Order execution is supported, but disabled by default. Elasticsearch-Evolution interprets the version parts as Integers, so each version part must be between 1 (inclusive) and 2,147,483,647 (inclusive).

Here is an example which indicates the ordering: 1.0.1 < 1.1 < 1.2.1 < (2.0.0 == 2). In this example version 1.0.1 is the smallest version and is executed first, after that version 1.1, 1.2.1 and in the end 2. 2 is the same as 2.0 or 2.0.0 - so trailing zeros will be trimmed.

NOTE: Versions with major version 0 are reserved for internal usage, so the smallest version you can define is 1

5 Configuration options

Elasticsearch-Evolution can be configured to your needs:

5.1 Spring Boot

You can set the above configurations via Spring Boots default configuration way. Just use the prefix spring.elasticsearch.evolution. Here is an example application.properties:

spring.elasticsearch.evolution.locations[0]=classpath:es/migration
spring.elasticsearch.evolution.locations[1]=classpath:es/more_migration_scripts
spring.elasticsearch.evolution.placeholderReplacement=true
spring.elasticsearch.evolution.placeholders.indexname=myIndexReplacement
spring.elasticsearch.evolution.placeholders.docType=_doc
spring.elasticsearch.evolution.placeholders.foo=bar
spring.elasticsearch.evolution.historyIndex=es_evolution

5.1.1 Elasticsearch AutoConfiguration (since spring boot 2.1)

Since spring boot 2.1 AutoConfiguration for Elasticsearchs REST client is provided (see org.springframework.boot.autoconfigure.elasticsearch.ElasticsearchRestClientAutoConfiguration). You can configure the RestClient, required for Elasticsearch-Evolution, just like that in your application.properties:

5.1.1.1 spring boot 2.6+
spring.elasticsearch.uris[0]=https://example.com:9200
spring.elasticsearch.username=my-user-name
spring.elasticsearch.password=my-secret-pw

5.1.2 Customize Elasticsearch-Evolutions AutoConfiguration

5.1.2.1 Custom RestClient

Elasticsearch-Evolutions just needs a RestClient as spring bean. If you don't have spring boot 2.1 or later or you need a special RestClient configuration e.g. to accept self signed certificates or disable hostname validation, you can provide a custom RestClient like this: 

@Bean
public RestClient restClient() {
    RestClientBuilder builder = RestClient.builder(HttpHost.create("https://localhost:9200"))
            .setHttpClientConfigCallback(httpClientBuilder -> {
                        CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("my-user-name", "my-secret-pw"));
                        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                        try {
                            httpClientBuilder
                                    .setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build())
                                    .setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
                        } catch (GeneralSecurityException e) {
                            throw new IllegalStateException("could not configure http client to accept all certificates", e);
                        }
                        return httpClientBuilder;
                    }
            );
    return builder.build();
}
5.1.2.2 Custom ElasticsearchEvolutionInitializer

Maybe you want to provide a customised Initializer for Elasticsearch-Evolution e.g with another order:

@Bean
public ElasticsearchEvolutionInitializer customElasticsearchEvolutionInitializer(ElasticsearchEvolution elasticsearchEvolution) {
    return new ElasticsearchEvolutionInitializer(elasticsearchEvolution) {
        @Override
        public int getOrder() {
            return Ordered.LOWEST_PRECEDENCE;
        }
    };
}

5.2 core library

You can set the above configurations via the ElasticsearchEvolutionConfig fluent builder like this:

ElasticsearchEvolution.configure()
    .setLocations(Collections.singletonList("classpath:es/migration"))
    .setPlaceholderReplacement(true)
    .setPlaceholders(Collections.singletonMap("indexname", "myIndexReplacement"))
    .setHistoryIndex("es_evolution");

6 changelog

v0.6.1-SNAPSHOT

v0.6.0

v0.5.2

v0.5.1

v0.5.0

v0.4.3

v0.4.2

v0.4.1

v0.4.0

v0.3.2

v0.3.1

v0.3.0

v0.2.1

v0.2.0

v0.1.3

v0.1.2

v0.1.1

v0.1.0