Support for ElasticSearch

[mbn18]

Hi,

Seems like there is no db version manager for ElasticSearch. I was wondering if it make sense to add ES support for shmig?

I guess client will be curl and the version will be kept in a document. Can it work and is it worth it?

[mbucc]

If you can make it work from the command-line, then it will work in shmig.

As to whether it is worth it, that depends on you. I would certainly accept such a pull request.

[mbn18]

@mbucc , Ill try to play with the code and see what can be done

[mbucc]

@mbn18 Cool. And I see there is a docker container for ElasticSearch so I should be able to spin up something locally to test against. This project needs that kind of test setup anyway ...

[mbn18]

@mbucc , few obstacles to consider

ElasticSearch use REST as interface. That make curl a valid client. The problem is that the migration is not one transaction. It is composed of X REST requests like PUT, POST, DELETE and etc.

Example for suggested migration file:

-- Migration: addWhatEver
-- Created at: 2016-08-06 01:07:22
-- ====  UP  ====

PUT /index_name
{
  "mappings": {
    "type_name": {
      "dynamic": "strict",
      "properties": {
        "fullName": {
          "type": "string",
        },
        "createdAt": {
          "type": "date",
          "format": "strict_date_time_no_millis"
        },
      }
    }
  }
}

POST /index_name/type_name/
{
  "fullName": "John Doe",
  "createdAt": "2016-07-31T17:38:54+03:00",
}

-- ==== DOWN ====

Chunks composed of:

• First line combine request type and address
• Rest of the lines until the next empty line are the json syntax.

I guess it will require subroutine (sed?) to load the instructions.

What do you think?

[mbucc]

You can split JSON up with multiple UP/DOWN sections. This migration works for sqlite3:

-- ====  UP  ====

BEGIN;
    PRAGMA foreign_keys = ON;

    create table name (id int, name string);

COMMIT;

-- ==== DOWN ====

BEGIN;

    drop table name;

COMMIT;

-- ====  UP  ====

BEGIN;
    PRAGMA foreign_keys = ON;

    create table name1 (id int, name string);

COMMIT;

-- ==== DOWN ====

BEGIN;

    drop table name1;

COMMIT;

I actually didn't realize it worked this way! So, out of the box shmig will split up your JSON input and send each piece to the elasticsearch_cli(), elasticsearch_check(), etc.

[mbn18]

@mbucc , Yes, this will make it easier though we still need to figure how to split the first line to type and path. And the rest as the JSON request.

[mbucc]

@mbn18. Yes. That logic can live in the elasticsearch_cli() method and the rest of shmig stays unchanged.

[mbucc]

I don't use ElasticSearch and there is no activity so I'm going to close this. Happy to revisit if someone else has this itch and wants to submit a pull request.

Re: @mbn18's point about lack of transaction handling for DDL statements; MySQL doesn't support it either! So it seems reasonable to use the REST api, and make sure the script fails when REST post fails. The user will have to manually clean things up.

This post (https://stackoverflow.com/a/4736346/1789168) provides the following data:

Is DDL transactional according to this document?

• PostgreSQL - yes
• MySQL - no; DDL causes an implicit commit
• Oracle Database 11g Release 2 and above - by default, no, but an alternative called edition-based redefinition exists
• Older versions of Oracle - no; DDL causes an implicit commit
• SQL Server - yes
• Sybase Adaptive Server - yes
• DB2 - yes
• Informix - yes
• Firebird (Interbase) - yes
• SQLite also appears to have transactional DDL as well