tests: include ES5 and unpin elasticsearch* requirements

[jirikuncar]

• [ ] add ES5 to travis matrix
• [ ] unpin elasticsearch requirements in setup.py

[hjhsalo]

@lnielsen I did some background reading (and googling) and now I have couple of questions:

• Which Elasticsearch 5 version should be used (5.5.x? -- 5.6.x?)

• Currently Travis builds are running Ubuntu Trusty Container-based environment, right?

  • There is https://docs.travis-ci.com/user/database-setup/#Installing-ElasticSearch-on-trusty-container-based-infrastructure which describes:

    env:
    - ES_VERSION=5.1.1 ES_DOWNLOAD_URL=https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
    install:
    - wget ${ES_DOWNLOAD_URL}
    - tar -xzf elasticsearch-${ES_VERSION}.tar.gz
    - ./elasticsearch-${ES_VERSION}/bin/elasticsearch &
    script:
    - wget -q --waitretry=1 --retry-connrefused -T 10 -O - http://127.0.0.1:9200

  • Should the above one be used to install Elasticsearch 5?
  • There is also install instructions for VM based Trusty environment here:
    • https://www.peterbe.com/plog/elasticsearch-5-in-travis-ci
    • https://github.com/peterbe/autocompeter/blob/dfe2db128e628d6dd300385cc6e422e6e1bed57e/.travis.yml

• Use same code for downloading ES2 and ES5?

  • Include new envvar ELASTIC_URL:

    - ELASTIC_2_URL=https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/${ES_VERSION}/elasticsearch-${ES_VERSION}.tar.gz
    - ELASTIC_5_URL=http://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
    
    ...
    
    env: ELASTIC_URL=$ELASTIC_5_URL

  • Change to wget -O ${ES_URL} | tar xz --directory=/tmp/elasticsearch --strip-components=1 and old ES install+startup method should continue to work.
  • (inspired by: https://github.com/mongodb-labs/elastic2-doc-manager/blob/b9230bae11b99ed7da2c3db26ea2bb1c7a1e224b/.travis.yml)

• How to propagate .travis.yml test matrix changes to setup.py as elasticsearch version changes?

  • When ES2 is used setup.py should use elasticsearch>=2.0.0,<3.0.0 and elasticsearch-dsl>=2.0.0,<3.0.0.
  • When ES5 is used setup.py should use elasticsearch>=5.0.0,<6.0.0 and elasticsearch-dsl>=5.0.0,<6.0.0.
  • I was thinking something like this:

    PYTHON_ELASTIC_VERSION = os.environ.get('PYTHON_ELASTIC_VERSION', '>=2.0.0,<3.0.0')
    PYTHON_ELASTIC_DSL_VERSION = os.environ.get('PYTHON_ELASTIC_DSL_VERSION', '>=2.0.0,<3.0.0')
    install_requires = [
        'Flask-BabelEx>=0.9.2',
        'dojson>=1.2.0',
        'elasticsearch' + PYTHON_ELASTIC_VERSION,
        'elasticsearch-dsl' + PYTHON_ELASTIC_DSL_VERSION,
        ...
    ]

  • (inspired by: https://github.com/mongodb-labs/elastic2-doc-manager/blob/683cc482e53a125b6115ff44aba18dd6031b5408/setup.py)

• Making note of Elasticsearch version in Travis logs

  • Should I add version output of Elasticsearch so that it is faster to identify which is elasticsearch version is used when going through logs of failed build?
  • For example:

    - "wget -O ..."
    - "elasticsearch --version"
    - "/tmp/elasticsearch/bin/elasticsearch &"

• Indexing in ES5 vs ES2

  • @slint mentioned something that different kind of indexing should be used for ES5 and ES2. How should this be taken into account? Will it cause changes in some other invenio-modules?

• How should I setup test matrix so that one set of test uses ES2 and another one ES5?

  • I'm not that used experienced in configuring Travis, so I'm not sure which is the best way to include new entries in test matrix.
  • If $ELASTIC_URL which I mentioned above is included and download+install code is identical for both ES2 and ES5, should I just add an new entries under env: -key which use ES5 instead of ES2?:

    - REQUIREMENTS=lowest EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=2.2.0 ELASTIC_URL=$ELASTIC_2_URL
    - REQUIREMENTS=release EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=2.2.0 ELASTIC_URL=$ELASTIC_2_URL
    - REQUIREMENTS=devel EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=2.2.0 ELASTIC_URL=$ELASTIC_2_URL
    - REQUIREMENTS=lowest EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=5.5.3 ELASTIC_URL=$ELASTIC_5_URL
    - REQUIREMENTS=release EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=5.5.3 ELASTIC_URL=$ELASTIC_5_URL
    - REQUIREMENTS=devel EXTRAS=all,sqlite SQLALCHEMY_DATABASE_URI="sqlite:///test.db" ES_HOST=127.0.0.1 ES_VERSION=5.5.3 ELASTIC_URL=$ELASTIC_5_URL

  • Or something else?

[hjhsalo]

I opened https://github.com/inveniosoftware/invenio-oaiserver/pull/132 so maybe we move discussion to there...

[slint]

Screenshot from running the official migration tool on a Zenodo ES 2.3 instance:

[krzysztof]

Some changes to suggesters between ES2 and ES5

ES5 does not support "output"/"payload" during indexing:

• Fix indexers for Invenio-OpenAIRE and Invenio-Opendefinition
• Also all deposit suggest fields will have to fetch the output from the _id, since it looks that payloads are also not returned: _source will contain the full document, and can be truncated using source filtering

Mapping "payloads" is no longer supported in ES5:

• Remove "payloads" from:
  • https://github.com/inveniosoftware/invenio-opendefinition/blob/7b08d9ff92431343a1418019c32524de95885679/invenio_opendefinition/mappings/licenses/license-v1.0.0.json#L52
  • https://github.com/inveniosoftware/invenio-opendefinition/blob/7b08d9ff92431343a1418019c32524de95885679/invenio_opendefinition/mappings/licenses/license-v1.0.0.json#L52
  • https://github.com/inveniosoftware/invenio-openaire/blob/75ba6ed213a9dbbde1550088544dc2339eae11b7/invenio_openaire/mappings/grants/grant-v1.0.0.json#L106

[slint]

One issue here is that the user-defined payload object could also contain some extra logic, as done in the invenio-openaire grants indexing (the legacy_id field is used by Javascript in the UI). Of course, this extra logic could be either moved to the body of the document (adding a legacy_id field in the mapping) or to the consumer of the response (Javascript in the UI).

[krzysztof]

Of course, this extra logic could be either moved to the body of the document (adding a legacy_id field in the mapping)

I guess that's the way to do it, since you would do this logic during indexing anyway, so now you simply write it to a first-class-citizen field instead of the subfield of "suggest"