Elasticsearch Storage Module

[alexrobin]

Develop a storage implementation to persist OSH data to an Elasticsearch backend. This module will be built against OSH persistence API.

• [x] Setup local Elasticsearch demo server
• [x] Test Elasticsearch Java API in a separate program
• [x] Initiate OSH module project with proper Gradle config (with dependency to ES Java)
• [x] Implement time series data store
• [x] Implement feature data store
• [x] Implement IObsStorage to wrap it all up
• [x] Implement the single-storage module (IObsStorageModule)
• [x] Implement the multi-storage module (IMultiSourceStorage)
• [x] JUnit Tests for Basic storage
• [x] Junit Tests for Obs storage
• [x] Junit Tests for Multi-storage
• [x] Integration Tests - Basic Storage
• [x] Integration Tests - ObsStorage
• [x] Integration Tests - MultiSource
• [ ] Documentation
• [ ] Blog Post

[nicolas-f]

I don't know why but the class org.elasticsearch.common.network.NetworkService can't be initialized. Have you got the same issue ? (it prevent storage-es from working)

[nicolas-f]

And also

We plan on deprecating the TransportClient in Elasticsearch 7.0 and removing it completely in 8.0. Instead, you should be using the Java High Level REST Client, which executes HTTP requests rather than serialized Java requests.

https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-api.html

[nicolas-f]

I will try to upgrade to the last version of non-rest elasticsearch in the first time.

[nicolas-f]

I updated the code but I have a lot of difficulty handling the error (Rejecting mapping update to[junit_81396cc9-a133-484d-99e3-981d539bba78] as the final mapping would have more than 1 type: [foi, data]) with the new version:

https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

[nicolas-f]

I done some progress on this module. For the ds3 data, the generated mapping is:

{
  "mapping": {
    "ds3": {
      "dynamic": "false",
      "properties": {
        "elt": {
          "type": "nested",
          "dynamic": "false",
          "properties": {
            "c1": {
              "type": "double"
            },
            "c2": {
              "type": "integer"
            },
            "c3": {
              "type": "text"
            }
          }
        },
        "producerID": {
          "type": "keyword"
        },
        "storageID": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "long"
        }
      }
    }
  }
}

And the data produced:

{
  "_index": "data_junit_testesbasicstorage_1530275630988_ds3",
  "_type": "ds3",
  "_id": "ds3##4621819117588971520##urn:domain:sensors:1",
  "_score": 1,
  "_source": {
    "producerID": "urn:domain:sensors:1",
    "timestamp": 10000,
    "storageID": "junit_testesbasicstorage_1530275630988",
    "elt": [
      {
        "c1": 0.5,
        "c2": 0,
        "c3": "test0"
      },
      {
        "c1": 1.5,
        "c2": 2,
        "c3": "test1"
      },
      {
        "c1": 2.5,
        "c2": 4,
        "c3": "test2"
      },
      {
        "c1": 3.5,
        "c2": 6,
        "c3": "test3"
      },
      {
        "c1": 4.5,
        "c2": 8,
        "c3": "test4"
      },
      {
        "c1": 5.5,
        "c2": 10,
        "c3": "test5"
      },
      {
        "c1": 6.5,
        "c2": 12,
        "c3": "test6"
      },
      {
        "c1": 7.5,
        "c2": 14,
        "c3": "test7"
      },
      {
        "c1": 8.5,
        "c2": 16,
        "c3": "test8"
      },
      {
        "c1": 9.5,
        "c2": 18,
        "c3": "test9"
      }
    ]
  }
}

In order to find a document that contains a specific c1 and c3:

GET *_ds3/_search
{
  "query": {
    "nested": {
      "path": "elt",
      "query": {
        "bool": {
          "must": [
            { "match": { "elt.c3": "test0" }},
            { "match": { "elt.c1": 0.5 }}
          ]
        }
      }
    }
  }
}

[nicolas-f]

2 units test remaining for BasicStorage. Looks good.

[nicolas-f]

I'm having issues with your observation storage unit test. Inserted values are not WGS84 valid coordinates, so Elastic Search is rejecting theses coordinates.

[mikebotts]

Hi Nicolas.

I’m not familiar with the code like Alex, but I know there have always been some challenges with spatial reference system (SRS) designations.

• have you tried reversing the order of lat/lon to lon/lat (or is this something that we have control of not you?)
• what are/is your SRS designation?

Alex is on vacation in Sardinia (poor guy) so he may not respond immediately.

Thanks. Mike

Dr. Mike Botts Botts Innovative Research, Inc. http://www.botts-inc.com http://www.botts-inc.com/ https://opensensorhub.org https://opensensorhub.org/ 256-652-0165

On Jul 3, 2018, at 6:03 AM, nicolas-f notifications@github.com wrote:

I'm having issues with your observation storage unit test. Inserted values are not WGS84 valid coordinates, so Elastic Search is rejecting theses coordinates.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/opensensorhub/osh-persistence/issues/1#issuecomment-402111243, or mute the thread https://github.com/notifications/unsubscribe-auth/AHH6BscbLkcpDSdXktoMcePcCYVJT4k-ks5uC0-VgaJpZM4L7XNa.

[nicolas-f]

Hi Mike,

Its only the unit test crafted to add latitude values over 90°.

I've created a pull request in order to push coordinates in the WGS84 bounds:

https://github.com/opensensorhub/osh-core/pull/113

[nicolas-f]

Unit test points:

[nicolas-f]

Done.

• [x] Integration Tests - Basic Storage
• [x] Integration Tests - ObsStorage
• [x] Integration Tests - MultiSource

Need some cleaning/doc if you want to merge. And I prefer to drop support for shape geometry as Elastic Search is doing heavy work on this (only point geometry support).

Work on https://github.com/nicolas-f/osh-persistence/tree/upgrade-rest-es-6.X