JSON-LD based mapping implementation of GM

[blankdots]

Description

Using compact and flattened JSON-LD as the output format it is ingested to the latest version of the ES. Implementation has:

• [x] Mapping capabilities
• [x] API endpoints
• [x] ES indexing

Require investigation of the old mapping config.

DoD

Transformation can be called using the update the GM API.

Testing

Unit Tests, BDD tests

[blankdots]

Related topics

• JSON-LD ES discussion https://discuss.elastic.co/t/loading-json-ld-into-es/19970/12
• curl example: https://gist.github.com/jprante/4232257
• https://hub.docker.com/r/stain/ops-search-elasticsearch/ - OpenPhacts elastic search
• Why You Need an Ontology (Schema.org, JSON-LD and ElasticSearch - Part 1) http://markbirbeck.com/2016/11/14/schema.org-json-ld-elasticsearch-1/
• JSON RDF Query Language, a JSON-LD based SPARQL serialisation https://github.com/gsvarovsky/json-rql
• https://github.com/lobid/lodmill - see Linked Data Records to Index
• Indexing Linked Bibliographic Data with JSON-LD, BibJSON and Elasticsearch http://journal.code4lib.org/articles/7949

[blankdots]

The unit tests will execute properly if there is Fuseki and Elasticsearch running.

--------------- coverage: platform linux2, python 2.7.6-final-0 ----------------
Name                             Stmts   Miss  Cover
----------------------------------------------------
src/gm_api/__init__                  0      0   100%
src/gm_api/api/__init__              0      0   100%
src/gm_api/api/clusterids           14      0   100%
src/gm_api/api/entitieslinks        19      0   100%
src/gm_api/api/mapping              32      1    97%
src/gm_api/app                      19      1    95%
src/gm_api/gmapi                    29      6    79%
src/gm_api/lib/__init__              0      0   100%
src/gm_api/lib/construct_ids        65     12    82%
src/gm_api/lib/construct_links       0      0   100%
src/gm_api/lib/construct_map       131     33    75%
src/gm_api/resources/__init__        8      0   100%
src/gm_api/utils/__init__            0      0   100%
src/gm_api/utils/db                 13      3    77%
src/gm_api/utils/logs                7      0   100%
src/gm_api/utils/prefixes           14      0   100%
src/gm_api/utils/validate           21      2    90%
----------------------------------------------------
TOTAL                              372     58    84%

[blankdots]

BDD Tests are not perfect and require running applications but are there.

      Scenario: Add map and get its status          # features/cluster.feature:2
        Given graph API and Graph Store are running # ClusterIDs.java:23
        When I run a clusterids job                 # ClusterIDs.java:41
        Then I should get the status processed.     # ClusterIDs.java:58
    Feature: Handle mapping and indexing from the graph to Elasticsearch

org.uh.attx.gc.graphcomponent.test.stepdefinitions.TestRunner > Scenario: Retrieve mapping results.classMethod STANDARD_OUT

      Scenario: Add map and get its status                         # features/gmapi.feature:2
        Given graph API, Elasticsearch and Graph Store are running # GraphAPI.java:32
        When I post a mapping                                      # GraphAPI.java:57
        Then I should be able to retrieve status of mapping.       # GraphAPI.java:89

org.uh.attx.gc.graphcomponent.test.stepdefinitions.TestRunner > Scenario: Delete map results.classMethod STANDARD_OUT

      Scenario: Retrieve mapping results                            # features/gmapi.feature:7
        Given graph API, Elasticsearch and Graph Store are running  # GraphAPI.java:32
        When I post a new mapping                                   # GraphAPI.java:73
        And I retrieve that mapping                                 # GraphAPI.java:134
        Then I should be able to see the resource in Elasticsearch. # GraphAPI.java:119

org.uh.attx.gc.graphcomponent.test.stepdefinitions.TestRunner STANDARD_OUT

      #        TBD when we have a proper end to end workflow
      #        Then I should get indexed data in JSON-LD from the Elasticsearch.
      Scenario: Delete map results                             # features/gmapi.feature:15
        Given graph API is running                             # GraphAPI.java:105
        When I delete a mapping                                # GraphAPI.java:148
        But I should not be able to retrieve that mapping      # GraphAPI.java:170
        Then the mapping result still exists in Elasticsearch. # GraphAPI.java:183

    4 Scenarios (4 passed)
    14 Steps (14 passed)
    0m20.074s