migrate templates to linked data fragments endpoints

[m1ci]

Currently, we face with performance and availability issues of the SPARQL endpoints which are consumed by the templates. As solution, we consider moving templates to LDF endpoints, which use following datasets:

• DBpedia
• GeoNames
• LinkedGeoData
• EventMedia

If possible, also:

• VIAF
• ORCID

[ghsnd]

In this context, it might be worthwhile to check the LOD Laundromat. You can browse / query the sets (using an online LDF client) in the wardrobe. There is VIAF data over there...

[jnehring]

We can do this in FREME 0.4 - to be discussed.

[m1ci]

in direction of supporting LDF in the templates - we need to distinguish whether the endpoint is LDF endpoint or SPARQL endpoint. We have two solutions:

1) extend the templates with additional parameter, e.g. endpoint-type which will identify the type of the endpoint: sparql or linked-data-fragments

2) automatically detect the endpoint type: this is more challenging, and IMHO not very stable solution. See this discussion on how to detect LDF endpoint.

@jnehring What are your opinions or this?

[ghsnd]

Using option 2, a client can detect if the endpoint is a Triple Pattern Fragment service (= the kind of LDF service we mean), or find out is a SPARQL endpoint, but this requires extra requests and logic. So IMO option 1 is the better solution here. Though I would name the type triple-pattern-fragments to be correct.

[jnehring]

Option 2 seems to be challenging for me. Lets use option 1. I suggest to use default value sparql for endpoint-type for backwards compatibility.

[m1ci]

deal

[m1ci]

@jnehring we need additional flag in the template storage model - whether the template is sparql or ldf. Since you did the persistence, can you please add this flag? I can provide the code for querying LDF endpoint in the DataEnricher - this requires also fixing https://github.com/freme-project/technical-discussion/issues/74

[jnehring]

I created #46 to update the template model. Will be finished tomorrow (hopefully)

[m1ci]

ok, thanks!

[m1ci]

this has be done. The templates now can be run against SPARQL or LDF endpoint.

A template can be created by submitting them as JSON which has the following structure:

{
   "query" : "the sparql query",
   "endpoint" : "the endpoint URL",
   "label" : "the title of the template",
   "description":"a description of the template",
   "type":"the endpoint type, sparql or ldf"
 }

A template can be submitted and created using the following cURL command:

curl -v "http://api-dev.freme-project.eu/current/e-link/templates?visibility=public&outformat=json&token=YOUR_TOKEN" -H "Content-Type: application/json" -d @template.json

Note: you should specify the visibility of the template using the visibility parameter.

Upon successful creation, in the response the client receives representation of the template including the ID. Example of such response:

{
  "id" : 302,
  "owner" : {
    "name" : "milan",
    "role" : "ROLE_USER"
  },
  "visibility" : "PUBLIC",
  "endpoint" : "http://live.dbpedia.org/sparql",
  "query" : "PREFIX dbpedia: <http://dbpedia.org/resource/> PREFIX dbo: <http://dbpedia.org/ontology/> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> CONSTRUCT {   ?museum <http://xmlns.com/foaf/0.1/based_near> <@@@entity_uri@@@> . } WHERE {  <@@@entity_uri@@@> geo:geometry ?citygeo .  OPTIONAL { ?museum rdf:type dbo:Museum . }  ?museum geo:geometry ?museumgeo .  FILTER (<bif:st_intersects>(?museumgeo, ?citygeo, 50)) } LIMIT 10",
  "label" : "the title of the template",
  "description" : "a description of the template",
  "type" : "SPARQL"
}

A template can be used for enrichment using the following cURL command:

curl -v "http://api-dev.freme-project.eu/current/e-link/documents?templateid=302&outformat=turtle&token=YOUR_TOKEN" -H "Content-Type: text/turtle" -d @nif.ttl

[m1ci]

closing the issue, you can test the LDF templates using the instruction in the previous comment.