search

CLARIAH / grlc

grlc builds Web APIs using shared SPARQL queries
http://grlc.io
MIT License
137 stars 32 forks source link
api linked-data linked-data-api open-api semantic-web sparql swagger-ui

readme

PyPI version DOI Build Status status

grlc, the git repository linked data API constructor, automatically builds Web APIs using shared SPARQL queries. http://grlc.io/

If you use grlc in your work, please cite it as:

@InProceedings{merono2016grlc,
 author = {Mero{\~{n}}o-Pe{\~{n}}uela, Albert and Hoekstra, Rinke},
 title = {{grlc Makes GitHub Taste Like Linked Data APIs}},
 booktitle = {The Semantic Web: ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 -- June 2,  2016},
 year = {2016},
 publisher = {Springer},
 pages = {342--353},
 isbn = {978-3-319-47602-5},
 doi = {10.1007/978-3-319-47602-5_48}
}

What is grlc?

grlc is a lightweight server that takes SPARQL queries (stored in a GitHub or GitLab repository, in your local filesystem, or listed in a URL), and translates them to Linked Data Web APIs. This enables universal access to Linked Data. Users are not required to know SPARQL to query their data, but instead can access a web API.

Quick tutorial

For a quick usage tutorial check out our wiki walkthrough and list of features.

Usage

grlc assumes that you have a collection of SPARQL queries as .rq files (like this). grlc will create one API operation for each SPARQL query/.rq file in the collection.

Your queries can add API parameters to each operation by using the parameter mapping syntax. This allows your query to define query variables which will be mapped to API parameters for your API operation (see here for an example).

Your queries can include special decorators to add extra functionality to your API.

Query location

grlc can load your query collection from different locations: from a GitHub repository (api-git), from a GitLab repository (api-gitlab), from local storage (api-local), and from a specification file (api-url). Each type of location has specific features and is accessible via different paths. However all location types produce the same beautiful APIs.

From a GitHub repository

API path: http://grlc-server/api-git/<user>/<repo>

grlc can build an API from any Github repository, specified by the GitHub user name of the owner (<user>) and repository name (<repo>).

For example, assuming your queries are stored on a Github repo: https://github.com/CLARIAH/grlc-queries/, point your browser to the following location http://grlc.io/api-git/CLARIAH/grlc-queries/

grlc can make use of git's version control mechanism to generate an API based on a specific version of queries in the repository. This can be done by including the commit sha in the URL path (http://grlc-server/api-git/<user>/<repo>/commit/<sha>), for example: http://grlc.io/api-git/CLARIAH/grlc-queries/commit/79ceef2ee814a12e2ec572ffaa2f8212a22bae23

grlc can also use a subdirectory inside your Github repo. This can be done by including a subdirectory in the URL path (http://grlc-server/api-git/<user>/<repo>/subdir/<subdir>).

From a GitLab repository

API path: http://grlc-server/api-gitlab/<user>/<repo>

grlc can build an API from any GitLab repository, specified by the GitLab user name of the owner (<user>) and repository name (<repo>).

For example, assuming your queries are stored on a GitLab repo: https://gitlab.com/c-martinez/grlc-queries, point your browser to the following location http://grlc.io/api-gitlab/c-martinez/grlc-queries/

grlc can make use of git's version control mechanism to generate an API based on a specific version of queries in the repository. This can be done by including the name of a branch in the URL path (http://grlc-server/api-gitlab/<user>/<repo>/branch/<branch>), for example: http://grlc.io/api-gitlab/c-martinez/grlc-queries/branch/master

grlc can also use a subdirectory inside your GitLab repo. This can be done by including a subdirectory in the URL path (http://grlc-server/api-gitlab/<user>/<repo>/subdir/<subdir>), for example: http://grlc-server/api-gitlab/c-martinez/grlc-queries/subdir/subdir.

From local storage

API path: http://grlc-server/api-local/

grlc can generate an API from a local directory in the computer where your grlc server runs. You can configure the location of this directory in your grlc server configuration file. See also how to install and run your own grlc instance.

When the API is generated from a local directory, API information can be loaded from a configuration file in that folder. This file must be called local-api-config.ini and it has the following format:

[repo_info]
repo_title = Some title
api_description = Description of my API
contact_name = My name
contact_url = https://mypage/
licence_url = https://mylicence/

From a specification file

API path: http://grlc-server/api-url/?specUrl=<specUrl>

grlc can generate an API from a yaml specification file accessible on the web.

For example, assuming your queries are listed on spec file: https://raw.githubusercontent.com/CLARIAH/grlc-queries/master/urls.yml, point your browser to the following location http://grlc.io/api-url?specUrl=https://raw.githubusercontent.com/CLARIAH/grlc-queries/master/urls.yml

Specification file syntax

A grlc API specification file is a YAML file which includes the necessary information to create a grlc API, most importantly a list of URLs to decorated and HTTP-dereferenceable SPARQL queries. This file should contain the following fields

For example:

title: Title of my API
description: Description of my API
contact:
  name: Contact Name
  url: https://www.mywebsite.org
licence: http://example.org/licence.html
queries:
  - https://www.mywebsite.org/query1.rq
  - https://www.mywebsite.org/query2.rq
  - https://www.otherwebsite.org/query3.rq
  - name: QueryFour
    url: https://www.mywebsite.org/query4.rq

grlc generated API

The API paths of all location types point to the generated swagger-ui style API documentation. On the API documentation page, you can explore available API calls and execute individual API calls.

You can also view the swagger spec of your API, by visiting <API-path>/swagger, for example: http://grlc.io/api-git/CLARIAH/grlc-queries/swagger

grlc query execution

When you call an API endpoint, grlc executes the SPARQL query for that endpoint by combining supplied parameters and decorators.

There are 4 options to specify your own endpoint:

The endpoint call will return the result of executing the query as a json representation of rdflib.query.QueryResult (for other result formats, you can use content negotiation via HTTP Accept headers). For json responses, the schema of the response can be modified by using the #+ transform: decorator.

Decorator syntax

Special decorators are available to make your swagger-ui look nicer and to increase functionality. These are provided as comments at the start of your query file, making it still syntactically valid SPARQL. All decorators start with #+, for example:

#+ decorator_1: decorator value
#+ decorator_1: decorator value

SELECT * WHERE {
  ?s ?p ?o .
}

The following is a list of available decorators and their function:

summary

Creates a summary of your query/operation. This is shown next to your operation name in the swagger-ui.

Syntax:

#+ summary: This is the summary of my query/operation

Example query and the equivalent API operation.

description

Creates a description of your query/operation. This is shown as the description of your operation in the swagger-ui.

Syntax:

#+ description: Extended description of my query/operation.

Example query and the equivalent API operation.

endpoint

Specifies a query-specific endpoint.

Syntax:

#+ endpoint: http://example.com/sparql

Example query and the equivalent API operation.

pagination

Paginates the results in groups of (for example) 100. Links to previous, next, first, and last result pages are provided as HTTP response headers to avoid polluting the payload (see details here)

Syntax:

#+ pagination: 100

Example query and the equivalent API operation.

method

Indicates the HTTP request method (GET and POST are supported).

Syntax:

#+ method: GET

Example query and the equivalent API operation.

tags

Assign tags to your query/operation. Query/operations with the same tag are grouped together in the swagger-ui.

Syntax:

#+ tags:
#+   - firstTag
#+   - secondTag

Example query and the equivalent API operation.

defaults

Set the default value in the swagger-ui for a specific parameter in the query.

Syntax:

#+ defaults:
#+   - param_name: default_value

Example query and the equivalent API operation.

enumerate

Indicates which parameters of your query/operation should get enumerations (and get dropdown menus in the swagger-ui) using the given values from the SPARQL endpoint. The values for each enumeration variable can also be specified into the query decorators to save endpoint requests and speed up the API generation.

Syntax:

#+ enumerate:
#+   - var1:
#+     - value1
#+     - value2

Example query and the equivalent API operation.

Notice that these should be plain variable names without SPARQL/BASIL conventions (so var1 instead of ?_var1_iri)

endpoint_in_url

Allows/disallows the endpoint parameter from being provided as a URL parameter (allowed by default).

Syntax:

#+ endpoint_in_url: False

Example query and the equivalent API operation.

transform

Allows query results to be converted to the specified JSON structure, by using SPARQLTransformer syntax. Notice that the response content type must be set to application/json for the transformation to take effect.

Syntax:

#+ transform: {
#+     "key": "?p",
#+     "value": "?o",
#+     "$anchor": "key"
#+   }

Example query and the equivalent API operation.

endpoint-method

Allows the query to be sent from the grlc server to the SPARQL endpoint using either GET or POST http method. (Default: POST)

Syntax:

#+ endpoint-method: GET

Example query and the equivalent API operation.

Example APIs

Check these out:

You'll find the sources of these and many more in GitHub

Use this GitHub search to see examples from other grlc users.

Install and run

You can use grlc in different ways:

More details for each of these options are given below.

grlc.io

The easiest way to use grlc is by visiting grlc.io and using this service to convert SPARQL queries into a RESTful API. Your queries can be stored on a github repo or can be listed on a specification file.

Docker

To run grlc via docker, you'll need a working installation of docker. To deploy grlc, just pull the latest image from Docker hub. :

docker run -it --rm -p 8088:80 clariah/grlc

The docker image allows you to setup several environment variable such as GRLC_SERVER_NAME GRLC_GITHUB_ACCESS_TOKEN,GRLC_GITLAB_ACCESS_TOKEN and GRLC_SPARQL_ENDPOINT:

docker run -it --rm -p 8088:80 -e GRLC_SERVER_NAME=grlc.io -e GRLC_GITHUB_ACCESS_TOKEN=xxx -e GRLC_GITLAB_ACCESS_TOKEN=yyy -e GRLC_SPARQL_ENDPOINT=http://dbpedia.org/sparql -e DEBUG=true clariah/grlc

Pip

If you want to run grlc locally or use it as a library, you can install grlc on your machine. Grlc is registered in PyPi so you can install it using pip.

Prerequisites

grlc has the following requirements:

pip install

Once the base requirements are satisfied, you can install grlc like this:

pip install grlc

Once grlc is installed, you have several options:

Standalone server

grlc includes a command line tool which you can use to start your own grlc server:

grlc-server

Using a WSGI server

You can run grlc using a WSGI server such as gunicorn as follows:

gunicorn grlc.server:app

If you want to use your own gunicorn configuration, for example gunicorn_config.py:

workers = 5
worker_class = 'gevent'
bind = '0.0.0.0:8088'

Then you can run it as:

gunicorn -c gunicorn_config.py grlc.server:app

Note: Since gunicorn does not work under Windows, you can use waitress instead:

waitress-serve --port=8088 grlc.server:app

If you want to run grlc at system boot as a service, you can find example upstart scripts at upstart/

grlc library

You can use grlc as a library directly from your own python script. See the usage example to find out more.

grlc server configuration

Regardless of how you are running your grlc server, you will need to configure it using the config.ini file. Have a look at the example config file to see how it this file is structured.

The configuration file contains the following variables:

Git access token

In order for grlc to communicate with GitHub and/or GitLab, you'll need to tell grlc what your access token is:

  1. Get a GitHub personal access token or GitLab personal access token.
  2. You'll get an access token string, copy it and save it somewhere safe.
  3. Edit your config.ini (github_access_token and gitlab_access_token respectively) and/or docker-compose.yml (GRLC_GITHUB_ACCESS_TOKEN and GRLC_GITLAB_ACCESS_TOKEN environment variables).

Contribute!

grlc needs you to continue bringing Semantic Web content to developers, applications and users. No matter if you are just a curious user, a developer, or a researcher; there are many ways in which you can contribute:

Check our contributing guidelines for these and more, and join us today!

If you cannot code, that's no problem! There's still plenty you can contribute:

Related tools

This is what grlc users are saying

Quotes from grlc users:

A cool project that can convert a random SPARQL endpoint into an OpenAPI endpoint

It enables us to quickly integrate any new API requirements in a matter of seconds, without having to worry about configuration or deployment of the system

You can store your SPARQL queries on GitHub and then you can run your queries on your favourite programming language (Python, Javascript, etc.) using a Web API (including swagger documentation) just as easily as loading data from a web page

Contributors: Albert Meroño, Rinke Hoekstra, Carlos Martínez

Copyright: Albert Meroño, Rinke Hoekstra, Carlos Martínez
License: MIT License (see LICENSE.txt)

Academic publications