kleio-server provides access to translation services for Kleio files.
Kleio filesKleio files are text files with a special notation designed to the transcription of historical sources. The notation was created by Manfred Thaller, as part of the Kleio historical database system (http://web.archive.org/web/20130603204750/http://www.hki.uni-koeln.de/kleio/old.website/).
Although Thaller's Kleio database system is no longer publically available, the notation developped for historical source transcritpion proved very powerfull, providing a concise way for the transcription of complex historical documents.
The Timelink-Kleio translator implements a subset of the Kleio notation designed for the Timelink database system. Timelink provides a set of data models designed for handling person-oriented information collected in historical documents.
This is how a (portuguese) baptism looks like in Timelink Kleio notation:
bap$b1714-2/12/11714/fl.117v./igreja de sao silvestre/manuel lopes serra (padre)
celebrante$manuel lopes serra
ls$profissao/padre
n$francisca/f
pn$antonio ferreira
ls$morada/espinheiro
ls$freguesia/lousa
mn$leonarda francisca
pad$joao fernandes ramalheiro
ls$morada/moita
ls$freguesia/lousa
mad$francisca
ls$ec/solteira
pmad$joao goncalves
ls$morada/espinheiro
ls$freguesia/lousaText files implementing the Timelink Kleio notation can be translated by
the Timelink kleio-server and imported into the Timelink relational database.
Translation is intelligent in the sense that it operates a normalization of the source information, infering information from the context, and so greatly reducing the overhead of producing normalized data.
Timelink database-services then implement a set of functions that allow the identification of people, reconstruction of biographies, inference of personal networks, and other funcionalities.
For more information on Timelink Kleio Server see: https://github.com/time-link/timelink-kleio
kleio-server APIThe API is designed to decouple Kleio source handling from other software components. It provides funcionality to translate source files, inspect results of translation for errors or warnings, obtain the generated data in XML format. It also allows basic file management and basic git operations, so that it can be used to isolate other software components from directly handling file related operations.
Main services privided by the API are:
API documentation is available in docs/api
API documentation is available at docs/api/index.html
$ docker run -v $(PWD):/kleio-home -p 8088:8088 -d timelink-server/kleio-servermake build-local
$ docker run -v $PWD:/kleio-home -p 8088:8088 -d kleio-serverTo have the server listening on port 8089
$ docker run -v $(PWD):/kleio-home -p 8088:8089 -d kleio-server$ docker run -v $(PWD):/kleio-home -e KLEIO_ADMIN_TOKEN=myprivatetoken -p 8088:8088 -d kleio-serverIf KLEIO_ADMIN_TOKEN is not set the server will generate
a token with admin privileges which can be
obtained in the .kleio.json
file in the directory mapped to /kleio-home
$docker run -v $PWD:/kleio-home -p 8088:8088 -d kleio-server
$cat $PWD/.kleio.jsonThe generated token is associated with user kleio_admin, and can be invalidated by a client after generating a new one.
On linux system docker will run under the root user. This makes the server generate files that will generate permission errors when accessed by the current user.
To run the kleio server user the current user:
docker run -v $PWD:/kleio-home -u $(id -u):$(id -g) -p 8088:8088 -d kleio-serverThere are more environment variables that can be
set to control the behaviour of kleio-server.
The easiest way to test in to use docker compose
with the docker-compose.yamlfile in this directory
and set the variables in a .env file:
.env-sample to .env.KLEIO_HOME should be set to a directory where kleio files reside.KLEIO_SERVER_IMAGE can be set to run a specific image otherwise timelinkserver/kleio-server:latest is used.KLEIO_ADMIN_TOKEN if you want to set a starting admin token. If the env variable KLEIO_ADMIN_TOKEN is unset
then a token named kleio-admin is created on server startup with full admin permissions.
The token is written to file /.admin_token in KLEIO_CONF_DIR.
This token can be used to generate another token through an api
call and invalidate this oneKLEIO_DEBUGif "true" will produce debug information in the log..env-sample for a full list.make kleio-run-latest or docker compose upKleio server is written in SWI-Prolog https://www.swi-prolog.org
Recommended tools:
VSCode with VSC-Prolog extension https://marketplace.visualstudio.com/items?itemName=arthurwang.vsc-prologPostman https://www.postman.com
Postman-doc-gen: generates static documentation from Postman collections https://github.com/karthiks3000/postman-doc-gennewman: the command line tool for running Postman generated test suites https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/Easiest way:
install swipl locally
install VSC-Prolog extension in VSCode
open serverStart.pl on VSCode
load the file with Option+X+L
in the Prolog terminal that appears do
setenv('KLEIO_ADMIN_TOKEN','mytoken'). setup_and_run_server(run_debug_server,[port(8089)]).
note that the server started in this way does not read the content of the .env file.
set spy points with tspy(predicate)
Clients can access the kleio-server using the token set above.
See the README.md file inside the tests directory for more information on testing.
After building the image with make build-local run a shell in the kleio-server container
docker compose run kleio shAt the prompt start swi Prolog and change to the kleio-home directory
# swipl -f serverStart.pl
?- working_directory(_,'/kleio-home').
?- run_debug_server.Currently MHK debugging is done by starting the current build together with the other containers (mySQL/MariaDB, Caddy, Portainer and Kleio) through a Docker compose file and by attaching the Intellij IDEA debugger to the running MHK app.
The recomended way is to stop the kleio server that is started with MHK and start a local server in VSCode as explained above, and associating it with the same mhk-home directory of the MHK beeing jointly tested, as described bellow.
In order for MHK to connect to the local running Kleio server the property mhk.kleio.service
should be set to http://host.docker.internal:8088
in the file mhk-home/system/conf/mhk_system.properties
This means that the MHK instance running in Docker will try to connect to port 8088 in the local host. Values of http://127.0.0.1:8088 ot http://localhost:8088
The sequence should be:
mhk update --local (make sure
mhk is updating from latest image wiht mhk use-tag latest)mhk startmhk stop kleioSteps 1-3 above are normally automated inside Intellij IDEA with a Debug configuration.
Start the Kleio server inside VSCode:
VSC-Prolog extension in VSCodeVSCodein the Prolog terminal that appears do
run_from_mhk_home(H,P)
This will start a kleio server from USER_HOME/mhk-home running in port 8088. If mhk-home used by MHK is at a different location pass the path in H. In a port other than 8088 is to be used pass it in P.
There two type of tests:
See README-tests.md in the directory tests for details. The tests directory also contains
reference files and the reference translator necessary to run both semantic and api tests.
make build-local: create a new local image, and tags it kleio-server:Knnn, with Knnn being a sequential build numberThe following tags are used:
To help manage the tags the following targets exist
make tag-multi-TAG | tag-local-TAG tag last image with TAG in latest | unique | stable, note that unique tags the last build numbermake inc_NUMBER increment version with NUMBER in major | minormake push-TAG push last image with TAG in latest | unique | stable, to the
docker repository defined in the DOCKER_REPOSITORY variable in the Makefile.make build-multi (also pushes to repository tag with current version)
timelinkserver/kleio-server login before with
docker login --username timelinkserver or other
authorized user.make inc-major or make inc-minormake build-multi (also pushes to repository tag with updated version)make show-current or
make show-lastmake tag-multi-stable (this also tags latest commit with last version number).stabledirectory in tests.make tag-multi-latestType make to see more targets that help in development.
make build-local build a local docker image and tag with new build number
make tag-local-TAG tag last local image with TAG in latest | stable
make build-multi build local mage and push multi platform docker imagea tagged with new build number
(requires login in dokerhub with 'docker login' )
make tag-multi-TAG tag last multi platform image with TAG in latest | stable
make show-build return the current build number
make show-version return the current version string (major.minor)
make show-current return the current version, build number
make show-last return the last image build date, version, build number
make show-env show KLEIO env variables currently defined
make inc-NUMBER increment version with NUMBER in major | minor
make gen-token generate a string suitable for KLEIO_ADMIN_TOKEN for .env file
make bootstrap-token generate and register a token for 'admin' during bootstrap
(only if no tokens exist and server running < 5 minutes)
make docs generate api docs (requires postman_doc_gen and api files)
make pull-tag tag=x.y.z pull image with tag x.y.z from docker hub
make kleio-run-latest start server with latest multi platform image, .env config and tests/docker_compose.yaml
make kleio-run-current start server with most recent build, .env config and tests/docker_compose.yaml
make kleio-run-tag tag=x.y.z start server with image with tag x.y.z, .env config and tests/docker_compose.yaml
make kleio-stop | stop stop running server
make test-semantics run semantic tests
make test-api run api tests (requires newman (npm install newman))( make show-last to have timestamp and version info)
Fixes problem with local token db that generated a false error of unkown method in some token operations.
Added property$NAME/VALUE subgroup to the kleio$ group.
This allows for specifying file properties that affect the way the source is translated.
The first (and only) property currently implemented is multuple-entry-flag
that sets the character interpreted as separating multiples entries in a field
(data flag 8 in the original Manfred Thaller implementation, see issue #34
Example, to set the separator to the pipe " | " character (ascii 124).
property$multiple-entry-flag/124Also:
Fix bugs, linked data ids now allow for dashes, improve error and warning line reporting
Fixes a bug in translation of identification files.
Fixes bugs, first tests with timelink-py client.
Adds linked data notation.
Declare the external source, with an attribute of the kleio document: kleio$... link$wikidata/"http://wikidata.org/wiki/$1"
The format of the link$ group is: link$short-name/url-pattern
where short-name is a short name for an external sources
url-pattern should be a url containing a place
holder ($1) for a specific id of the data item to be linked
Anotate element values with external ids
ls$jesuita-entrada/Goa, Índia# @wikidata:Q1171/15791200
The format of an external link annotation is:
@short-name:idOn translation annotation such as
ls$jesuita-entrada/Goa, Índia# @wikidata:Q1171/15791200
will generate
ls$ jesuita-entrada@/https://www.wikidata.org/wiki/Q1171/15791200/obs=%Goa, Índia
Internal:
Implements issue #7 allowing str files to be located close to the sources
At startup the server generates a .kleio.json file
with the running configuration, including an admin token.
New doc about client setup
New translation result summary in json translation results
Clean up targets for build and run
Adds multi-architecture docker builds.
Fixes issue 5 generates bootstrap token at each startup.
Fixes CORS pre-flight requests.
Adds cors capability. Set env KLEIO_CORS_SITES with comma separeted list of allowed sites or "*" for all.
More consistent usage of semantic versioning.
Now stable images are tagged with major version number, minor version number and patch e.g. "10", "10.16" and "10.16.452"
Development versions are only tagged with full patch tag, e.g. "10.16.452"
Added make targets for api-testing. Type "make help" for a list of targets
See development section bellow.