Publishing geospatial data online through REST API.
Linux kernel v4.11+ (because of QGIS server and its Qt 5.10+/libQt5Core dependency), for example
You can check your kernel version with uname -r
.
Optionally
make
command
git clone https://github.com/LayerManager/layman.git
cd layman
# checkout the latest release in current branch
git checkout $(git describe --abbrev=0 --tags)
There are following ways how to run Layman:
This is the easiest way how to start layman for demonstration purposes. It includes also external dependencies. However it's not meant for production and it's not safe for production. Performance might be also an issue.
# use demo settings
cp .env.demo .env
# start dockerized containers in background
make start-demo-full
Initial startup may take few minutes (download docker images, build it, run it). You are interested in container named layman
. You can check its logs with command
docker logs -f layman
Wait until you see something like
[2019-10-30 13:45:36 +0000] [12] [INFO] Layman successfully started!
Then visit http://localhost/. You will see simple web client that interacts with REST API. To check if Layman is running, call GET Version.
To stop running service, run make stop-demo
.
By default, demo starts without authentication provider. You can publish layers and maps as anonymous user, but you can't log in nor publish any private data.
To be able to use authentication and publish private data, follow these steps:
make stop-demo
OAUTH_*
environment variables from .env.dev
to .env
file..env
change environment variables:
LAYMAN_AUTHN_MODULES=layman.authn.oauth2,layman.authn.http_header
OAUTH2_CALLBACK_URL=http://localhost/client/authn/oauth2-provider/callback
make start-demo-full-with-optional-deps
Then you can log in with automatically provided Wagtail user layman
and password laymanpwd
.
Layman's source code provides settings suitable for development, testing and demo purposes. Furthermore, there exists Makefile
with predefined commands for each purpose including starting all necessary services (both in background and foreground) and stoping it.
Layman's configuration is split into three levels:
docker-compose.*.yml
files used as docker-compose configuration files with most general settings of docker containers including volume mappings, port mappings, container names and startup commands.env*
files with environment settings of both build stage and runtime of docker containerssrc/layman_settings.py
Python module with settings of Layman's Python modules for runtimeFiles at all three levels are suffixed with strings that indicates what they are intended to:
demo
to demonstration purposesdev
to developmenttest
to automatic testsdeps
to external dependenciesWhen you are switching between different contexts (e.g. between demo and dev), always check that you are using settings intended for your context, especially
.env*
file (check env_file
properties in docker-compose.*.yml
file)Also, anytime you change .env
file, remember to rebuild docker images as some environment variables affect build stage of docker images. Rebuild happens automatically as a part of make start-demo*
commands.
Layman has many dependencies. Most of them are shipped with Layman. However, there are some external dependencies that should be treated carefully:
These external dependencies are shipped with Layman for development, testing and demo purposes. They are grouped in docker-compose.deps*.yml
files.
However, if you want to run Layman in production, it is strongly recommended to install external dependencies separately. Recommended (i.e. tested) versions are:
Layman follows semantic versioning, so any change in MINOR and PATCH version should be backwards compatible. Still, human make mistakes, so it's recommended to backup data directories as a part of each upgrade.
General steps to upgrade layman to MINOR or PATCH version:
make stop-demo
, for dev configuration make stop-dev
.layman_data
deps/*/data
(data directories of external dependencies)make start-dev
), run also
make build-dev
make client-build
make timgen-build
make wagtail-build
make micka-build
make start-demo
, run make upgrade-demo
.make start-demo-full
or make start-demo-full-with-optional-deps
, run make upgrade-demo-full
.make start-dev
, you don't need to run standalone migration.NOTE: If upgrade failed due to timeout of request to GeoServer, run
make upgrade-after-timeout
to finish upgrade. NOTE: If you run in problems during upgrade process, especially before v1.12.0, you can try to upgrade gradually through individual minor versions.
To run layman in production, you need to provide external dependencies and configure layman manually.
When providing external dependencies, check their production-related documentation:
Within PostgreSQL, you need to provide one database for Layman and one database for Micka. For Layman, you also need to provide one user LAYMAN_PG_USER who needs enough privileges to create new schemas in LAYMAN_PG_DBNAME database. The user also needs access to public
schema where PostGIS must be installed.
If you are using other DB schema than internal role service schema as role service, you need to provide all admin records.
Within QGIS Server, you do not need to provide anything special.
Within GeoServer, you need to provide either admin password GEOSERVER_ADMIN_PASSWORD, or one Layman user LAYMAN_GS_USER. If admin password is provided, Layman will create the Layman user automatically. URL path of the GeoServer must be /geoserver/
.
Within Redis, you need to provide two databases, one for Layman, second for Layman Test Client. Connection strings are defined by LAYMAN_REDIS_URL and LTC_REDIS_URL.
Within Micka, you need to provide one user with editor privileges, whose credentials are defined by CSW_BASIC_AUTHN.
After providing external dependencies there is time to provide internal dependencies (system-level, python-level and node.js-level dependencies). You can either use our docker and docker compose configuration to generate docker images that already provides internal dependencies, or you can provide internal dependencies by yourself (if you prefer not to use docker in production).
System-level dependencies includes
Pipenv is recommended tool for installing python-level dependencies. Both Pipfile and Pipfile.lock are located in docker/
directory.
Npm is recommended tool for installing node.js-level dependencies. Both package.json and package-lock.json are located in timgen/
directory.
Next you need to choose how you deploy Layman. As Layman is Flask application, check Flask's deployment options. Layman is safe to run with multiple WSGI Flask processes and with multiple Celery worker processes.
Configure Layman using environment settings. Demo configuration is a good starting point to setup Layman for production, however it needs to be adjusted carefully.
Last, start layman and necessary services:
timgen
docker compose service)layman_client
docker compose service)layman
docker compose service)celery_worker
docker compose service)Suitable for development only.
Before the first run:
# use dev settings
cp .env.dev .env
Now everything is ready to start:
# start all needed dockerized containers
make start-dev
Initial startup may take few minutes (download docker images, build it, run it). Wait until you see something like
[2020-12-13 09:57:00,968] INFO in __init__: Layman successfully started!
in log of layman_dev
container:
# see logs from Layman
docker logs -f layman_dev
Then visit http://localhost:8000/. You will see simple web client that interacts with REST API. You can also log in with automatically provided Wagtail user layman
and password laymanpwd
.
To stop running service run:
# stop all needed dockerized containers
make stop-dev
By default, docker run all containers as root user. It's possible to change it by defining UID_GID
permanent environment variable. First stop all running containers:
make stop-dev
The UID_GID
variable should look like "<user id>:<group id>"
of current user, e.g. UID_GID="1000:1000"
. As it should be permanent, you can solve it for example by adding following line to your ~/.bashrc
file:
export UID_GID="1000:1000"
and restart terminal. Verification:
$ echo $UID_GID
1000:1000
Then change ownership of some directories
make prepare-dirs
sudo chown -R 1000:1000 layman_data/
sudo chown -R 1000:1000 layman_data_test/
sudo chown -R 1000:1000 src/
sudo chown -R 1000:1000 tmp/
and restart layman
make start-dev
:warning: It will delete
Default values are defined in .env.test
# test related mainly to REST API endpoints of layers and maps
make test-static
make test-separated
# other tests
make test
For more information about tests, look at test-related documentation for developers.