Docker development environment

[hiddewie]

From https://github.com/gravitystorm/openstreetmap-carto/blob/master/DOCKER.md and https://github.com/cyclosm/cyclosm-cartocss-style.

This PR supplies a development environment based on Docker without additional system dependencies.

This setup greatly simplifies setting up a development environment. Most developers have Docker running on their system, while a Postgres (with extensions), NodeJS with YAML and many dependencies, C++ and Make tools, installed system fonts and Python 3 is more difficult to get right.

In addition, the SETUP.md contains many manual steps that need to be done to set a simple development environment.

With a Docker setup, the development environment is down to 2 to import the data and run Kosmtik (the database dependency will build & start automatically). Starting is as simple as

# download .osm.pbf file
docker compose up import
docker compose up kosmtik

Documentation in https://github.com/OpenRailwayMap/OpenRailwayMap-CartoCSS/pull/88/files#diff-0040e607148899aef28b61411ebc22c4266046b5035b7bd2a45424ff2027521f.

If such a development environment is not wanted, feel free to close this PR (I will keep my fork/branch for developing locally).

[hiddewie]

I added the osmium command as well to give a huge speed boost to the import.

[besentv]

I'm trying to get this running by following the SETUP.md but for some reason I don't see any ORM style lines in the Kosmtik viewer. Any ideas why that might be?

The import command shows this error, but I'm not sure if it's related: openrailwaymap-cartocss-import-1 | Osm2pgsql failed due to ERROR: Open failed for 'filtered.osm.pbf': No such file or directory

[hiddewie]

@besentv Something probably went wrong importing the data. Can you post the full output of the import container? The file data.osm.pbf is filtered into filtered.osm.pbf during the import.

[besentv]

Sure, this is the output:

docker compose up import                                                                                                         ✔ 
[+] Running 2/0
 ✔ Container openrailwaymap-cartocss-db-1      Running                                                                                                                                                      0.0s 
 ✔ Container openrailwaymap-cartocss-import-1  Created                                                                                                                                                      0.0s 
Attaching to openrailwaymap-cartocss-db-1, openrailwaymap-cartocss-import-1
openrailwaymap-cartocss-import-1  | Waiting for PostgreSQL to be running
openrailwaymap-cartocss-import-1  | PostgreSQL running
openrailwaymap-cartocss-import-1  | CREATE EXTENSION
openrailwaymap-cartocss-import-1  | NOTICE:  extension "postgis" already exists, skipping
openrailwaymap-cartocss-import-1  | CREATE EXTENSION
openrailwaymap-cartocss-import-1  | NOTICE:  extension "hstore" already exists, skipping
openrailwaymap-cartocss-import-1  | PBF error: invalid BlobHeader size (> max_blob_header_size)
openrailwaymap-cartocss-import-1  | osm2pgsql version 1.2.2 (64 bit id space)
openrailwaymap-cartocss-import-1  | 
openrailwaymap-cartocss-import-1  | Allocating memory for dense node cache
openrailwaymap-cartocss-import-1  | Allocating dense node cache in one big chunk
openrailwaymap-cartocss-import-1  | Allocating memory for sparse node cache
openrailwaymap-cartocss-import-1  | Sharing dense sparse
openrailwaymap-cartocss-import-1  | Node-cache: cache=512MB, maxblocks=8192*65536, allocation method=11
openrailwaymap-cartocss-import-1  | Mid: pgsql, cache=512
openrailwaymap-cartocss-import-1  | 
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_nodes
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_ways
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_rels
openrailwaymap-cartocss-import-1  | Using lua based tag processing pipeline with script setup/openstreetmap-carto.lua
openrailwaymap-cartocss-import-1  | Using projection SRS 3857 (Spherical Mercator)
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_point
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_line
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_polygon
openrailwaymap-cartocss-import-1  | Setting up table: planet_osm_roads
openrailwaymap-cartocss-import-1  | 
openrailwaymap-cartocss-import-1  | Reading in file: filtered.osm.pbf
openrailwaymap-cartocss-import-1  | Using PBF parser.
openrailwaymap-cartocss-import-1  | 
openrailwaymap-cartocss-import-1  | node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
openrailwaymap-cartocss-import-1  | Osm2pgsql failed due to ERROR: Open failed for 'filtered.osm.pbf': No such file or directory
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE FUNCTION
openrailwaymap-cartocss-import-1  | CREATE INDEX
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | CREATE VIEW
openrailwaymap-cartocss-import-1  | SELECT 0
openrailwaymap-cartocss-import-1 exited with code 0

And my input file is this.

data.osm.pbf.zip

[hiddewie]

I get the same result locally. It reports an invalid header size for your PBF file.

PBF error: invalid BlobHeader size (> max_blob_header_size)

[besentv]

Ohhh I provided XML instead of PBF, my bad. :-)

[hiddewie]

On Ubuntu, one has to execute docker-compose instead of docker compose. However, I do not think that it is necessary to write this down in the instructions.

The latest few Ubuntu versions support the latest Docker versions, with Docker Compose version 2, which is embedded into the docker CLI.

I'm running on Ubuntu on multiple PCs, and none of them need docker-compose anymore.

[hiddewie]

Oh this pull request is very old, I cannot get the Ubuntu bionic image to work anymore. Updated to Debian 12 Bookworm.

[Nakaner]

I will re-run my test tomorrow (I run it this afternoon but maybe the Bionic image was cached by my computer in the office).