search

Overv / openstreetmap-tile-server

Docker file for a minimal effort OpenStreetMap tile server
Apache License 2.0
1.2k stars 482 forks source link

Import not completing #409

Closed TheGor1lla closed 7 months ago

TheGor1lla commented 7 months ago

I deployed the tile-server to my k8s instance with a dedicated "import" deployment. When running it successfully completes the processing and clustering the tables by geometry, but afterwards just finishes.

+ sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 20 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf
2024-01-27 21:42:04  osm2pgsql version 1.6.0
2024-01-27 21:42:04  Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1)
2024-01-27 21:42:04  PostGIS version: 3.3
2024-01-27 21:42:04  Setting up table 'planet_osm_point'
2024-01-27 21:42:04  Setting up table 'planet_osm_line'
2024-01-27 21:42:04  Setting up table 'planet_osm_polygon'
2024-01-27 21:42:04  Setting up table 'planet_osm_roads'
2024-01-27 21:56:22  Reading input files done in 858s (14m 18s).
2024-01-27 21:56:22    Processed 51631763 nodes in 206s (3m 26s) - 251k/s
2024-01-27 21:56:22    Processed 8498226 ways in 420s (7m 0s) - 20k/s
2024-01-27 21:56:22    Processed 128432 relations in 232s (3m 52s) - 554/s
2024-01-27 21:56:23  Clustering table 'planet_osm_roads' by geometry...
2024-01-27 21:56:23  Clustering table 'planet_osm_point' by geometry...
2024-01-27 21:56:23  Clustering table 'planet_osm_line' by geometry...
2024-01-27 21:56:23  Clustering table 'planet_osm_polygon' by geometry...
2024-01-27 21:56:23  Done postprocessing on table 'planet_osm_nodes' in 0s
2024-01-27 21:56:23  Building index on table 'planet_osm_rels'
2024-01-27 21:56:23  Building index on table 'planet_osm_ways'

Deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: osm
  labels:
    app.kubernetes.io/name: osm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: osm
  template:
    metadata:
      labels:
        app.kubernetes.io/name: osm
    spec:
      serviceAccountName: default
      securityContext:
        runAsNonRoot: false
      containers:
        - name: osm
          image: overv/openstreetmap-tile-server
          env:
            - name: THREADS
              value: "20"
          args:  ["import"]
          securityContext:
            allowPrivilegeEscalation: false
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          resources:
            limits:
              cpu: 4000m
              memory: 2048Mi
            requests:
              cpu: 500m
              memory: 512Mi
          volumeMounts:
            - name: tiles
              mountPath: "/data/database/"
            - name: map-data
              mountPath: "/data/region.osm.pbf"
              subPath: "region.osm.pbf"
      volumes:
        - name: tiles
          persistentVolumeClaim:
            claimName: osm-data
        - name: map-data
          persistentVolumeClaim:
            claimName: osm-data-temp

Testing it locally on my docker node with a seperated instance, the import continues with: 

2024-01-27 21:49:54  Clustering table 'planet_osm_line' by geometry...
2024-01-27 21:49:54  Clustering table 'planet_osm_roads' by geometry...
2024-01-27 21:49:54  Clustering table 'planet_osm_polygon' by geometry...
2024-01-27 21:49:54  Clustering table 'planet_osm_point' by geometry...
2024-01-27 21:50:01  Creating geometry index on table 'planet_osm_roads'...
2024-01-27 21:50:03  Creating osm_id index on table 'planet_osm_roads'...
2024-01-27 21:50:03  Analyzing table 'planet_osm_roads'...
2024-01-27 21:50:03  Done postprocessing on table 'planet_osm_nodes' in 0s
2024-01-27 21:50:03  Building index on table 'planet_osm_ways'
2024-01-27 21:50:12  Creating geometry index on table 'planet_osm_point'...
2024-01-27 21:50:18  Creating osm_id index on table 'planet_osm_point'...
2024-01-27 21:50:21  Analyzing table 'planet_osm_point'...
2024-01-27 21:50:21  Building index on table 'planet_osm_rels'
2024-01-27 21:50:29  Creating geometry index on table 'planet_osm_line'...
2024-01-27 21:50:36  Creating osm_id index on table 'planet_osm_line'...
2024-01-27 21:50:39  Analyzing table 'planet_osm_line'...
2024-01-27 21:50:40  Creating geometry index on table 'planet_osm_polygon'...
2024-01-27 21:50:50  Creating osm_id index on table 'planet_osm_polygon'...
2024-01-27 21:50:53  Analyzing table 'planet_osm_polygon'...
2024-01-27 21:51:52  Done postprocessing on table 'planet_osm_ways' in 109s (1m 49s)
2024-01-27 21:51:52  Done postprocessing on table 'planet_osm_rels' in 3s
2024-01-27 21:51:52  All postprocessing on table 'planet_osm_point' done in 27s.
2024-01-27 21:51:52  All postprocessing on table 'planet_osm_line' done in 45s.
2024-01-27 21:51:52  All postprocessing on table 'planet_osm_polygon' done in 59s.
2024-01-27 21:51:52  All postprocessing on table 'planet_osm_roads' done in 9s.
2024-01-27 21:51:52  osm2pgsql took 375s (6m 15s) overall.
+ '[' -f /nodes/flat_nodes.bin ']'
+ '[' -f /data/style/indexes.sql ']'
+ sudo -u postgres psql -d gis -f /data/style/indexes.sql
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
+ chown -R renderer: /home/renderer/src/ /data/style/
+ '[' -f /data/style/scripts/get-external-data.py ']'
+ '[' -f /data/style/external-data.yml ']'
+ sudo -E -u renderer python3 /data/style/scripts/get-external-data.py -c /data/style/external-data.yml -D /data/style/data
INFO:root:Starting load of external data into database
INFO:root:Checking table simplified_water_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/simplified-water-polygons-split-3857.zip
INFO:root:  Download complete (23828793 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table water_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip
INFO:root:  Download complete (850086766 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table icesheet_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/antarctica-icesheet-polygons-3857.zip
INFO:root:  Download complete (52393123 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table icesheet_outlines
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/antarctica-icesheet-outlines-3857.zip
INFO:root:  Download complete (53057139 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table ne_110m_admin_0_boundary_lines_land
INFO:root:  Fetching https://naturalearth.s3.amazonaws.com/110m_cultural/ne_110m_admin_0_boundary_lines_land.zip
INFO:root:  Download complete (57325 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
+ sudo -u renderer touch /data/database/planet-import-complete
+ service postgresql stop
 * Stopping PostgreSQL 15 database server
   ...done.
+ exit 0

In the end the docker instance displays the map as expected, while the k8s just displays an empty map. k8s_logs.txt

Istador commented 7 months ago

Your import process or its database is likely oom killed.

resources:
  limits:
    cpu: 4000m
    memory: 2048Mi

2 GB maximum memory is way too low.

env:
  - name: THREADS
    value: "20"

Especially with such a high threads setting of 20, the memory is not enough.

(You give the pod 4 virtual CPUs, but want the import to run with 20 threads? Seems odd to me.)

Increase the memory and reduce the amount of threads. The default settings with 4 threads is meant for systems with a minimum of 4 GB of memory.

With 20 threads you'd need at least about 38 GB of memory not 2 GB, and you'd need to increase the maximum connections for the database, which exceeds the default limit of 250.

Your configuration is also missing something to increase the shm size.

volumeMounts:
  - name: tiles
    mountPath: "/data/database/"

Small misconception on your part, because you gave the volume mount for /data/database/ the name tiles: the tiles (image files) that are rendered later are placed at /data/tiles/ and not at /data/database/.

TheGor1lla commented 7 months ago

Thanks for the elaborate answer! I definitely screwed up on the resources. After fixing this, it works as expected.

Do you have any recommendations on setting the shm size?

Istador commented 7 months ago

Do you have any recommendations on setting the shm size?

Nope, haven't done that on k8s myself yet.

Quickly searched for a possible solution (untested) source 1, source 2

  volumeMounts:
  - mountPath: /dev/shm
    name: dev-shm
volumes:
- emptyDir:
    medium: Memory
    sizeLimit: 192Mi
  name: dev-shm
Piutschi commented 7 months ago

Hi everyone,

I am trying to deploy this in k8s as well. I have a very similar deployment but my pod is just restarting every time the import is complete. What am I doing wrong?

Cheers, Lena

Piutschi commented 7 months ago 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: osm-tile-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: osm
  template:
    metadata:
      labels:
        app: osm
    spec:
      restartPolicy: Always
      containers:
        - args:  ["import"]
          env:
            - name: DOWNLOAD_PBF
              value: "https://download.geofabrik.de/europe/austria-latest.osm.pbf"
            - name: DOWNLOAD_POLY
              value: "https://download.geofabrik.de/europe/austria.poly"
            #- name: THREADS
            #  value: "8"
          image: overv/openstreetmap-tile-server
          name: osm-tile-server
          securityContext:
            allowPrivilegeEscalation: false
          resources:
            limits:
              cpu: 24
              memory: 32769Mi
            requests:
              cpu: 16
              memory: 32769Mi
          volumeMounts:
          - name: nfs-opendatacube
            mountPath: /data/austria-latest.osm.pbf
            subPath: austria-latest.osm.pbf
          - name: database
            mountPath: "/data/tiles/"
          #- mountPath: /dev/shm
          #  name: dev-shm

      volumes:
          - name: nfs-opendatacube
            nfs:
              server: <-ip>
              path: <-path->
          - name: database
            persistentVolumeClaim:
              claimName: osm-data-volume-claim
          #- emptyDir:
          #    medium: Memory
          #    sizeLimit: 192Mi
          #    name: dev-shm
Istador commented 7 months ago 
restartPolicy: Always
containers:
  - args:  ["import"]

The import command only needs to run once. It exits when it is finished. After it successfully completed, the image needs to be started with the run command instead.

Your restartPolicy: Always always starts a new pod with import when it finishes or runs into an error (which happens when you call import again).

restartPolicy: Always only makes sense for the run command, where it is running as a service that, if it fails for whatever reason, should be restarted automatically.

Piutschi commented 7 months ago

Hi Istador,

thanks for the quick reply. I now start the import with an initcontainer and the run command in the main pod. Everything looks good but now the pod is stuck loading parameterizaton. Do you maybe know a solution for this? Thanks!

+ sudo -u renderer renderd -f -c /etc/renderd.conf
** INFO: 17:11:26.076: Rendering daemon started
** INFO: 17:11:26.076: Initialising request_queue
** INFO: 17:11:26.076: Parsing section renderd
** INFO: 17:11:26.076: Parsing render section 0
** INFO: 17:11:26.076: Parsing section mapnik
** INFO: 17:11:26.076: Parsing section default
** INFO: 17:11:26.076: config renderd: unix socketname=/run/renderd/renderd.sock
** INFO: 17:11:26.076: config renderd: num_threads=4
** INFO: 17:11:26.076: config renderd: num_slaves=0
** INFO: 17:11:26.076: config renderd: tile_dir=/var/cache/renderd/tiles
** INFO: 17:11:26.076: config renderd: stats_file=/run/renderd/renderd.stats
** INFO: 17:11:26.076: config mapnik:  plugins_dir=/usr/lib/mapnik/3.1/input
** INFO: 17:11:26.076: config mapnik:  font_dir=/usr/share/fonts
** INFO: 17:11:26.076: config mapnik:  font_dir_recurse=1
** INFO: 17:11:26.076: config renderd(0): Active
** INFO: 17:11:26.076: config renderd(0): unix socketname=/run/renderd/renderd.sock
** INFO: 17:11:26.076: config renderd(0): num_threads=4
** INFO: 17:11:26.076: config renderd(0): tile_dir=/var/cache/renderd/tiles
** INFO: 17:11:26.076: config renderd(0): stats_file=/run/renderd/renderd.stats
** INFO: 17:11:26.076: config map 0:   name(default) file(/home/renderer/src/openstreetmap-carto/mapnik.xml) uri(/tile/) htcp() host(localhost)
** INFO: 17:11:26.076: Initialising unix server socket on /run/renderd/renderd.sock
** INFO: 17:11:26.077: Renderd is using mapnik version 3.1.0
** INFO: 17:11:26.157: Running in foreground mode...
** INFO: 17:11:26.159: Loading parameterization function for
** INFO: 17:11:26.159: Loading parameterization function for
** INFO: 17:11:26.160: Loading parameterization function for
** INFO: 17:11:26.160: Loading parameterization function for