Segmentation fault during import

[Eric64534]

Any ideas?? Here's the log:

tech@4YF0GT3:/mnt/c/Windows/system32$ docker volume create osm-data osm-data tech@4YF0GT3:/mnt/c/Windows/system32$ docker run \ -v /mnt/e/planet-231002.osm.pbf:/data/region.osm.pbf \ -v osm-data:/data/database/ \ -e "FLAT_NODES=enabled" \ -e "OSM2PGSQL_EXTRA_ARGS=-C 8192" \ -e THREADS=18 \ overv/openstreetmap-tile-server \ import Unable to find image 'overv/openstreetmap-tile-server:latest' locally latest: Pulling from overv/openstreetmap-tile-server 6b851dcae6ca: Pull complete bb071db28a33: Pull complete a862358c8622: Pull complete 57ea4a769709: Pull complete 7637307ae531: Pull complete fa1264c2a6e6: Pull complete 5591dd38bd35: Pull complete c82b1fb2ea15: Pull complete 47ecf7831eac: Pull complete 18125c639464: Pull complete 2b0335ae4896: Pull complete 4c9807a2e2f3: Pull complete 649b76086468: Pull complete b52cd0b2e16e: Pull complete 9874a5bac373: Pull complete acd614dbc6df: Pull complete 8a950b4e43d0: Pull complete 269690f5a06c: Pull complete 19ed3f017280: Pull complete 4b7f8bb34a12: Pull complete 25b1f3b221b7: Pull complete 0056ac6bf411: Pull complete 57110a116d22: Pull complete 208240568089: Pull complete Digest: sha256:b6a79da39b6d0758368f7c62d22e49dd3ec59e78b194a5ef9dee2723b1f3fa79 Status: Downloaded newer image for overv/openstreetmap-tile-server:latest ++ ls -A /data/style/

• '[' '!' '' ']'

• mv /home/renderer/src/openstreetmap-carto-backup/CARTOGRAPHY.md /home/renderer/src/openstreetmap-carto-backup/CHANGELOG.md /home/renderer/src/openstreetmap-carto-backup/CODE_OF_CONDUCT.md /home/renderer/src/openstreetmap-carto-backup/CONTRIBUTING.md /home/renderer/src/openstreetmap-carto-backup/DOCKER.md /home/renderer/src/openstreetmap-carto-backup/Dockerfile /home/renderer/src/openstreetmap-carto-backup/Dockerfile.db /home/renderer/src/openstreetmap-carto-backup/Dockerfile.import /home/renderer/src/openstreetmap-carto-backup/INSTALL.md /home/renderer/src/openstreetmap-carto-backup/LICENSE.txt /home/renderer/src/openstreetmap-carto-backup/README.md /home/renderer/src/openstreetmap-carto-backup/RELEASES.md /home/renderer/src/openstreetmap-carto-backup/USECASES.md /home/renderer/src/openstreetmap-carto-backup/docker-compose.yml /home/renderer/src/openstreetmap-carto-backup/external-data.yml /home/renderer/src/openstreetmap-carto-backup/indexes.sql /home/renderer/src/openstreetmap-carto-backup/indexes.yml /home/renderer/src/openstreetmap-carto-backup/openstreetmap-carto.lua /home/renderer/src/openstreetmap-carto-backup/openstreetmap-carto.style /home/renderer/src/openstreetmap-carto-backup/patterns /home/renderer/src/openstreetmap-carto-backup/preview.png /home/renderer/src/openstreetmap-carto-backup/project.mml /home/renderer/src/openstreetmap-carto-backup/road-colors.yaml /home/renderer/src/openstreetmap-carto-backup/scripts /home/renderer/src/openstreetmap-carto-backup/style /home/renderer/src/openstreetmap-carto-backup/symbols /data/style/

• '[' '!' -f /data/style/mapnik.xml ']'

• cd /data/style/

• carto project.mml Warning: style/landcover.mss:609:4 line-offset is unstable. It may change in the future. ... additional warnings removed ...

• '[' import == import ']'

• mkdir -p /data/database/postgres/

• chown renderer: /data/database/

• chown -R postgres: /var/lib/postgresql /data/database/postgres/

• '[' '!' -f /data/database/postgres/PG_VERSION ']'

• createPostgresConfig

• cp /etc/postgresql/15/main/postgresql.custom.conf.tmpl /etc/postgresql/15/main/conf.d/postgresql.custom.conf

• sudo -u postgres echo 'autovacuum = on'

• cat /etc/postgresql/15/main/conf.d/postgresql.custom.conf

  Suggested minimal settings from

  https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB min_wal_size = 1GB

max_wal_size = 2GB # Overridden below

maintenance_work_mem = 256MB

Suggested settings from

https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250 temp_buffers = 32MB work_mem = 128MB wal_buffers = 1024kB wal_writer_delay = 500ms commit_delay = 10000

checkpoint_segments = 60 # unrecognized in psql 10.7.1

max_wal_size = 2880MB random_page_cost = 1.1 track_activity_query_size = 16384 autovacuum_vacuum_scale_factor = 0.05 autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*' autovacuum = on

• service postgresql start
  • Starting PostgreSQL 15 database server ...done.
• sudo -u postgres createuser renderer
• sudo -u postgres createdb -E UTF8 -O renderer gis
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;' CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;' CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;' ALTER TABLE
• sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;' ALTER TABLE
• setPostgresPassword
• sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\''' ALTER ROLE
• '[' '!' -f /data/region.osm.pbf ']'
• '[' -n '' ']'
• '[' disabled == enabled ']'
• '[' disabled == 1 ']'
• '[' -f /data/region.poly ']'
• '[' enabled == enabled ']'
• OSM2PGSQL_EXTRA_ARGS='-C 8192 --flat-nodes /data/database/flat_nodes.bin'
• sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 18 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/database/flat_nodes.bin 2023-10-23 01:56:34 osm2pgsql version 1.6.0 2023-10-23 01:56:34 Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1) 2023-10-23 01:56:34 PostGIS version: 3.3 2023-10-23 01:56:34 Setting up table 'planet_osm_point' 2023-10-23 01:56:34 Setting up table 'planet_osm_line' 2023-10-23 01:56:35 Setting up table 'planet_osm_polygon' 2023-10-23 01:56:35 Setting up table 'planet_osm_roads' 2023-10-25 22:14:48 Reading input files done in 245893s (68h 18m 13s). 2023-10-25 22:14:48 Processed 8658483163 nodes in 3446s (57m 26s) - 2513k/s 2023-10-25 22:14:48 Processed 971692934 ways in 160275s (44h 31m 15s) - 6k/s 2023-10-25 22:14:48 Processed 11340554 relations in 82172s (22h 49m 32s) - 138/s 2023-10-25 22:14:48 Clustering table 'planet_osm_point' by geometry... 2023-10-25 22:14:48 Clustering table 'planet_osm_polygon' by geometry... 2023-10-25 22:14:48 Clustering table 'planet_osm_roads' by geometry... 2023-10-25 22:14:48 Clustering table 'planet_osm_line' by geometry... 2023-10-25 22:14:51 Done postprocessing on table 'planet_osm_nodes' in 0s 2023-10-25 22:14:51 Building index on table 'planet_osm_rels' 2023-10-25 22:14:51 Building index on table 'planet_osm_ways' /run.sh: line 130: 76 Segmentation fault sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/${NAME_LUA:-openstreetmap-carto.lua} --number-processes ${THREADS:-4} -S /data/style/${NAME_STYLE:-openstreetmap-carto.style} /data/region.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-} tech@4YF0GT3:/mnt/c/Windows/system32$

[Istador]

18 THREADS seems too high, that'd result in up to (3 + (1+18) * 14) = 269 connections, whereas max_connections in postgresql.custom.conf.tmpl is configured to 250.

Also resulting from this: does your system really have 269 * 128 MB + 8192 MB = 42 GB of memory?

See #330

[Eric64534]

Thank you so much for the reply! Yes, this system has an Intel i9-10900X (20 threads) and 64 GB of RAM. I'll try again with less threads.

Best, Eric

[Eric64534]

Well, I got a little further. Now it says I ran out of disk space. The osm-data volume resides on an external USB3-to-SATA adapter with a 1.9TB SSD. With everything idle now (after import aborted), the volume says it is 926 GB and windows says the disk (e:) has 760 GB free. My boot drive (c:) has 500 GB free. Any assistance would be greatly appreciated!

tech@4YF0GT3:/mnt/c/Users/ehughes$ docker run \ -v /mnt/e/planet-231002.osm.pbf:/data/region.osm.pbf \ -v osm-data:/data/database/ \ -e "FLAT_NODES=enabled" \ -e "OSM2PGSQL_EXTRA_ARGS=-C 8192" \ -e THREADS=10 \ overv/openstreetmap-tile-server \ import

• '[' import == import ']'
• mkdir -p /data/database/postgres/
• chown renderer: /data/database/
• chown -R postgres: /var/lib/postgresql /data/database/postgres/
• '[' '!' -f /data/database/postgres/PG_VERSION ']'
• createPostgresConfig
• cp /etc/postgresql/15/main/postgresql.custom.conf.tmpl /etc/postgresql/15/main/conf.d/postgresql.custom.conf
• sudo -u postgres echo 'autovacuum = on'
• cat /etc/postgresql/15/main/conf.d/postgresql.custom.conf

  Suggested minimal settings from

  https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB min_wal_size = 1GB

max_wal_size = 2GB # Overridden below

maintenance_work_mem = 256MB

Suggested settings from

https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250 temp_buffers = 32MB work_mem = 128MB wal_buffers = 1024kB wal_writer_delay = 500ms commit_delay = 10000

checkpoint_segments = 60 # unrecognized in psql 10.7.1

max_wal_size = 2880MB random_page_cost = 1.1 track_activity_query_size = 16384 autovacuum_vacuum_scale_factor = 0.05 autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*' autovacuum = on

• service postgresql start
  • Starting PostgreSQL 15 database server ...done.
• sudo -u postgres createuser renderer
• sudo -u postgres createdb -E UTF8 -O renderer gis
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;' CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;' CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;' ALTER TABLE
• sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;' ALTER TABLE
• setPostgresPassword
• sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '\''renderer'\''' ALTER ROLE
• '[' '!' -f /data/region.osm.pbf ']'
• '[' -n '' ']'
• '[' disabled == enabled ']'
• '[' disabled == 1 ']'
• '[' -f /data/region.poly ']'
• '[' enabled == enabled ']'
• OSM2PGSQL_EXTRA_ARGS='-C 8192 --flat-nodes /data/database/flat_nodes.bin'
• sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 10 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/database/flat_nodes.bin 2023-10-26 18:54:07 osm2pgsql version 1.6.0 2023-10-26 18:54:07 Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1) 2023-10-26 18:54:07 PostGIS version: 3.3 2023-10-26 18:54:08 Setting up table 'planet_osm_point' 2023-10-26 18:54:08 Setting up table 'planet_osm_line' 2023-10-26 18:54:08 Setting up table 'planet_osm_polygon' 2023-10-26 18:54:08 Setting up table 'planet_osm_roads' 2023-10-29 18:59:03 Reading input files done in 259495s (72h 4m 55s). 2023-10-29 18:59:03 Processed 8658483163 nodes in 3478s (57m 58s) - 2490k/s 2023-10-29 18:59:03 Processed 971692934 ways in 175068s (48h 37m 48s) - 6k/s 2023-10-29 18:59:03 Processed 11340554 relations in 80949s (22h 29m 9s) - 140/s 2023-10-29 18:59:03 Clustering table 'planet_osm_polygon' by geometry... 2023-10-29 18:59:03 Clustering table 'planet_osm_roads' by geometry... 2023-10-29 18:59:03 Clustering table 'planet_osm_line' by geometry... 2023-10-29 18:59:03 Clustering table 'planet_osm_point' by geometry... 2023-10-29 18:59:06 Done postprocessing on table 'planet_osm_nodes' in 0s 2023-10-29 18:59:06 Building index on table 'planet_osm_rels' 2023-10-29 18:59:06 Building index on table 'planet_osm_ways' 2023-10-29 22:22:17 ERROR: Database error: ERROR: could not extend file "base/16385/7440649.3": No space left on device HINT: Check free disk space.

[Istador]

If I remember correctly, the docker settings on Windows might impose a maximum volume size limit? IDK.

Likely it cleaned up the files from the step it was at after the error happened and that is why there is more free space now than when the error occured. According to the logs the data import took 3 days, and it failed 3 hours later when creating the indicies for the planet_osm_ways table.

According to https://wiki.openstreetmap.org/wiki/Planet.osm the uncompressed data is 1.8 TB, so 1.9 TB for the whole planet might not be enough? But the import doesn't import all the data, so it should be less, though the database also needs space for its own and for indicies and stuff. I think 1.9 TB should be enough if there is nothing else on the disk. What I read from about two years ago the size was about 1.2 TB imported.

[Eric64534]

Just to follow-up, the issue was indeed Windows (and/or docker) imposing a volume size limit of 1TB. I tried all sorts of things to increase it, but nothing worked. I finally gave up and moved to Ubuntu and everything worked. Really appreciate your help!!