Slow import

[trohsb]

Hi,

I'm trying to do a planet import on my machine but the Way processing is quite slow.

Processing: Node(6616414k 1154.7k/s) Way(133713k 5.37k/s) Relation(0 0.00/s)

Processing the Nodes is CPU bound and processing the Ways is IO bound.

My osm2pgsql (version 1.2.1) command is:

osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/osm/openstreetmap-carto/openstreetmap-carto.lua -C 32000 --number-processes 8 -S /home/osm/openstreetmap-carto/openstreetmap-carto.style planet-210111.osm.pbf

Operating System: Ubuntu 20.04.1 LTS

PostgreSQL (version 12) settings:

shared_buffers = 4GB
autovacuum_work_mem = 2GB
effective_io_concurrency = 2
fsync = off
synchronous_commit = off
full_page_writes = off
checkpoint_timeout = 10min
max_wal_size = 1GB
min_wal_size = 80MB
checkpoint_completion_target = 0.9
autovacuum = off
effective_cache_size = 24GB

Hardware: CPU: Intel® Core™ i9-9900K RAM: 96GB 3200MHz SSD: Force Series™ MP510 1920GB M.2

My SSD gives the following speeds from hdparm:

sudo hdparm -Tt /dev/nvme0n1
/dev/nvme0n1:
 Timing cached reads:   38672 MB in  2.00 seconds = 19369.32 MB/sec
 Timing buffered disk reads: 4018 MB in  3.00 seconds = 1338.89 MB/sec

Is there anymore I can do to speed this up on my hardware?

[lonvia]

For planet imports you should use flatnode store together with -C 0.

[trohsb]

For planet imports you should use flatnode store together with -C 0.

That has vastly improved the Way processing speed.

Processing: Node(6616414k 2586.6k/s) Way(14692k 24.65k/s) Relation(0 0.00/s)

Thank you.