SQL Error: pq: unexpected message type 0x2A during COPY from stdin

Context

If importing the full planet data or Illinois from http://download.geofabrik.de/north-america.html it fails with "message type 0x2A during COPY from stdin" which appears to be a generic error from PostgreSQL that is doesn't like something about what it received via stdin. However, importing Iowa (and possibly others) from http://download.geofabrik.de/north-america.html does not have any issues.

Expected Behavior

Expected to complete without issue

Actual Behavior

[2019-10-26T13:50:15-06:00] 1:27:00 [progress] 1h27m0s C: 1846000/s (5538063680) N: 24700/s (74291108) W: 257100/s (570912000) R: 0/s (0) [2019-10-26T13:51:15-06:00] 1:28:00 [progress] 1h28m0s C: 1846000/s (5538063680) N: 24700/s (74291108) W: 257000/s (586176000) R: 0/s (0) [2019-10-26T13:52:15-06:00] 1:29:00 [progress] 1h29m0s C: 1846000/s (5538063680) N: 24700/s (74291108) W: 257000/s (601560000) R: 0/s (0) [2019-10-26T13:53:15-06:00] 1:30:00 [progress] 1h30m0s C: 1846000/s (5538063680) N: 24700/s (74291108) W: 255800/s (614164096) R: 200110/s (2316258) [2019-10-26T13:53:31-06:00] 1:30:15 [progress] 1h30m15s C: 1846000/s (5538063680) N: 24700/s (74291108) W: 255800/s (614164096) R: 490000/s (5671748) [2019-10-26T13:53:31-06:00] 1:30:16 [step] Finished: Reading OSM data in 1h30m16.037847208s [2019-10-26T13:53:32-06:00] 1:30:16 [step] Starting: Importing OSM data [2019-10-26T13:53:32-06:00] 1:30:16 [step] Starting: Writing OSM data [2019-10-26T13:54:30-06:00] 1:31:15 [fatal] bulk insert into "osm_admin": SQL Error: pq: unexpected message type 0x2A during COPY from stdin in query COPY "osm_import"."osm_admin" ("osm_id", "geometry", "name", "type", "admin_level") FROM STDIN

Possible Fix

Steps to Reproduce

Download latest. In my case it was https://planet.openstreetmap.org/planet/2019/planet-191021.osm.bz2
imposm import \ -dbschema-import osm_import \ -mapping /mnt/mapdata/imposm-0.8.1-linux-x86-64/mapping.json \ -srid 4326 \ -read planet-latest.osm.pbf \ -cachedir /mnt/mapdata/cache \ -diff -diffdir /mnt/mapdata/diff \ -connection postgis://dbuser:dbpassword@dbserver:5432/mydb \ -write

Context

I am trying to import all planet data for use in GeoServer. I would like imposm3 to eventually maintain hourly updates.

Your Environment

Version used: 0.8.1 release binary
Environment name and version: AWS Aurora PostgreSQL
Server type and version: AWS EC2 t3.2xlarge
Operating System and version: CentOS 7 (latest patches)

AWS Aurora PostgreSQL

SSL connections ? You can connect to a DB instance in your Amazon Aurora PostgreSQL DB cluster using the same tools that you use to connect to a PostgreSQL database. As part of this, you use the same public key for Secure Sockets Layer (SSL) connections. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Connecting.html#Aurora.Connecting.AuroraPostgreSQL

If SSL then check the readme TLS/SSL support part ...

I am not running TLS/SSL that I am aware of. I don't think there is anything wrong with my setup as I can import in Iowa without issue but Illinois fails (see below). I had a similar problem with osm2pgsql with an older version until I updated to a newer version. I suspect that there is something new in the data that is causing an error on the COPY command. Possibly something not being escaped properly?

# imposm import \
>     -dbschema-import osm_import \
>     -mapping /mnt/mapdata/imposm-0.8.1-linux-x86-64/mapping.json \
>     -srid 4326 \
>     -read illinois-latest.osm.pbf \
>     -cachedir /mnt/mapdata/cache \
>     -diff -diffdir /mnt/mapdata/diff \
>     -connection postgis://dbuser:dbpassword@dbserver:5432/mydb \
>     -write
[2019-10-30T14:49:06-06:00] 0:00:00 [step] Starting: Imposm
[2019-10-30T14:49:06-06:00] 0:00:00 [step] Starting: Reading OSM data
[2019-10-30T14:49:06-06:00] 0:00:00 [info] reading illinois-latest.osm.pbf with data till 2019-10-24 14:19:03 -0600 MDT
[2019-10-30T14:49:34-06:00] 0:00:28 [progress]    28s C:       0/s (22022044) N:       0/s (142983) W:       0/s (2445195) R:      0/s (11343)
[2019-10-30T14:49:34-06:00] 0:00:28 [step] Finished: Reading OSM data in 28.848812325s
[2019-10-30T14:49:35-06:00] 0:00:29 [step] Starting: Importing OSM data
[2019-10-30T14:49:35-06:00] 0:00:29 [step] Starting: Writing OSM data
[2019-10-30T14:50:33-06:00] 0:01:27 [fatal] bulk insert into "osm_transport_points": SQL Error: pq: unexpected message type 0x2A during COPY from stdin in query COPY "osm_import"."osm_transport_points" ("osm_id", "geometry", "name", "type", "ref") FROM STDIN

# imposm import \
>     -dbschema-import osm_import \
>     -mapping /mnt/mapdata/imposm-0.8.1-linux-x86-64/mapping.json \
>     -srid 4326 \
>     -read iowa-latest.osm.pbf \
>     -cachedir /mnt/mapdata/cache \
>     -diff -diffdir /mnt/mapdata/diff \
>    -connection postgis://dbuser:dbpassword@dbserver:5432/mydb \
>     -write
[2019-10-30T14:43:32-06:00] 0:00:00 [step] Starting: Imposm
[2019-10-30T14:43:32-06:00] 0:00:00 [step] Starting: Reading OSM data
[2019-10-30T14:43:32-06:00] 0:00:00 [info] reading iowa-latest.osm.pbf with data till 2019-10-24 14:19:03 -0600 MDT
[2019-10-30T14:43:46-06:00] 0:00:13 [progress]    13s C:       0/s (12566724) N:       0/s (44778) W:       0/s (741456) R:      0/s (6310)
[2019-10-30T14:43:46-06:00] 0:00:13 [step] Finished: Reading OSM data in 13.914779258s
[2019-10-30T14:43:47-06:00] 0:00:14 [step] Starting: Importing OSM data
[2019-10-30T14:43:47-06:00] 0:00:14 [step] Starting: Writing OSM data
[2019-10-30T14:44:15-06:00] 0:00:43 [progress]    28s C:       0/s ( 0.0%) N:       0/s (100.0%) W:       0/s (100.0%) R:      0/s (100.0%)
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Finished: Writing OSM data in 29.058726268s
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Starting: Creating generalized tables
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Starting: Generalizing osm_roads into osm_roads_gen1
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Starting: Generalizing osm_waterways into osm_waterways_gen1
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Starting: Generalizing osm_waterareas into osm_waterareas_gen1
[2019-10-30T14:44:16-06:00] 0:00:43 [step] Starting: Generalizing osm_landusages into osm_landusages_gen1
[2019-10-30T14:44:16-06:00] 0:00:44 [step] Finished: Generalizing osm_waterareas into osm_waterareas_gen1 in 653.848263ms
[2019-10-30T14:44:16-06:00] 0:00:44 [step] Finished: Generalizing osm_landusages into osm_landusages_gen1 in 654.224577ms
[2019-10-30T14:44:19-06:00] 0:00:47 [step] Finished: Generalizing osm_roads into osm_roads_gen1 in 3.196827024s
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Finished: Generalizing osm_waterways into osm_waterways_gen1 in 8.572533068s
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Starting: Generalizing osm_landusages into osm_landusages_gen0
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Starting: Generalizing osm_waterareas into osm_waterareas_gen0
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Starting: Generalizing osm_roads into osm_roads_gen0
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Starting: Generalizing osm_waterways into osm_waterways_gen0
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Finished: Generalizing osm_landusages into osm_landusages_gen0 in 13.620513ms
[2019-10-30T14:44:24-06:00] 0:00:52 [step] Finished: Generalizing osm_waterareas into osm_waterareas_gen0 in 18.131096ms
[2019-10-30T14:44:26-06:00] 0:00:53 [step] Finished: Generalizing osm_roads into osm_roads_gen0 in 1.357325745s
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Generalizing osm_waterways into osm_waterways_gen0 in 1.767535951s
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Creating generalized tables in 10.34060258s
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry indices
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_barrierpoints
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_buildings
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_aeroways
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_admin
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_roads
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_transport_areas
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_waterways
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_amenities
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_transport_areas in 217.696717ms
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_waterareas
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_aeroways in 217.785837ms
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_housenumbers
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_admin in 377.647091ms
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_barrierways
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_barrierpoints in 377.791456ms
[2019-10-30T14:44:26-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_transport_points
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_amenities in 513.421955ms
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_housenumbers_interpolated
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_housenumbers_interpolated in 249.822374ms
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_places
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Finished: Creating geometry index on osm_housenumbers in 571.654184ms
[2019-10-30T14:44:27-06:00] 0:00:54 [step] Starting: Creating geometry index on osm_landusages
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Finished: Creating geometry index on osm_barrierways in 548.652649ms
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Starting: Creating OSM id index on osm_waterways_gen1
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Finished: Creating geometry index on osm_places in 326.703061ms
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Starting: Creating OSM id index on osm_landusages_gen1
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Finished: Creating OSM id index on osm_landusages_gen1 in 190.963135ms
[2019-10-30T14:44:27-06:00] 0:00:55 [step] Starting: Creating geometry index on osm_landusages_gen1
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Finished: Creating geometry index on osm_landusages_gen1 in 159.99991ms
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Starting: Creating OSM id index on osm_landusages_gen0
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Finished: Creating OSM id index on osm_waterways_gen1 in 798.654987ms
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Starting: Creating geometry index on osm_waterways_gen1
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Finished: Creating OSM id index on osm_landusages_gen0 in 284.007607ms
[2019-10-30T14:44:28-06:00] 0:00:55 [step] Starting: Creating geometry index on osm_landusages_gen0
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Finished: Creating geometry index on osm_landusages_gen0 in 127.861128ms
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Starting: Creating OSM id index on osm_waterareas_gen1
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Finished: Creating OSM id index on osm_waterareas_gen1 in 172.815681ms
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Starting: Creating geometry index on osm_waterareas_gen1
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Finished: Creating geometry index on osm_waterareas_gen1 in 80.505753ms
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Starting: Creating OSM id index on osm_waterareas_gen0
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Finished: Creating OSM id index on osm_waterareas_gen0 in 154.403292ms
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Starting: Creating geometry index on osm_waterareas_gen0
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Finished: Creating geometry index on osm_waterareas_gen0 in 70.457743ms
[2019-10-30T14:44:28-06:00] 0:00:56 [step] Starting: Creating OSM id index on osm_roads_gen0
[2019-10-30T14:44:29-06:00] 0:00:56 [step] Finished: Creating geometry index on osm_transport_points in 2.128714264s
[2019-10-30T14:44:29-06:00] 0:00:56 [step] Starting: Creating OSM id index on osm_roads_gen1
[2019-10-30T14:44:29-06:00] 0:00:56 [step] Finished: Creating OSM id index on osm_roads_gen0 in 461.567703ms
[2019-10-30T14:44:29-06:00] 0:00:56 [step] Starting: Creating geometry index on osm_roads_gen0
[2019-10-30T14:44:29-06:00] 0:00:57 [step] Finished: Creating OSM id index on osm_roads_gen1 in 338.379764ms
[2019-10-30T14:44:29-06:00] 0:00:57 [step] Starting: Creating geometry index on osm_roads_gen1
[2019-10-30T14:44:34-06:00] 0:01:01 [step] Finished: Creating geometry index on osm_waterareas in 7.552660125s
[2019-10-30T14:44:34-06:00] 0:01:01 [step] Starting: Creating OSM id index on osm_waterways_gen0
[2019-10-30T14:44:34-06:00] 0:01:02 [step] Finished: Creating OSM id index on osm_waterways_gen0 in 511.448517ms
[2019-10-30T14:44:34-06:00] 0:01:02 [step] Starting: Creating geometry index on osm_waterways_gen0
[2019-10-30T14:44:35-06:00] 0:01:03 [step] Finished: Creating geometry index on osm_landusages in 8.181140938s
[2019-10-30T14:44:35-06:00] 0:01:03 [step] Finished: Creating geometry index on osm_roads_gen0 in 6.308831603s
[2019-10-30T14:44:35-06:00] 0:01:03 [step] Finished: Creating geometry index on osm_roads_gen1 in 6.323877439s
[2019-10-30T14:44:37-06:00] 0:01:04 [step] Finished: Creating geometry index on osm_waterways in 10.710751983s
[2019-10-30T14:44:37-06:00] 0:01:04 [step] Finished: Creating geometry index on osm_waterways_gen1 in 9.020719671s
[2019-10-30T14:44:42-06:00] 0:01:09 [step] Finished: Creating geometry index on osm_waterways_gen0 in 7.213771183s
[2019-10-30T14:44:42-06:00] 0:01:10 [step] Finished: Creating geometry index on osm_buildings in 16.12472137s
[2019-10-30T14:44:48-06:00] 0:01:15 [step] Finished: Creating geometry index on osm_roads in 21.723818291s
[2019-10-30T14:44:48-06:00] 0:01:15 [step] Finished: Creating geometry indices in 21.724061405s
[2019-10-30T14:44:48-06:00] 0:01:15 [step] Finished: Importing OSM data in 1m1.123505175s
[2019-10-30T14:44:48-06:00] 0:01:15 [step] Finished: Imposm in 1m15.873329991s

@olt any thoughts on this? I would really love to use this tool.

@olt I have noticed that older versions of osm2pgsql seem to have the same issue but newer i.e. 1.2.0 seem to have resolved it.

osm2pgsql version 0.94.0 (64 bit id space)

Password: Using built-in tag processing pipeline Using projection SRS 4326 (Latlong) Setting up table: planet_osm_point Setting up table: planet_osm_line Setting up table: planet_osm_polygon Setting up table: planet_osm_roads Allocating memory for dense node cache Allocating dense node cache in one big chunk Allocating memory for sparse node cache Sharing dense sparse Node-cache: cache=480000MB, maxblocks=7680000*65536, allocation method=11 Mid: loading persistent node cache from /mnt/mapdata/flat_nodes.bin Mid: pgsql, cache=480000 Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels

Reading in file: planet-latest.osm.pbf Using PBF parser. Processing: Node(5538063k 3507.3k/s) Way(613964k 164.87k/s) Relation(0 0.00/s)COPY_END for COPY planet_osm_ways FROM STDIN; failed: ERROR: unexpected message type 0x2A during COPY from stdin CONTEXT: COPY planet_osm_ways, line 1

Error occurred, cleaning up

If importing ... Illinois from http://download.geofabrik.de/north-america.html it fails with "message type 0x2A during COPY from stdin"

strange ... I can't replicate with

illinois-191101.osm.pbf ( imho: the illinois-latest.osm.pbf is changing every day; so it is not perfect for testing )
local (dockerized) PostGIS server

Can you re-test with local postgres server? ( or with docker-postgis ? ) Can you share your psql output? for example this is my output

root@75c3d31bd31e:/tools# psql
psql (12.0 (Debian 12.0-2.pgdg90+1))
Type "help" for help.

app=#

imposm3 - illinois-191101.osm.pbf log:

root@75c3d31bd31e:/tools# wget http://download.geofabrik.de/north-america/us/illinois-191101.osm.pbf
--2019-11-03 01:28:12--  http://download.geofabrik.de/north-america/us/illinois-191101.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 88.99.105.98, 88.99.142.44
Connecting to download.geofabrik.de (download.geofabrik.de)|88.99.105.98|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 178456604 (170M) [application/octet-stream]
Saving to: ‘illinois-191101.osm.pbf’

illinois-191101.osm.pbf                            100%[================================================================================================================>] 170.19M  3.24MB/s    in 69s     

2019-11-03 01:29:22 (2.46 MB/s) - ‘illinois-191101.osm.pbf’ saved [178456604/178456604]

root@75c3d31bd31e:/tools# 
root@75c3d31bd31e:/tools# PGHOST=$PGHOSTADDR
root@75c3d31bd31e:/tools# export PGHOST
root@75c3d31bd31e:/tools# unset PGHOSTADDR
root@75c3d31bd31e:/tools# 
root@75c3d31bd31e:/tools# readonly PG_CONNECT="postgis://${PGUSER}:${PGPASSWORD}@${PGHOST}:${PGPORT}/${PGDATABASE}" 
root@75c3d31bd31e:/tools# readonly PG_CONNECT="postgis://${PGUSER}:${PGPASSWORD}@${PGHOST}:${PGPORT}/${PGDATABASE}" 
bash: PG_CONNECT: readonly variable
root@75c3d31bd31e:/tools# /tools/imposm-0.8.1-linux-x86-64/imposm import \
>    -mapping /tools/imposm-0.8.1-linux-x86-64/mapping.json \
>    -read illinois-191101.osm.pbf \
>    -write    \
>    -optimize  \
>    -srid 4326 \
>    -overwritecache \
>    -deployproduction \
>    -connection $PG_CONNECT
[2019-11-03T01:30:16Z] 0:00:00 [step] Starting: Imposm
[2019-11-03T01:30:16Z] 0:00:00 [step] Starting: Reading OSM data
[2019-11-03T01:30:16Z] 0:00:00 [info] reading illinois-191101.osm.pbf with data till 2019-11-01 21:18:02 +0000 UTC
[2019-11-03T01:30:43Z] 0:00:26 [progress]    26s C:       0/s (22105787) N:       0/s (143904) W:       0/s (2455563) R:      0/s (11397)
[2019-11-03T01:30:43Z] 0:00:26 [step] Finished: Reading OSM data in 26.931630165s
[2019-11-03T01:30:43Z] 0:00:26 [step] Starting: Importing OSM data
[2019-11-03T01:30:43Z] 0:00:26 [step] Starting: Writing OSM data
[2019-11-03T01:31:33Z] 0:01:16 [step] Finished: Writing OSM data in 50.010791287s
[2019-11-03T01:31:33Z] 0:01:16 [step] Starting: Creating generalized tables
[2019-11-03T01:31:33Z] 0:01:16 [step] Starting: Generalizing osm_roads into osm_roads_gen1
[2019-11-03T01:31:33Z] 0:01:16 [step] Starting: Generalizing osm_waterways into osm_waterways_gen1
[2019-11-03T01:31:33Z] 0:01:16 [progress]    50s C:       0/s ( 0.0%) N:       0/s (100.0%) W:       0/s (100.0%) R:      0/s (100.0%)
[2019-11-03T01:31:33Z] 0:01:16 [step] Starting: Generalizing osm_landusages into osm_landusages_gen1
[2019-11-03T01:31:33Z] 0:01:16 [step] Starting: Generalizing osm_waterareas into osm_waterareas_gen1
[2019-11-03T01:31:34Z] 0:01:17 [step] Finished: Generalizing osm_landusages into osm_landusages_gen1 in 852.739049ms
[2019-11-03T01:31:34Z] 0:01:17 [step] Finished: Generalizing osm_waterareas into osm_waterareas_gen1 in 877.399288ms
[2019-11-03T01:31:35Z] 0:01:19 [step] Finished: Generalizing osm_waterways into osm_waterways_gen1 in 2.456752066s
[2019-11-03T01:31:35Z] 0:01:19 [step] Finished: Generalizing osm_roads into osm_roads_gen1 in 2.552737415s
[2019-11-03T01:31:35Z] 0:01:19 [step] Starting: Generalizing osm_waterareas into osm_waterareas_gen0
[2019-11-03T01:31:35Z] 0:01:19 [step] Starting: Generalizing osm_roads into osm_roads_gen0
[2019-11-03T01:31:35Z] 0:01:19 [step] Starting: Generalizing osm_waterways into osm_waterways_gen0
[2019-11-03T01:31:35Z] 0:01:19 [step] Starting: Generalizing osm_landusages into osm_landusages_gen0
[2019-11-03T01:31:35Z] 0:01:19 [step] Finished: Generalizing osm_waterareas into osm_waterareas_gen0 in 15.728988ms
[2019-11-03T01:31:35Z] 0:01:19 [step] Finished: Generalizing osm_landusages into osm_landusages_gen0 in 67.220349ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Generalizing osm_waterways into osm_waterways_gen0 in 676.815367ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Generalizing osm_roads into osm_roads_gen0 in 775.4874ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating generalized tables in 3.328510007s
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry indices
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_housenumbers
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_transport_points
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_landusages
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_buildings
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_waterareas
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_roads
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_barrierpoints
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_aeroways
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_aeroways in 77.84359ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_places
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_barrierpoints in 99.83354ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_housenumbers_interpolated
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_housenumbers_interpolated in 1.014213ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_admin
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_places in 26.083613ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_barrierways
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_admin in 45.714542ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_waterways
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_barrierways in 64.90692ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_amenities
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_amenities in 34.856712ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_transport_areas
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_transport_areas in 6.457409ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating OSM id index on osm_roads_gen0
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating geometry index on osm_housenumbers in 335.489204ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating OSM id index on osm_roads_gen1
[2019-11-03T01:31:36Z] 0:01:20 [step] Finished: Creating OSM id index on osm_roads_gen0 in 140.061828ms
[2019-11-03T01:31:36Z] 0:01:20 [step] Starting: Creating geometry index on osm_roads_gen0
[2019-11-03T01:31:37Z] 0:01:20 [step] Finished: Creating OSM id index on osm_roads_gen1 in 185.670655ms
[2019-11-03T01:31:37Z] 0:01:20 [step] Starting: Creating geometry index on osm_roads_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_waterareas in 746.647194ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_waterways_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_waterways_gen0 in 55.847423ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_waterways_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_transport_points in 917.671701ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_waterways_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_waterways_gen1 in 52.909307ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_waterways_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_waterways in 968.266479ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_landusages_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_landusages_gen1 in 14.628575ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_landusages_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_landusages_gen1 in 11.075038ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_landusages_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_landusages_gen0 in 2.511703ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_landusages_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_landusages_gen0 in 977.912µs
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_waterareas_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_waterareas_gen1 in 3.979921ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_waterareas_gen1
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_waterareas_gen1 in 1.272403ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating OSM id index on osm_waterareas_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating OSM id index on osm_waterareas_gen0 in 2.754495ms
[2019-11-03T01:31:37Z] 0:01:21 [step] Starting: Creating geometry index on osm_waterareas_gen0
[2019-11-03T01:31:37Z] 0:01:21 [step] Finished: Creating geometry index on osm_waterareas_gen0 in 948.358µs
[2019-11-03T01:31:38Z] 0:01:21 [step] Finished: Creating geometry index on osm_waterways_gen0 in 823.312002ms
[2019-11-03T01:31:38Z] 0:01:22 [step] Finished: Creating geometry index on osm_waterways_gen1 in 792.075819ms
[2019-11-03T01:31:38Z] 0:01:22 [step] Finished: Creating geometry index on osm_roads_gen0 in 1.523238068s
[2019-11-03T01:31:38Z] 0:01:22 [step] Finished: Creating geometry index on osm_roads_gen1 in 1.456556846s
[2019-11-03T01:31:38Z] 0:01:22 [step] Finished: Creating geometry index on osm_landusages in 2.147270722s
[2019-11-03T01:31:45Z] 0:01:29 [step] Finished: Creating geometry index on osm_roads in 9.339854853s
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Creating geometry index on osm_buildings in 15.451982383s
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Creating geometry indices in 15.452167745s
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Importing OSM data in 1m8.79156253s
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Clustering on geometry
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_amenities" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_transport_areas" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_landusages" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_buildings" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_transport_points" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_roads" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_housenumbers" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_waterways" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Indexing "osm_transport_areas" on geohash in 67.144706ms
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Clustering "osm_transport_areas" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Indexing "osm_amenities" on geohash in 144.892389ms
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Clustering "osm_amenities" on geohash
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Clustering "osm_transport_areas" on geohash in 121.248595ms
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Analysing "osm_transport_areas"
[2019-11-03T01:31:52Z] 0:01:35 [step] Finished: Analysing "osm_transport_areas" in 62.590573ms
[2019-11-03T01:31:52Z] 0:01:35 [step] Starting: Indexing "osm_aeroways" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Clustering "osm_amenities" on geohash in 137.54421ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Analysing "osm_amenities"
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_aeroways" on geohash in 38.661139ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_aeroways" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Clustering "osm_aeroways" on geohash in 81.556172ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Analysing "osm_aeroways"
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Analysing "osm_aeroways" in 38.20895ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Indexing "osm_places" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Analysing "osm_amenities" in 131.407929ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Indexing "osm_waterareas" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_places" on geohash in 53.038571ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_places" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_housenumbers" on geohash in 544.351963ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_housenumbers" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Clustering "osm_places" on geohash in 115.298476ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Analysing "osm_places"
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Analysing "osm_places" in 68.937527ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Indexing "osm_barrierpoints" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_waterways" on geohash in 651.330801ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_waterways" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_barrierpoints" on geohash in 87.345571ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_barrierpoints" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_transport_points" on geohash in 834.750503ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_transport_points" on geohash
[2019-11-03T01:31:52Z] 0:01:36 [step] Finished: Indexing "osm_waterareas" on geohash in 475.904894ms
[2019-11-03T01:31:52Z] 0:01:36 [step] Starting: Clustering "osm_waterareas" on geohash
[2019-11-03T01:31:53Z] 0:01:36 [step] Finished: Clustering "osm_barrierpoints" on geohash in 182.566918ms
[2019-11-03T01:31:53Z] 0:01:36 [step] Starting: Analysing "osm_barrierpoints"
[2019-11-03T01:31:53Z] 0:01:36 [step] Finished: Indexing "osm_landusages" on geohash in 1.009346685s
[2019-11-03T01:31:53Z] 0:01:36 [step] Starting: Clustering "osm_landusages" on geohash
[2019-11-03T01:31:53Z] 0:01:36 [step] Finished: Analysing "osm_barrierpoints" in 98.333671ms
[2019-11-03T01:31:53Z] 0:01:36 [step] Starting: Indexing "osm_admin" on geohash
[2019-11-03T01:31:53Z] 0:01:36 [step] Finished: Indexing "osm_admin" on geohash in 89.001396ms
[2019-11-03T01:31:53Z] 0:01:36 [step] Starting: Clustering "osm_admin" on geohash
[2019-11-03T01:31:53Z] 0:01:37 [step] Finished: Clustering "osm_housenumbers" on geohash in 1.287370367s
[2019-11-03T01:31:53Z] 0:01:37 [step] Starting: Analysing "osm_housenumbers"
[2019-11-03T01:31:54Z] 0:01:37 [step] Finished: Clustering "osm_admin" on geohash in 902.464789ms
[2019-11-03T01:31:54Z] 0:01:37 [step] Starting: Analysing "osm_admin"
[2019-11-03T01:31:54Z] 0:01:38 [step] Finished: Analysing "osm_housenumbers" in 840.77891ms
[2019-11-03T01:31:54Z] 0:01:38 [step] Starting: Indexing "osm_barrierways" on geohash
[2019-11-03T01:31:54Z] 0:01:38 [step] Finished: Indexing "osm_barrierways" on geohash in 66.878377ms
[2019-11-03T01:31:54Z] 0:01:38 [step] Starting: Clustering "osm_barrierways" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Clustering "osm_barrierways" on geohash in 203.682084ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Analysing "osm_barrierways"
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Analysing "osm_barrierways" in 137.141951ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Indexing "osm_housenumbers_interpolated" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Indexing "osm_housenumbers_interpolated" on geohash in 19.502826ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Clustering "osm_housenumbers_interpolated" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Clustering "osm_housenumbers_interpolated" on geohash in 38.239549ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Analysing "osm_housenumbers_interpolated"
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Analysing "osm_housenumbers_interpolated" in 2.368534ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Indexing "osm_landusages_gen0" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Indexing "osm_landusages_gen0" on geohash in 3.379174ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Clustering "osm_landusages_gen0" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Clustering "osm_landusages_gen0" on geohash in 26.558521ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Analysing "osm_landusages_gen0"
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Analysing "osm_landusages_gen0" in 13.964257ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Indexing "osm_waterareas_gen1" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Indexing "osm_waterareas_gen1" on geohash in 3.275511ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Clustering "osm_waterareas_gen1" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Clustering "osm_waterareas_gen1" on geohash in 24.754686ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Analysing "osm_waterareas_gen1"
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Analysing "osm_waterareas_gen1" in 462.353µs
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Indexing "osm_waterareas_gen0" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Indexing "osm_waterareas_gen0" on geohash in 3.20893ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Clustering "osm_waterareas_gen0" on geohash
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Clustering "osm_waterareas_gen0" on geohash in 21.263519ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Analysing "osm_waterareas_gen0"
[2019-11-03T01:31:55Z] 0:01:38 [step] Finished: Analysing "osm_waterareas_gen0" in 14.635016ms
[2019-11-03T01:31:55Z] 0:01:38 [step] Starting: Indexing "osm_roads_gen0" on geohash
[2019-11-03T01:31:55Z] 0:01:39 [step] Finished: Clustering "osm_transport_points" on geohash in 2.485492762s
[2019-11-03T01:31:55Z] 0:01:39 [step] Starting: Analysing "osm_transport_points"
[2019-11-03T01:31:55Z] 0:01:39 [step] Finished: Analysing "osm_admin" in 1.359077367s
[2019-11-03T01:31:55Z] 0:01:39 [step] Starting: Indexing "osm_roads_gen1" on geohash
[2019-11-03T01:31:56Z] 0:01:39 [step] Finished: Clustering "osm_waterways" on geohash in 3.33271769s
[2019-11-03T01:31:56Z] 0:01:39 [step] Starting: Analysing "osm_waterways"
[2019-11-03T01:31:56Z] 0:01:39 [step] Finished: Indexing "osm_roads_gen0" on geohash in 809.594487ms
[2019-11-03T01:31:56Z] 0:01:39 [step] Starting: Clustering "osm_roads_gen0" on geohash
[2019-11-03T01:31:56Z] 0:01:40 [step] Finished: Clustering "osm_waterareas" on geohash in 3.380152493s
[2019-11-03T01:31:56Z] 0:01:40 [step] Starting: Analysing "osm_waterareas"
[2019-11-03T01:31:56Z] 0:01:40 [step] Finished: Analysing "osm_transport_points" in 967.092574ms
[2019-11-03T01:31:56Z] 0:01:40 [step] Starting: Indexing "osm_waterways_gen0" on geohash
[2019-11-03T01:31:56Z] 0:01:40 [step] Finished: Indexing "osm_roads_gen1" on geohash in 989.863508ms
[2019-11-03T01:31:56Z] 0:01:40 [step] Starting: Clustering "osm_roads_gen1" on geohash
[2019-11-03T01:31:56Z] 0:01:40 [step] Finished: Indexing "osm_waterways_gen0" on geohash in 460.449528ms
[2019-11-03T01:31:56Z] 0:01:40 [step] Starting: Clustering "osm_waterways_gen0" on geohash
[2019-11-03T01:31:57Z] 0:01:41 [step] Finished: Indexing "osm_roads" on geohash in 5.517340211s
[2019-11-03T01:31:57Z] 0:01:41 [step] Starting: Clustering "osm_roads" on geohash
[2019-11-03T01:31:58Z] 0:01:41 [step] Finished: Clustering "osm_waterways_gen0" on geohash in 1.302818643s
[2019-11-03T01:31:58Z] 0:01:41 [step] Starting: Analysing "osm_waterways_gen0"
[2019-11-03T01:31:58Z] 0:01:41 [step] Finished: Clustering "osm_landusages" on geohash in 5.239004386s
[2019-11-03T01:31:58Z] 0:01:41 [step] Starting: Analysing "osm_landusages"
[2019-11-03T01:31:58Z] 0:01:42 [step] Finished: Indexing "osm_buildings" on geohash in 6.353242944s
[2019-11-03T01:31:58Z] 0:01:42 [step] Starting: Clustering "osm_buildings" on geohash
[2019-11-03T01:31:58Z] 0:01:42 [step] Finished: Clustering "osm_roads_gen0" on geohash in 2.51242892s
[2019-11-03T01:31:58Z] 0:01:42 [step] Starting: Analysing "osm_roads_gen0"
[2019-11-03T01:31:58Z] 0:01:42 [step] Finished: Analysing "osm_waterways_gen0" in 555.196084ms
[2019-11-03T01:31:58Z] 0:01:42 [step] Starting: Indexing "osm_waterways_gen1" on geohash
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Indexing "osm_waterways_gen1" on geohash in 337.266756ms
[2019-11-03T01:31:59Z] 0:01:42 [step] Starting: Clustering "osm_waterways_gen1" on geohash
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Analysing "osm_waterways" in 3.001176302s
[2019-11-03T01:31:59Z] 0:01:42 [step] Starting: Indexing "osm_landusages_gen1" on geohash
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Indexing "osm_landusages_gen1" on geohash in 2.710626ms
[2019-11-03T01:31:59Z] 0:01:42 [step] Starting: Clustering "osm_landusages_gen1" on geohash
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Clustering "osm_roads_gen1" on geohash in 2.647264019s
[2019-11-03T01:31:59Z] 0:01:42 [step] Starting: Analysing "osm_roads_gen1"
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Clustering "osm_landusages_gen1" on geohash in 24.658854ms
[2019-11-03T01:31:59Z] 0:01:42 [step] Starting: Analysing "osm_landusages_gen1"
[2019-11-03T01:31:59Z] 0:01:42 [step] Finished: Analysing "osm_landusages_gen1" in 536.791µs
[2019-11-03T01:32:00Z] 0:01:43 [step] Finished: Analysing "osm_waterareas" in 3.914210819s
[2019-11-03T01:32:01Z] 0:01:44 [step] Finished: Analysing "osm_roads_gen0" in 2.405781776s
[2019-11-03T01:32:01Z] 0:01:45 [step] Finished: Clustering "osm_waterways_gen1" on geohash in 2.480972647s
[2019-11-03T01:32:01Z] 0:01:45 [step] Starting: Analysing "osm_waterways_gen1"
[2019-11-03T01:32:01Z] 0:01:45 [step] Finished: Analysing "osm_roads_gen1" in 2.547467119s
[2019-11-03T01:32:01Z] 0:01:45 [step] Finished: Analysing "osm_waterways_gen1" in 363.365643ms
[2019-11-03T01:32:03Z] 0:01:46 [step] Finished: Analysing "osm_landusages" in 4.876017643s
[2019-11-03T01:32:14Z] 0:01:58 [step] Finished: Clustering "osm_roads" on geohash in 17.274824804s
[2019-11-03T01:32:14Z] 0:01:58 [step] Starting: Analysing "osm_roads"
[2019-11-03T01:32:17Z] 0:02:01 [step] Finished: Clustering "osm_buildings" on geohash in 19.4096715s
[2019-11-03T01:32:17Z] 0:02:01 [step] Starting: Analysing "osm_buildings"
[2019-11-03T01:32:23Z] 0:02:07 [step] Finished: Analysing "osm_roads" in 8.975077227s
[2019-11-03T01:32:27Z] 0:02:10 [step] Finished: Analysing "osm_buildings" in 9.422741163s
[2019-11-03T01:32:27Z] 0:02:10 [step] Finished: Clustering on geometry in 35.185881447s
[2019-11-03T01:32:27Z] 0:02:10 [step] Starting: Rotating tables
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_admin from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_barrierways from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_housenumbers_interpolated from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_amenities from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_transport_areas from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_waterways from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_landusages from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_buildings from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_housenumbers from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_transport_points from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_roads from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_aeroways from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_places from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:10 [info] Rotating osm_waterareas from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_barrierpoints from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_roads_gen1 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_waterways_gen0 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_waterways_gen1 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_landusages_gen1 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_landusages_gen0 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_waterareas_gen1 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_waterareas_gen0 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [info] Rotating osm_roads_gen0 from import -> public -> backup
[2019-11-03T01:32:27Z] 0:02:11 [step] Finished: Rotating tables in 108.915804ms
[2019-11-03T01:32:27Z] 0:02:11 [step] Finished: Imposm in 2m11.01812417s
root@75c3d31bd31e:/tools# 
root@75c3d31bd31e:/tools#

I just downloaded and tried http://download.geofabrik.de/north-america/us/illinois-191101.osm.pbf using the exact same options you used. I even spun up a brand new AWS EC2 Ubuntu 18.04 instead of CentOS 7 that I had been using but still no luck.

I think we may have narrowed it down to the PostGIS version. It looks like AWS Aurora is running Postgres 10.7 and PostGIS 2.4.

# psql --host=mydbserver mydb
psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1), server 10.7)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 12
Type "help" for help.

mydb=> SHOW server_version;
 server_version
----------------
 10.7

psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1), server 10.7)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

mydb=> SELECT PostGIS_Version();
            postgis_version
---------------------------------------
 2.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
(1 row)

mydb=>

Per your recommendation I spun up a Docker instance of Postgres. I tried the following combinations which all worked (I had to docker exec into the container to apt-get postgresql-postgis and then psql CREATE EXTENSION postgis;): Postgres 10.7 with PostGIS 2.5 Postgres 10.7 with PostGIS 3.0 Postgres 12.0 with PostGIS 2.5 Postgres 12.0 with PostGIS 3.0

Sadly there was no apt-get package for PostGIS 2.4, so I didn't get to specifically test it. But I think it might be reasonable to say that PostGIS 2.4 (what AWS Aurora uses) may not be compatible. Are there any known compatibility issues with PostGIS 2.4?

But I think it might be reasonable to say that PostGIS 2.4 (what AWS Aurora uses) may not be compatible. Are there any known compatibility issues with PostGIS 2.4?

imho: It should work with PostGIS 2.4 :)

My best theory is still - the SSL connections ... see your logs...

SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 12

Can you check the PostgreSQL server logs? Any errors? Low disk space? Any low memory problems ?

If you check the osm input data sizes ( http://download.geofabrik.de/north-america.html )

Illinois : (osm size: 170 MB) - not working
Iowa : (osm size: 85 MB) - working

so my theory - your import success is depend of the import data sizes

any osm file <= ~85 MB - is working
any osm file >= ~200MB - is not working

Can you verify my data size theory?

PostgreSQL TLS/SSL support - please verify your ssl_renegotiation_limit value ;

my log:

root@9ebef703e4e8:/map_matching# psql
psql (12.0 (Debian 12.0-2.pgdg90+1))
Type "help" for help.

app=# SHOW ssl_renegotiation_limit;
 ssl_renegotiation_limit 
-------------------------
 0
(1 row)

psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1), server 10.7)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)
Type "help" for help.

mmi=> SHOW ssl_renegotiation_limit;
 ssl_renegotiation_limit
-------------------------
 0
(1 row)

mmi=>

I also found this: https://www.postgresql.org/docs/9.5/release-9-5.html

Decommission server configuration parameter ssl_renegotiation_limit, which was deprecated in earlier releases (Andres Freund)

While SSL renegotiation is a good idea in theory, it has caused enough bugs to be considered a net negative in practice, and it is due to be removed from future versions of the relevant standards. We have therefore removed support for it from PostgreSQL. The ssl_renegotiation_limit parameter still exists, but cannot be set to anything but zero (disabled). It's not documented anymore, either.

I tried with AWS Aurora:

Alabama (72MB): Worked
Iowa (89MB): Worked
Illinois: (177MB): Failed
Florida (237MB): Failed

All work with Docker Postgres. Are there any other Postgres settings I should be looking at? I found out that I seem to have full access to most config options for the server.

I have a ticket open with AWS for how to get the Postgres logs. I have followed their directions but am not seeing anything in them. I also will be reviewing our PostGIS config with them to make sure we didn't miss anything.

All work with Docker Postgres. Are there any other Postgres settings I should be looking at?

Sorry, My knowledge stopped here ..
You can ask help in the community mailing list --> https://github.com/omniscale/imposm3/blob/master/README.md#support Maybe somebody has an experience with AWS Aurora.

ps:

You can also test with RDS Postgresql : https://aws.amazon.com/rds/postgresql/

I'm curious on this as well. I'm using osm2pgsql instead, but seeing the same error, and the same situations as you. e.g. I can successfully import a smaller data set, such as Delaware, but if I try to import, Oregon or California, they both fail with this exact same message. I'm trying this with AWS Aurora Serverless Postgres and running it from an r5a.large EC2 instance running Ubuntu 18.04. I'm able to import these same .pbf files on my local machine (which has a much newer Postgres).

Following up on this... I spun up an Aurora Postgres 11.4 instance and had no problems! I was able to import the California, and even Germany dataset which is quite large. So, it's either something with the serverless variant, or the Postgres 10.7/PostGIS variant on serverless. I have filed a support request with Amazon.

Some good news: I have a workaround. I was able to create a regular (non-serverless) Aurora Postgres 10.7 instance, do the data import on that, and then take a snapshot and restore that to an Aurora Postgres Serverless instance! Queries seem to work fine, etc. So, I think the problem is something to do with however these OSM import tools interacts with the Serverless variant (and not the Postgres version - since the tool worked fine on the same Postgres version non-serverless). I've asked AWS to see if they can shed any light on the error message so that possibly we can fix these OSM tools for this, but in the mean time this will at least allow one to use Aurora Postgres Serverless, even if not quite the ideal setup.

One other tidbit, which is mentioned in the osm2pgsql GitHub issue referenced above. I switched to a newer version there, and was able to get further. I no longer get the strange 0x2A error, but now can get connection timeouts on larger DB's (decreasing cache size allowed some progress, but I've get to get the Germany DB to import for example, but was now able to get oregon and california). I did have to set Aurora Serverless to not sleep, but even with that, I got connection timeouts.

@chris one of the things I found was causing timeouts whenever the severless would auto-scale up. You have to lock the upper limit and lower limit to be a fixed size. We have back burned the project for a few weeks and I haven't tried anything else. We also noticed that the version of PostGIS on the serverless is pretty old. We have a ticket open with AWS to get it updated to a newer version, but we have yet to get an update from AWS and when that will be.

Ah, ya, and it may be scaling when it does a big copy, good point. I will try that. I've asked them to get a newer Postgres as well. Thanks @mikes2277 .

@mikes2277 I tried that, and it failed again per my comments in the issue on the osm2pgsql issue I opened. I had it set to not sleep, and fixed ACU of 8. I tried importing a very large data set (the North America latest, which is about 9GB), and it failed after running for 24+ hours :( . Anyway, I'll just be using the workaround of using a regular instance, snapshotting that, and then restoring that to serverless.

omniscale / imposm3