Devours all RAM and computer hangs on 'docker-compose up import'

[OnkelTem]

I decided to give a try to docker images in the repo, but it ended up unhappily. Whenever I run docker-compose up import, after the message that it loads data.osm.pbf it just consumes all the RAM (16GB) so that I had to restart the computer.

$ docker-compose up import
Starting openstreetmapcarto_db_1 ...
Starting openstreetmapcarto_db_1 ... done
Starting openstreetmapcarto_import_1 ...
Starting openstreetmapcarto_import_1 ... done
Attaching to openstreetmapcarto_import_1
import_1   | Waiting for PostgreSQL to be running
import_1   | PostgreSQL running
import_1   | CREATE EXTENSION
import_1   | NOTICE:  extension "postgis" already exists, skipping
import_1   | CREATE EXTENSION
import_1   | NOTICE:  extension "hstore" already exists, skipping
import_1   | osm2pgsql SVN version 0.88.1 (64bit id space)
import_1   |
import_1   | Using lua based tag processing pipeline with script openstreetmap-carto.lua
import_1   | Using projection SRS 900913 (Spherical Mercator)
import_1   | Setting up table: planet_osm_point
import_1   | Setting up table: planet_osm_line
import_1   | Setting up table: planet_osm_polygon
import_1   | Setting up table: planet_osm_roads
import_1   | Allocating memory for dense node cache
import_1   | Allocating dense node cache in one big chunk
import_1   | Allocating memory for sparse node cache
import_1   | Sharing dense sparse
import_1   | Node-cache: cache=1000MB, maxblocks=128000*8192, allocation method=3
import_1   | Mid: Ram, scale=100
import_1   |
import_1   | Reading in file: data.osm.pbf
# Here it stops... 

I tried to change osm2pgsql settings:

      - PG_WORK_MEM
      - PG_MAINTENANCE_WORK_MEM
      - OSM2PGSQL_CACHE
      - OSM2PGSQL_NUMPROC

but even with default options it just ends like that.

My data.osm.pbf file is approximately 500Mb of size.

[matthijsmelissen]

Thanks for letting us know! Does it work fine with a smaller data cile?

[OnkelTem]

Thanks for letting us know! Does it work fine with a smaller data cile?

Yeah, the same. Just re-tried with http://download.geofabrik.de/russia/north-caucasus-fed-district.html (59M) - the same. Memory begins draining at noticeable pace, in 5 minutes 16Gbs are off.

[matthijsmelissen]

@pnorman Are you aware of anything that could cause this?

[matkoniecz]

For start one may try running osm2pgsql with --verbose.

Can you try modifying https://github.com/gravitystorm/openstreetmap-carto/blob/master/scripts/docker-startup.sh#L42 and running again? Maybe it will give some useful info.

[OnkelTem]

@matkoniecz I've tried it. The results are the same, there is no new output. Could it be that 0.88 of osm2pgsql is buggy? I'm going to try another version, a newer one (0.94)

[OnkelTem]

The issue is fixed now. Sorry I had to rewrite your Dockerfile.import:

FROM debian:stretch

ARG OSM2PGSQL_VERSION=0.94.0
ARG OSM2PGSQL_DEPS="make cmake g++ libboost-dev libboost-system-dev libboost-filesystem-dev libexpat1-dev \
  zlib1g-dev libbz2-dev libpq-dev libproj-dev lua5.2 liblua5.2-dev autoconf"
# Install tools, utilities and osm2pgsql build deps
RUN set -x && \
    DEBIAN_FRONTEND=noninteractive && \
    apt-get update && \
    apt-get install -y postgresql-client wget git curl ${OSM2PGSQL_DEPS}

# Get, build and install osm2pgsql
RUN set -x && \
    cd /tmp && \
    git clone --depth 1 --branch ${OSM2PGSQL_VERSION} https://github.com/openstreetmap/osm2pgsql.git && \
    mkdir -p osm2pgsql/build && \
    cd osm2pgsql/build  && \
    cmake .. && \
    make install && \
    rm -rf /tmp/osm2pgsql

# Clean up the system
RUN set -x && \
    DEBIAN_FRONTEND=noninteractive && \
    ldd /usr/local/bin/osm2pgsql | awk '/=>/{print $3}' | while read -r f; do test -f $f && dpkg -S "$f" | head -n 1 | cut -d: -f1 | sort -u; done | paste -sd " " | xargs apt-mark manual && \
    apt-get autoremove --purge -y ${OSM2PGSQL_DEPS} && \
    rm -rf /var/lib/apt/lists/*

ADD openstreetmap-carto.style /

RUN mkdir -p /openstreetmap-carto
WORKDIR /openstreetmap-carto

CMD sh scripts/docker-startup.sh import

The base image is also changed, now it's debian:stretch. I've rerun docker-compose up import and now it worked w/o any issues.

Probably such an approach may seem to be an overkill, so let me try stock osm2pgsql from debian:stretch.

[OnkelTem]

Ok, so I've just tested the stock osm2pgsql from debian:stretch (0.92). Works fine either. The image size is 291MB.

Wanna PR?

[matthijsmelissen]

Yes please!

Just to be sure can you also try the original version once more and check if it still fails? We've had some other random/intermittent issues in the past, just to exclude that.

[OnkelTem]

@math1985 Sure, just checked one more time - the same.

[OnkelTem]

Probably now it's a good idea to update the base images in the other Dockerfiles.

[pnorman]

@pnorman Are you aware of anything that could cause this?

No idea, and I don't debug or run osm2pgsql in docker.

[matthijsmelissen]

The Docker file did not specify to run osm2pgsql in slim mode. Using slim mode uses temporary tables, so running the process takes more disk space, while less memory is used. Using slim mode should likely fix this issue.

[StyXman]

What about flat nodes?