mediagis / nominatim-docker

100% working container for Nominatim
Creative Commons Zero v1.0 Universal
1.07k stars 437 forks source link

osm2pgsql consistently crashes after half an hour or so #454

Closed sid9102 closed 1 year ago

sid9102 commented 1 year ago

Describe the bug My server never gets up and running, osm2pgsql crashes and the container exits at that point.

To Reproduce Steps to reproduce the behavior:

  1. Use command
    docker run --shm-size=8gb -it \
    -e PBF_URL=https://download.geofabrik.de/north-america-latest.osm.pbf \
    -e REPLICATION_URL=https://download.geofabrik.de/north-america-updates/ \
    -p 8080:8080 \
    --name nominatim \
    mediagis/nominatim:4.2
  2. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots & Logs

docker run --shm-size=8gb -it \
  -e PBF_URL=https://download.geofabrik.de/north-america-latest.osm.pbf \
  -e REPLICATION_URL=https://download.geofabrik.de/north-america-updates/ \
  -p 8080:8080 \
  --name nominatim \
  mediagis/nominatim:4.2
Unable to find image 'mediagis/nominatim:4.2' locally
4.2: Pulling from mediagis/nominatim
e2456498b9c1: Pull complete
4f4fb700ef54: Pull complete
6f92986e9982: Pull complete
Digest: sha256:bde205ab9c80a4c5fa2bb9defb1110997067a34c025af28af536514677dc3838
Status: Downloaded newer image for mediagis/nominatim:4.2
+ tailpid=0
+ replicationpid=0
+ trap stopServices SIGTERM TERM INT
+ /app/config.sh
+ id nominatim
+ useradd -m -p qaIACxO6wMR3 nominatim
+ IMPORT_FINISHED=/var/lib/postgresql/14/main/import-finished
+ '[' '!' -f /var/lib/postgresql/14/main/import-finished ']'
+ /app/init.sh
+ OSMFILE=/nominatim/data.osm.pbf
+ CURL=("curl" "-L" "-A" "${USER_AGENT}" "--fail-with-body")
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Wikipedia importance import'
Skipping optional Wikipedia importance import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional GB postcode import'
Skipping optional GB postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional US postcode import'
Skipping optional US postcode import
+ '[' '' = true ']'
+ '[' -f '' ']'
+ echo 'Skipping optional Tiger addresses import'
Skipping optional Tiger addresses import
+ '[' https://download.geofabrik.de/north-america-latest.osm.pbf '!=' '' ']'
+ echo Downloading OSM extract from https://download.geofabrik.de/north-america-latest.osm.pbf
Downloading OSM extract from https://download.geofabrik.de/north-america-latest.osm.pbf
+ curl -L -A mediagis/nominatim-docker:4.2.3 --fail-with-body https://download.geofabrik.de/north-america-latest.osm.pbf -C - --create-dirs -o /nominatim/data.osm.pbf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.5G  100 12.5G    0     0   884k      0  4:08:05  4:08:05 --:--:--  420k
+ '[' '' '!=' '' ']'
+ '[' '!' -f /var/lib/postgresql/14/main/PG_VERSION ']'
+ cp /etc/postgresql/14/main/conf.d/postgres-import.conf.disabled /etc/postgresql/14/main/conf.d/postgres-import.conf
+ sudo service postgresql start
 * Starting PostgreSQL 14 database server                                                                                                                                                                              [ OK ]
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''nominatim'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -s nominatim
+ sudo -E -u postgres psql postgres -tAc 'SELECT 1 FROM pg_roles WHERE rolname='\''www-data'\'''
+ grep -q 1
+ sudo -E -u postgres createuser -SDR www-data
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER nominatim WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -tAc 'ALTER USER "www-data" WITH ENCRYPTED PASSWORD '\''qaIACxO6wMR3'\'''
ALTER ROLE
+ sudo -E -u postgres psql postgres -c 'DROP DATABASE IF EXISTS nominatim'
NOTICE:  database "nominatim" does not exist, skipping
DROP DATABASE
+ chown -R nominatim:nominatim /nominatim
+ cd /nominatim
+ '[' '' = true ']'
+ sudo -E -u nominatim nominatim import --osm-file /nominatim/data.osm.pbf --threads 16
2023-06-08 10:54:31: Using project directory: /nominatim
2023-06-08 10:54:33: Creating database
2023-06-08 10:54:34: Setting up country tables
2023-06-08 10:54:35: Importing OSM data file
2023-06-08 10:54:35  osm2pgsql version 1.7.2
2023-06-08 10:54:35  Database version: 14.7 (Ubuntu 14.7-0ubuntu0.22.04.1)
2023-06-08 10:54:35  PostGIS version: 3.2
2023-06-08 10:54:35  Parsing gazetteer style file '/usr/local/etc/nominatim/import-full.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(554920k 812.5k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 94, in run
    database_import.import_osm_data(files,
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 113, in import_osm_data
    run_osm2pgsql(options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 159, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '10292', '--style', '/usr/local/etc/nominatim/import-full.style', '--output', 'gazetteer', '--create', '--with-forward-dependencies', 'false', '/nominatim/data.osm.pbf']' died with <Signals.SIGKILL: 9>.

Desktop / Server (please complete the following information):

Additional context Add any other context about the problem here.

leonardehrenfried commented 1 year ago

This looks like you're running out of memory.

sid9102 commented 1 year ago

Can confirm that I traced this back to attempting to load a whole continent on very little RAM. Smaller pbf is working fine. Thanks @leonardehrenfried!