Post processing fails during indices.sql

[jfberry]

Importing Great Britain nominatim data into a fresh 4.1 install

docker run -it --shm-size=1g \
  -e PBF_URL=https://download.geofabrik.de/europe/great-britain/england-latest.osm.pbf \
  -e IMPORT_WIKIPEDIA=false \
  -e NOMINATIM_PASSWORD=very_secure_password \
  -v nominatim-data:/var/lib/postgresql/14/main \
  -p 1234:8080 \
  --name nominatim \
  mediagis/nominatim:4.1

Eventually fails like this:

2022-10-13 15:45:11: Done 357280 in 569 @ 626.998 per second - postcodes (location_postcode) ETA (seconds): 1.15
2022-10-13 15:45:12: Done 357900 in 570 @ 627.008 per second - postcodes (location_postcode) ETA (seconds): 0.16
2022-10-13 15:45:12: Done 358003/358003 in 571 @ 626.809 per second - FINISHED postcodes (location_postcode)

2022-10-13 15:45:45: Post-process tables
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 134, in run
    database_import.create_search_indices(conn, args.config,
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 246, in create_search_indices
    sql.run_sql_file(conn, 'indices.sql', drop=drop)
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/sql_preprocessor.py", line 97, in run_sql_file
    cur.execute(sql)
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/connection.py", line 36, in execute
    super().execute(query, args)
  File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 146, in execute
    return super().execute(query, vars)
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

I would guess the sql takes too long to execute? Any tips?

[leonardehrenfried]

Are you perhaps running out of memory?

[jfberry]

8gb, no swap configured... I'll set up a swapfile and start again (edit) now starting again with an +8gb swap

[jfberry]

increased memory allowed this to complete... quite a misleading message, but thank you for helping