Describe the bug
During planet import unexpectedly throws bad_alloc and crashes.
To Reproduce
Steps to reproduce the behavior:
Import planet with --memory=20g --shm-size=20g
osm2pgsql throws std::bad_alloc
Expected behavior
The lack of memory should not quietly crash the app. Possibly, some way to pre-check if the system has enough memory or at least a way to predict how much memory is needed.
Screenshots & Logs
2023-02-27 00:54:57 ERROR: std::bad_alloc
Traceback (most recent call last):
File "/usr/local/bin/nominatim", line 14, in <module>
exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
return parser.run(**kwargs)
File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
return args.command.run(args)
File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 94, in run
database_import.import_osm_data(files,
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 113, in import_osm_data
run_osm2pgsql(options)
File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 159, in run_osm2pgsql
subprocess.run(cmd, cwd=options.get('cwd', '.'),
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '40193', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--output', 'gazetteer', '--create', '--with-forward-dependencies', 'false', '/sources/data.osm.pbf']' returned non-zero exit status 1.
Desktop / Server (please complete the following information):
OS & Version: CentOS 7
Docker Version: 19.03.13
Nominatim Version: 4.2.1
Additional context
The app runs alongside another Nominatim instance, but I've tried monitoring the system stats at the time of the crash, the server never had less than 10 gigabytes of extra memory available (over the container limit). It looks like osm2pgsql at some point either tries to allocate arbitrary amounts of memory without checking how much is actually present on the machine (container), or just makes a broken allocation syscall.
Describe the bug During planet import unexpectedly throws bad_alloc and crashes.
To Reproduce Steps to reproduce the behavior:
Expected behavior The lack of memory should not quietly crash the app. Possibly, some way to pre-check if the system has enough memory or at least a way to predict how much memory is needed.
Screenshots & Logs
Desktop / Server (please complete the following information):