search

osm-search / Nominatim

Open Source search based on OpenStreetMap data
https://nominatim.org
GNU General Public License v3.0
2.99k stars 701 forks source link

nominatim admin --warm python timeout error in Nominatim 4.3.2 fresh install #3351

Closed wgds60 closed 4 months ago

wgds60 commented 4 months ago

Describe the bug

To Reproduce Install Nominatim 4.3.2 on Ubuntu 22.04 following the install instructions.

After installing, importing and updating, start a nominatim admin --warm in the project_dir.

Example:

nominatim admin --warm
2024-02-29 14:13:33: Using project directory: <nominatim_project_dir>
2024-02-29 14:13:33: Warming database caches
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/ext/asyncio/engine.py", line 457, in execute
    result = await greenlet_spawn(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2030, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 460, in execute
    self._adapt_connection.await_(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 423, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/admin.py", line 60, in run
    return self._warm(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/admin.py", line 109, in _warm
    api.search(word)
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/core.py", line 709, in search
    return self._loop.run_until_complete(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/core.py", line 223, in search
    return await geocoder.lookup(phrases)
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/geocoder.py", line 200, in lookup
    results = await self.execute_searches(query, searches[:50])
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/geocoder.py", line 88, in execute_searches
    lookup_results = await search.lookup(self.conn, self.params)
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/db_searches.py", line 734, in lookup
    for row in await conn.execute(sql, _details_to_bind_params(details)):
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/connection.py", line 63, in execute
    return await asyncio.wait_for(self.connection.execute(sql, params), self.query_timeout)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Instead I would expect the warming to run through (as it did in older versions of Nominatim).

Software Environment (please complete the following information):

Hardware Configuration (please complete the following information):

Postgresql Configuration:

shared_buffers = 60GB
maintenance_work_mem = 10GB
autovacuum_work_mem = 2GB
work_mem = 300MB
effective_cache_size = 60GB
effective_io_concurrency = 500
synchronous_commit = off
random_page_cost = 1.5
max_wal_size = 2GB
checkpoint_timeout = 10min
checkpoint_completion_target = 0.9
max_parallel_workers_per_gather = 0
jit = off
max_connections = 250

Nominatim Configuration:

NOMINATIM_FLATNODE_FILE="/var/nominatim/flatnode.bin"
NOMINATIM_IMPORT_STYLE="extratags"
NOMINATIM_REPLICATION_URL="https://planet.openstreetmap.org/replication/day"
# How often upstream publishes diffs
NOMINATIM_REPLICATION_UPDATE_INTERVAL=86400
# How long to sleep if no update found yet
NOMINATIM_REPLICATION_RECHECK_INTERVAL=900
NOMINATIM_USE_US_TIGER_DATA=yes
NOMINATIM_SEARCH_WITHIN_COUNTRIES=no

Additional context

Other nominatim cli commands like e.g. nominatim admin --check-database or nominatim status or nominatim search --query Paris are fine.

The importing missed some indexes first (as reported by check-database), those were added by "nominatim index" after nominatim import. After that, check-database was OK.

mtmail commented 4 months ago

Can you try setting the timeout settings ? The separate docker project added some logic recently https://github.com/mediagis/nominatim-docker/commit/c06e9bbb2936f3121000a54a95ce3b176df67f2d

wgds60 commented 4 months ago

With NOMINATIM_QUERY_TIMEOUT=600 it seems to be running, not seeing an error so far. Only those nice dots as running indicators (that were present in older Nominatim versions, php-based warm commands) are missing. I will report back if the warm finishes successfully.

Ok, with the extra setting the warm process finished in ~30 minutes without any errors. Thank you @mtmail.