mediagis / nominatim-docker

100% working container for Nominatim
Creative Commons Zero v1.0 Universal
1.07k stars 437 forks source link

Timeout error when warming up database after planet import #478

Closed iAlex97 closed 10 months ago

iAlex97 commented 11 months ago

Describe the bug After rank imports and database check, the import process failed when warming up the database. I think this is due to https://github.com/osm-search/Nominatim/pull/3172 which sets a default query timeout of 10 seconds.

To Reproduce Start a planet import and wait for it

Expected behavior The import and warm up process should complete without errors

Screenshots & Logs

2023-09-30 05:50:36: Using project directory: /nominatim
2023-09-30 05:50:36: Checking database
Checking database connection ... OK
Checking for placex table ... OK
Checking for placex content ... OK
Checking that tokenizer works ... OK
Checking for wikipedia/wikidata data ... OK
Checking indexing status ... OK
Checking that database indexes are complete ... OK
Checking that all database indexes are valid ... OK
Checking TIGER external data table. ... OK
+ '[' https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/ '!=' '' ']'
+ sudo -E -u nominatim nominatim replication --init
2023-09-30 05:50:42: Using project directory: /nominatim
2023-09-30 05:50:42: Initialising replication updates
2023-09-30 05:50:47: Updates initialised at sequence 4022 (2023-09-17 20:56:04+00:00)
2023-09-30 05:50:47: Create functions
+ '[' '' = true ']'
+ '[' '' = true ']'
+ sudo -H -E -u nominatim nominatim admin --warm
2023-09-30 05:50:48: Using project directory: /nominatim
2023-09-30 05:50:48: Warming database caches
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sqlalchemy/ext/asyncio/engine.py", line 457, in execute
    result = await greenlet_spawn(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
    result = context.switch(value)
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2030, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 460, in execute
    self._adapt_connection.await_(
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 423, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/usr/lib/python3/dist-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/admin.py", line 60, in run
    return self._warm(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/admin.py", line 106, in _warm
    api.search(word)
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/core.py", line 709, in search
    return self._loop.run_until_complete(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/core.py", line 223, in search
    return await geocoder.lookup(phrases)
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/geocoder.py", line 151, in lookup
    results = await self.execute_searches(query, searches[:50])
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/geocoder.py", line 86, in execute_searches
    for result in await search.lookup(self.conn, self.params):
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/search/db_searches.py", line 734, in lookup
    for row in await conn.execute(sql, _details_to_bind_params(details)):
  File "/usr/local/lib/nominatim/lib-python/nominatim/api/connection.py", line 63, in execute
    return await asyncio.wait_for(self.connection.execute(sql, params), self.query_timeout)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Desktop / Server (please complete the following information):

Additional context I will try another planet import with NOMINATIM_QUERY_TIMEOUT set to 600. If that fails, I will disable it entirely while warming up and re-set it afterwards as this mechanism is a good idea. If everything is works out, I will submit a PR with said fixes.