search

osmcode / pyosmium

Python bindings for libosmium
https://osmcode.org/pyosmium
BSD 2-Clause "Simplified" License
318 stars 65 forks source link

Odd error with minutely diffs when server returns 500 #235

Closed lonvia closed 1 year ago

lonvia commented 1 year ago

There are intermittent errors with the minutely planet diffs. pyosmium fails then with the following error when it encounters one:

2023-11-16 10:52:29: Using project directory: /srv/nominatim.openstreetmap.org/planet-project
Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 226, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 199, in run
    self._update(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 154, in _update
    state = replication.update(dsn, params, socket_timeout=args.socket_timeout)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/replication.py", line 128, in update
    endseq = repl.apply_diffs(outhandler, startseq + 1,
  File "/usr/lib/python3/dist-packages/osmium/replication/server.py", line 171, in apply_diffs
    diffs = self.collect_diffs(start_id, max_size)
  File "/usr/lib/python3/dist-packages/osmium/replication/server.py", line 137, in collect_diffs
    left_size -= rd.add_buffer(diffdata, self.diff_type)
RuntimeError: gzip error: inflate failed: incorrect header check

According to the osm server logs, the server has returned a 500 in that case:

0f474499af5eca52a75b973ff124d6c5de371cb82e51ed089ac52d08b49a14a2 osm-planet-eu-central-1 [16/Nov/2023:10:52:35 +0000] 2605:bc80:3010:700::8cd3:a764 - ZH16QHVG5WA39ZSW REST.GET.OBJECT planet/replication/minute/005/836/144.osc.gz "GET /planet/replication/minute/005/836/144.osc.gz HTTP/1.1" 500 InternalError 282 - 2090 - "-" "Nominatim (pyosmium/3.5.0)" - ftbR6xrHyxr/l7nUEAOJuhB7HC3zyR/TQVOPNupr/SOrSr+kxb5YW1xkrCqbYa3AXfUit796pD8= - ECDHE-RSA-AES128-GCM-SHA256 - [osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com](http://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/) TLSv1.2 - -

pyosmium seem to have ignored the non-200 return code and then failed on unpacking the error response. It should simply retry fetching the diff instead.