search

Overv / openstreetmap-tile-server

Docker file for a minimal effort OpenStreetMap tile server
Apache License 2.0
1.22k stars 485 forks source link

Trouble loading external data #318

Open flamchops opened 2 years ago

flamchops commented 2 years ago

I am having trouble completing the loading of external data into the database. After searching other issues, another user had nearly the same problem but when I try to troubleshoot using the same method I find that there is nothing out of place. The source URL seems valid and complete, yet it's still timing out. I have tried this process twice now and each time I got the same result.

Also, forgive me for my ignorance (I am quite new to a lot of this) but assuming I find a fix, how can I start this one task over to try again and resume the import from where I left off without going all the way back to the very beginning of the import process and starting over?

+ chown -R renderer: /home/renderer/src/ /data/style/
+ '[' -f /data/style/scripts/get-external-data.py ']'
+ '[' -f /data/style/external-data.yml ']'
+ sudo -E -u renderer python3 /data/style/scripts/get-external-data.py -c /data/style/external-data.yml -D /data/style/data
INFO:root:Starting load of external data into database
INFO:root:Checking table simplified_water_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/simplified-water-polygons-split-3857.zip
INFO:root:  Download complete (23715086 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table water_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip
INFO:root:  Download complete (810110834 bytes)
INFO:root:  Decompressing file
INFO:root:  Importing into database
INFO:root:  Import complete
INFO:root:Checking table icesheet_polygons
INFO:root:  Fetching https://osmdata.openstreetmap.de/download/antarctica-icesheet-polygons-3857.zip
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f25403020e0>, 'Connection to osmdata.openstreetmap.de timed out. (connect timeout=None)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='osmdata.openstreetmap.de', port=443): Max retries exceeded with url: /download/antarctica-icesheet-polygons-3857.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f25403020e0>, 'Connection to osmdata.openstreetmap.de timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/style/scripts/get-external-data.py", line 311, in <module>
    main()
  File "/data/style/scripts/get-external-data.py", line 235, in main
    download = s.get(source["url"], headers=headers)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='osmdata.openstreetmap.de', port=443): Max retries exceeded with url: /download/antarctica-icesheet-polygons-3857.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f25403020e0>, 'Connection to osmdata.openstreetmap.de timed out. (connect timeout=None)'))
nielsole commented 2 years ago

This sounds like one of two possible causes:

Importing the external data is the last step during import. It should already be able to serve tiles without some of the data.

https://github.com/Overv/openstreetmap-tile-server/blob/54624a969efc60fd5da58253c23850ecc4cbca60/run.sh#L121 You can execute that command, but might have to modify the yaml to remove entries that were already downloaded.