search

gravitystorm / openstreetmap-carto

A general-purpose OpenStreetMap mapnik style, in CartoCSS
Other
1.53k stars 819 forks source link

Unchanged docker image does not produce valid images #4303

Closed zelosos closed 3 years ago

zelosos commented 3 years ago

Expected behavior

After installing the docker, and importing the EU-map from: https://download.geofabrik.de/europe-latest.osm.pbf a valid map should be created.

Actual behavior

On all zoom levels the map keeps blank, only the landmass is colored white and ocean is light-blue.

Detailed discription

My exact steps I took:

  1. clone the repository with git clone https://github.com/gravitystorm/openstreetmap-carto
  2. download the data from https://download.geofabrik.de/europe-latest.osm.pbf
  3. rename the data file as "data.osm.pbf" and move it into the root folder of the cloned repo
  4. make sure not postgresql instance is running on the machine
  5. start the import with: docker-compose up import (no error)
  6. start the kosmtik with: docker-compose up kosmtik (only font errors, but i red, that they should be ignored here in the issues)
  7. open the webpage http://127.0.0.1:6789 This is basicly the step by step tutorial of the DOCKER.md.

Also when downloading the tiles with jTileDownloader they are rendered blank too.

I can confirm that the db docker is running and listening on port 5432.

zelosos commented 3 years ago

Same result on a different laptop. Both run Linux Mint. No errors for low memory or disk space.

jeisenbe commented 3 years ago

What was the result of docker-compose up import? How long did it take?

Have you tried importing a smaller file first, such as luxembourg.osm.pbf?

europe-latest.osm.pbf is a very large file, containing about half the data in planet.osm.pbf. I am not able to import this on my laptop because I do not have sufficient RAM.

zelosos commented 3 years ago

The import took around an hour or two. There was long no output and then listed a lot of number pairs in th cli. Something like: 4455664612 444135485 91249962 After this it checked the polygons of water, icesheet, ... At the end it gave: openstreet-carto_import_1 exited with code 0 which sounds a success to me.

RAM was no problem, did not even use the SWAP.

I will try the luxembourg.osm.pbf.

zelosos commented 3 years ago

Here is the output when importing Luxembourg: ` import_1 | Copying planet_osm_point to cluster by geometry finished

import_1 | Creating geometry index on planet_osm_point import_1 | Creating indexes on planet_osm_point finished import_1 | All indexes on planet_osm_point created in 3s import_1 | Completed planet_osm_point import_1 | Sorting data and creating indexes for planet_osm_line import_1 | Copying planet_osm_line to cluster by geometry finished import_1 | Creating geometry index on planet_osm_line import_1 | Creating indexes on planet_osm_line finished import_1 | All indexes on planet_osm_line created in 5s import_1 | Completed planet_osm_line import_1 | Sorting data and creating indexes for planet_osm_polygon import_1 | Copying planet_osm_polygon to cluster by geometry finished import_1 | Creating geometry index on planet_osm_polygon import_1 | Creating indexes on planet_osm_polygon finished import_1 | All indexes on planet_osm_polygon created in 17s import_1 | Completed planet_osm_polygon import_1 | Sorting data and creating indexes for planet_osm_roads import_1 | Copying planet_osm_roads to cluster by geometry finished import_1 | Creating geometry index on planet_osm_roads import_1 | Creating indexes on planet_osm_roads finished import_1 | All indexes on planet_osm_roads created in 1s import_1 | Completed planet_osm_roads import_1 | import_1 | Osm2pgsql took 59s overall import_1 | node cache: stored: 3019436(100.00%), storage efficiency: 49.94% (dense blocks: 2, sparse nodes: 3015060), hit rate: 100.00% import_1 | INFO:root:Checking table simplified_water_polygons import_1 | INFO:root:Checking table water_polygons import_1 | INFO:root:Checking table icesheet_polygons import_1 | INFO:root:Checking table icesheet_outlines import_1 | INFO:root:Checking table ne_110m_admin_0_boundary_lines_land import_1 | Traceback (most recent call last): import_1 | File "scripts/get-external-data.py", line 280, in import_1 | main() import_1 | File "scripts/get-external-data.py", line 219, in main import_1 | download.raise_for_status() import_1 | File "/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status import_1 | raise HTTPError(http_error_msg, response=self) import_1 | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://www.naturalearthdata.com/http/www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip openstreetmap-carto_import_1 exited with code 1`

So there was a failure in the URL.

zelosos commented 3 years ago

I deleted all the docker images before the import to get a fresh start.

mboeringa commented 3 years ago

Here is the output when importing Luxembourg: ... import_1 | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://www.naturalearthdata.com/http/www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_boundary_lines_land.zip openstreetmap-carto_import_1 exited with code 1

So there was a failure in the URL.

It sounds like this is relevant: https://github.com/gravitystorm/openstreetmap-carto/issues/4304 and https://github.com/gravitystorm/openstreetmap-carto/pull/4306

After installing the docker, and importing the EU-map from: https://download.geofabrik.de/**europe-latest.osm.pbf** a valid map should be created.

The import took around an hour or two.

That's extremely fast for a data extract the size of Europe. Just curious, what hardware are you using for this?

zelosos commented 3 years ago

My specs: CPU: i7-5600U 4 core 12GB RAM 512GB SSD

Yes its the same error as mensioned in #4304 .

jeisenbe commented 3 years ago

Please try updating to the latest master branch, after https://github.com/gravitystorm/openstreetmap-carto/pull/4306 and it should now work.

jeisenbe commented 3 years ago

I believe 12GB of RAM is just barely enough to import the europe.osm.pbf database file. I would recommend testing with smaller files: start with e.g. luxembourg.osm.pbf, then step up to a mid-sized country in Europe, and then try to import the whole of europe.osm.pbf if the previous two tests worked well. My understanding is that importing a single mid-sized country (or US State) will usually take about an hour, importing all of Europe will take several hours. See https://switch2osm.org/serving-tiles/ which recommend 24GB+ of RAM for rendering the planet (which is about twice the size of Europe, in OpenStreetMap data size)

zelosos commented 3 years ago

With the updated master it works. Luxembourg and Germany worked, so I assume Europe will work as well. Thanks a lot for the help.