Map download at Planet Extractor not possible, almost same errors as in issue #1072

martin-ac commented 3 years ago

Hi, again like in end 2020 there are problems with the download of maps with Planet Extractor at "http://maps9.navit-project.org/" as well as "http://maps3.navit.project.org" Partly I do not even get error messages, partly - from maps9 - i receive them in german language and translated them to english for this issue:

At "maps3.navit-project.org" (Map Version 210326)

for download of Europe -> WesternEurope -> Germany -> Northrheinwestfalia the error message is xyz.bin.part couldn'd be saved because the source file couldn't be read. Try again later or get in contact with the server administrator.

same for Germany same for Germany+Austria+Switzerland

for WesternEurope: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator at webmaster@maps3.navit-project.org to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log. Apache/2.4.29 (Ubuntu) Server at maps3.navit-project.org Port 80

At "maps9.navit-project.org" (Map Version 210326)

for download of Europe -> WesternEurope -> Germany -> Northrheinwestfalia download doesn't start or stops after few (4) megabytes

same for Germany same for Germany+Austria+Switzerland

for WesternEurope: Serverfailure! The request can not be answered because of an internal error. Errormessage: End of script output before headers: index.cgi If you think this is a server malfunction please inform the webmaster. Error 500 maps9.navit-project.org Apache/2.4.33 (Linux/SUSE)

To me it looks like currently any download from both of the servers to not work no matter what size the maps have. Does this error (always) occur only after may be faulty data in maps data updates? Maybe others have faced the same errors but do not know where to report this to because the only information about the source of the web interface in the upper right corner is outdated: it points to Tordt Schmidt / SMT-WebServices and Tordt answered to my mail that he doesn't do anything for the servers since more but several years!

Could please again someone take care of this? Many thanks in advance and please give me feedback as soon as I should test again!!!

ghost commented 3 years ago

I have the same problem. I tried different browsers (Firefox with and without ad- and script blocker, old Chrome), but none of them works. The download fails instantly or after 4 to 8 MB. Please help.

Budman1758 commented 3 years ago

For what its worth I just downloaded a map of the entire United States (6.3Gig) without any problem at all. Microsoft Edge browser.

martin-ac commented 3 years ago

I can confirm @Budman1758 's statement: download is possible without error for predefined area USA.

I did some additional test and found that currently (Map Version 210326) downloads work without errors for NorthAmerica, SouthAmerica and Africa as predefined areas too

but show the same errors as with Europe for Asia und Australia. And of course I again tested the downloads for WesternEurope, Germany+Austria+Switzerland, Germany and NorthRheinWestfalia again and found the same errors as before.

I only checked at "http://maps9.navit-project.org/". So again: please help - thanks.

kruthoff commented 3 years ago

Same problem here. I'm trying to download an area of ~1GB. The download starts over and over again without downloading anything at all. Chromium 88 on Raspberry Pi.

NGC935 commented 3 years ago

As stated above the maps for Europe are broken and the planet-210326.bin file crashes my navit. Downloaded it three times from maps3.navit… and maps9…. Someone wrote on the earlyer issues to use the mirror maps3 but I guess here is another problem, my result was the same.

Right now I'm downloading an OSM data dump to create the binfile myself, as stated in the manual here. The dump is about 40GB (.osm.bz2), I guess the output might be around 20GB. This filesize is above the volume I'm able to upload in 24h for sharing, so if anyone needs Europe-210502.bin after a successful test, we need an idea how to share it. Torrent might be a solution if enough people help.

And BTW: maptool crashes with segfault if I try to convert from protobuf:

~/Downloads$ maptool --protobuf -i antarctica-latest.osm.pbf antarctica.bin
PROGRESS: Phase 1: reading input data 0:00 0 MB
Speicherzugriffsfehler

So if you try to do the same processing on Debian, download .osm.bz2 . Antartika is a good testfile here.

This is a good example for our Internet connections in Germany: We are shouting: "Set a course, maximum warp!" while jumping into Mum's 1969er VW Beetle.

jkoan commented 3 years ago

Hi everyone, currently the infrastructure for the map extractor is broken, yes this is true. Unfortunately i cant fix this because i don't have access to those servers. The reason behind this is that the software running on the Map Extractor is proprietary and only a few people have access to those servers with those software. Also exactly those people (for example @pgrandin ) have been busy for quite some time.

So the most possible solution is to rewrite the map extractor in its entirety. This takes a lot of time and resources and the core team of navit is unfortunately is quite small. I'm already in the process to research all the needed details to write a new map extract server, but this takes time as well.

From what i analysed so for another problem is that the zip64 support for the binfile plugin (binfiles are just special zipfiles) is broken as well, so it might be better to fix those as well to be able to use more standard librarys on the map extractor.

jkoan commented 3 years ago

And BTW: maptool crashes with segfault if I try to convert from protobuf:
~/Downloads$ maptool --protobuf -i antarctica-latest.osm.pbf antarctica.bin
PROGRESS: Phase 1: reading input data 0:00 0 MB
Speicherzugriffsfehler
So if you try to do the same processing on Debian, download .osm.bz2 . Antartika is a good testfile here.

This is a good example for our Internet connections in Germany: We are shouting: "Set a course, maximum warp!" while jumping into Mum's 1969er VW Beetle.

Please try this again on a dev version and report anew issue if this is still an issue. Recent builds can be found here: https://download.navit-project.org/

Right now I'm downloading an OSM data dump to create the binfile myself, as stated in the manual here. The dump is about 40GB (.osm.bz2), I guess the output might be around 20GB. This filesize is above the volume I'm able to upload in 24h for sharing, so if anyone needs Europe-210502.bin after a successful test, we need an idea how to share it. Torrent might be a solution if enough people help.

I think torrents aren't really an option if we wanna keep the previously available daily updates, but yes, as a temporary solution this would be an option

metalstrolch commented 3 years ago

If considering re-doing something like the planet extractor, I want to remind my open source tool to extract regions out of bin files I created when implementing multipolygon support for binfile. Have a look there: https://github.com/metalstrolch/navit_binfile_extractor It's a command line application to do the task. The parameters for it could be given by a web-gui like the current extractor page. Have fun.

jkoan commented 3 years ago

If considering re-doing something like the planet extractor, I want to remind my open source tool to extract regions out of bin files I created when implementing multipolygon support for binfile. Have a look there: https://github.com/metalstrolch/navit_binfile_extractor It's a command line application to do the task. The parameters for it could be given by a web-gui like the current extractor page. Have fun.

ah thx, for this information, this is more or less what we need, i will build something on top of this. Then we only need to clarify how we want to update the planet.bin itself. maptool on my machine takes several days to finish panet.osm.pbf. There we need optimizations or at least workarounds

NGC935 commented 3 years ago

OK, if you guys get the extractor ready in a few weeks, we don't need to setup home build map distribution. The maptool eats my whole HDD space and my laptop is still much weaker than a ryzen. I'm gonna stop my clean-up if you say your solution is in reach.

NGC935 commented 3 years ago

Update from my side:

I'm stil working on a map for europe with my old workstation with enough disc space and the europe-latest.osm.bz2. This computer is placed in my bedroom, so I only do 2 stages a day, about 4-6h each. This is why…
… I also converted a map of Germany for my Jolla phone. Done in 90 min, less than 2GB RAM max, infile: 3,6 GB, out 2,7 GB temporary HD about 30GB.
I used the PBF and it worked out of the box with the latest version linked above:
```
ngc935@host:~/Downloads/navit$ bin/navit -v 
navit 0.5.6+git:xdevxgitxnotxfound-
```
I started the same process with the same binary on my workstation and with europe-latest.osm.pbf and it stopped with a segfault. Please give me a hint if we need to investigate this in another issue, since the same procedure is working with a smaller file.
The map of Germany is working and up to date (7.5.21), I checked my own changes. I'll use that as long as the extractor is brocken. If anyone needs that map, get in touch with me. If anyone is interested in Europe (if it will finish in a few days) do the same. Converting is easy, uploading hard…
If anyone tries to convert own maps consiter this: RAM is stable below 2 GB, temporary hard drive space about 8 to 10 times the download file, result is about 75% of the downloaded PBF or 50% of the bz2.

Thanks for your services, I'll try to give something back.

Edit 5 days later: It's done and working. Nice and about 18,1GB. If anyone needs Europe-2105003, please get in touch.

NGC935 commented 3 years ago

Here are the torrents for my maps, europe and Germany. Both tested on my SailfishOS. torr.zip

BruhP8 commented 3 years ago

Hello I can confirm the problem is still present and only applies to certain parts of the map (eg : i could download the "basse normandie" map from France, but not the "Ile de France" map) @NGC935 thank you for sharing. There are not any pairs right now, at least ones that have the full file for europe. Can you start peering ?

mvglasow commented 3 years ago

@BruhP8 I’m currently peering, let me know if/when you have the full file.

BruhP8 commented 3 years ago

@mvglasow Currently i can see only one peer, from which i get 71KB/s so its gonna take a bit lol, i'll let you know thanks !

NGC935 commented 3 years ago

@BruhP8 I'm sorry, we had a thunderstorm the last days and an outage broke the jail with the torrents on my home server. Thanks for reminding me to fix it, I'll do that in the next days. I'd like to share those files until we have newer ones or this issue is fixed.

BruhP8 commented 3 years ago

@NGC935 Dont worry about it, i have time ! Slow is better than nothing ! Thanks again for sharing those

BruhP8 commented 3 years ago

@NGC935 @mvglasow Yaay after a whopping 1 day and 3h i have the file ! Thank you both for sharing, i'll try to keep peering a bit for others

jkoan commented 3 years ago

Hi all, the last days i have been working on a new idea as a solution. Based on GitHub Actions its possible to process maps and upload them as a Github Release. This works quite well but has one downside. Its not possible to process yourself an individual map.

The current approach is to convert every map at County size, except for Germany where i chose the size of federal states for now as i was testing with this. The currently biggest run can be seen here: https://github.com/jkoan/gh-actions-mapserver/actions/runs/1017546633 Once the run is over if will have published the files here: https://github.com/jkoan/gh-actions-mapserver/releases/tag/2021-07-10

To future improve on this the idea would be to reduce the processed areas to the smallest areas available on https://download.geofabrik.de/ as this would help to come as close as possible to a individual map. If this system is working reliably we would probably move this under the umbrella of navit-gps.

mvglasow commented 3 years ago

Nice! So we’ll finally be able to get up-to-date maps again.

@jkoan I suppose there is a typo and it should be country, not county size...

One thing that still needs to be tackled is that the resulting maps are rather small (I ususally carry a map of Europe on my device, which is around 16 GiB in size). Adding multiple maps is possible but may cause undesirable side effects in border areas: if a road appears in more then one map, you will get phantom maneuvers. I don’t know for sure how Geofabrik splits the data at borders, but if there is any overlap between maps of neighboring areas, we will run into this issue.

On the other hand, I understand that, unlike the old planet extractor, which was an interactive web app, the new solution limits us to prebuilt binfiles – we can no longer assemble things on the fly as the user requests them.

An idea to counter than would be a binfile merge tool running on the user’s local system. This merge tool would take two binfiles and combine them into one, taking from each binfile the tile which has data in it:

if the left file has an empty tile and the right one has a tile with data in it, use the one from the right (and vice versa)
if both files have a tile and they both have the same data, use one at random

With the current approach of taking an area extract from OSM and processing that, this would run into two potential issues:

area extracts need to reflect the exact same version of the OSM data – if one file includes changes that the other one doesn’t, and these changes affect the border area, we will end up with inconsistencies and broken maps around the border
where the border cuts through a tile, the “left” and “right” file would have different data in that tile (each limited to objects on “their” sides of the border – so we would need to merge tiles where contents differ, which might make the whole thing more complex

As an alternative, we could do the following:

process the whole planet into one big binfile
create area extracts of that binfile by dropping tiles which are entirely outside the desired area (or replacing them with empty tiles, whatever is mandated by the binfile format)
supply a binfile merge tool which combines the non-empty tiles from a set of files (the above approach ensures that no two map files from the same release will have different data in the same tile, which eliminates the need to merge tiles and allows us to pick the first non-empty copy of a tile we encounter)

What I cannot answer is processing time. The approach below means we would be processing the entire planet, thus possibly hitting running time and/or storage limits for Github Actions.

jkoan commented 3 years ago

I don’t know for sure how Geofabrik splits the data at borders, but if there is any overlap between maps of neighboring areas, we will run into this issue.

See here: https://download.geofabrik.de/technical.html

An idea to counter than would be a binfile merge tool running on the user’s local system

We already have this, its called maptool ;-) and should be able to do the job

area extracts need to reflect the exact same version of the OSM data

currently i just take "-latest.osm.pbf" from every file which is already included, we could probably update this to the timestamped file which should guarantee this

As an alternative, we could do the following:
process the whole planet into one big binfile
This wouldn't be possible and takes days to convert (on my machine Ryzen 2600 and NVMe SSD). With the current approch it only takes about 1h but with north-america, south-america and europe (except Germany) missing. (I Need to copy and paste all those regions to filter out geofabrik's "special Regions"

What i don't know if the problem with phantom maneuvers is still current, as i think last time i read though the trac ticket i read it was fixed(?). Haven't made any tests on this

mvglasow commented 3 years ago

Thanks for the link. As I understand, each country extract includes a certain buffer around the border. So we would have duplicates in that buffer zone.

As for maptool, can it take two binfiles and merge them into one? How does it deal with tiles which have different information in binfile 1 and binfile 2? Can it merge tiles?

As for processing time, how does the current solution work around this? As far as I can see up to now, only select parts of the world get processed – which is obviously faster than processing the whole planet, but some parts of the world will not get map updates.

As for the phantom maneuvers, I last used overlapping maps some 9 years ago and it still was an issue then, no idea if anything has changed since – a map server which offered split binfiles became available soon after, and by now I am no longer constrained by file size restrictions on FAT, so I never bothered with separate map files again.

jkoan commented 2 years ago

As for maptool, can it take two binfiles and merge them into one? How does it deal with tiles which have different information in binfile 1 and binfile 2? Can it merge tiles?

I just found out that this is indeed possible by unzip'ing all binfiles, so that you get the tiles as individual files. After that its possible to run maptool like this: cat * | ~/git/navit-new/build/default/navit/maptool/maptool --binfile out.bin i am also trying this with multiple maps with find -type f -exec cat + | ~/git/navit-new/build/default/navit/maptool/maptool --binfile out.bin In the later case i have one subdirectory per map. Results pending, but the first try (with one map) did run throught.

jkoan commented 2 years ago

Okay, so the second merged went through as well, and it produced a valid binfile, so that's a win. The only problem is that now everything's flooded in water because of poly_water_tiled's interestingly all with 5 Coords. Also i needed to patch this line: https://github.com/navit-gps/navit/blob/0ed1660858a6b3055077f3b0d245c0316beb621c/navit/map/binfile/binfile.c#L846 Here i changed the 8 to a 16, which worked so far

martin-ac commented 2 years ago

Hi everybody, once in December last year and again some days ago I've been able to download several in size very different maps with size up to 18 GB without any error. Does someone know something about what happened to the download servers? The date the maps have been set up is now shown as 220106.

martin-ac commented 1 year ago

Hi, from time to time I take a look at >Navit :: Planet Extractor< at http://maps9.navit-project.org/ because this still is the place where I've been able to download .bin's e.g. for WesternEurope and others until dated 2022-07-14. Yesterday and today I tried again and found, that the maps9-server is currently not reachabel while maps3 is. Maps on maps3-server still are from 2022-07-14 and currently a download of a 7.9 GByte bin has reached 2.9 GByte at my pc and is still running with full speed.

The possibilities of Planet Extractor still are great and it would be VERY NICE to get something like this running again:

choose an area by country name or sub or a rectangle area
start the download
and have a usable bin as fast as possible

I saw activities from @jkoan at github navit-gps / gh-actions-mapserver: great and many thanks. I have to read above comments to see how to build something like "WesternEurope" out of this but the first look tells me this is more heavy to handle as a big download from GeoFabrik and use maplink to get the bin.

I think it would be a good idea to at least try and get maps9 running again still not knowing who really is responsible for this? And in addition maybe put an info onto the pages of all mapsx-servers on where to find help for the download of NEW maps in bin-file-format just to be able to get Navit running. I'm telling this because I'm afraid newbies might try to reach the Planet Extractor, fail without notice and so turn away from Navit ...

NGC935 commented 12 months ago

I'm with you, we have 2 servers left, one is a bit outdated (maps from 2022) and the last one is providing the last hope. The status of each server, map server nr.:

pesents a white page (hey at least), the linked js file should be from sedoparking.com, unused domain.
500 Gateway timeout, 62.75.137.162
Up and serving, maps from 14 juli 2022 :-/ more than nothing.
500 Gateway timeout, 81.17.243.218
500 Gateway timeout, 140.203.56.30
Up and serving, maps from 12. April 2023
500 Gateway timeout, 140.203.204.51
500 Gateway timeout, 88.198.161.18
500 Gateway timeout, 85.25.97.211

Navit is such a help while traveling without connectivity and those services save me a lot of space on my phone. Is there any activity?

navit-gps / navit

Map download at Planet Extractor not possible, almost same errors as in issue #1072 #1112