Contour lines

[vanyasvl]

Hello. Is it possible to add contour lines (altitude lines) to map?

[treee111]

Hi, actually it is not possible. I heard this idea once but never digged into it...

[limex]

Well it is - kind of - possible. But needs extra steps.

• Use Program srtm2osm (https://wiki.openstreetmap.org/wiki/Srtm2Osm), to download the the elevation data for a specified area and convert it into an OSM-format contour file
• I can't remember if there was a step to merge the data to one OSM File?
• This is the OSM data that needs to be fed to the existing application.

Disclaimer:

• I didn't check the code of wahooMapsCreator
• Didn't test the thesis.

[geekitus]

Hello,

PHYGHTMAP seems to be a better choice no ?

A good exemple from https://oruxmaps.forumotion.com/t2216-contourlines-added-for-oruxmapsforge

#test of merging osm + srtm and render in mapsforge using a small aera in LowerAustria along the river Danube
#Merging Srtm + osm: Thanks to user Fireball2 at http://forum.openstreetmap.org/viewtopic.php?id=5359

#Contours with PHYGHTMAP (output is in version 0.6 so no further conversion is needed)
C:\Python27\Scripts\phyghtmap -a 14.0:48.0:16.5:48.5 -o D:\maps\Mapsforge\at_test\srtm6 --source=view1,view3,srtm3 -s 25 -c 500,100 --start-node-id=2000000000 --start-way-id=1000000000 --write-timestamp --max-nodes-per-tile=0 --max-nodes-per-way=0

#cut out osm-data from austria.osm
osmosis --rx D:\maps\OSM_Data\austria.osm.bz2 --tee 1 --bounding-box top="48.50" bottom="48.00" left="14.00" right="16.50" --wx D:\maps\Mapsforge\at_test\at_test.osm

#merge srtm + osm
osmosis --rx D:\maps\Mapsforge\at_test\at_test.osm --sort-0.6 --rx D:\maps\Mapsforge\at_test\srtm6.osm --sort-0.6 --merge --wx D:\maps\Mapsforge\at_test\at_test_srtm.osm

#mapsforge output (use mapwriter plugin WITH dependencies)
osmosis --rx file=D:\maps\Mapsforge\at_test\at_test_srtm.osm --mw file=D:\maps\Mapsforge\at_test\at_test_srtm.map bbox=48.0,14.0,48.5,16.5 type=hd tag-conf-file=D:\maps\Mapsforge\at_test\tag-mapping-srtm.xml

#Custom tag-mapping-srtm.xml:
#added lines to tag-mapping.xml:

   <!-- Elevation TAGS -->
   <ways>
      <osm-tag key="contour_ext" value="elevation_minor" zoom-appear="13" />
      <osm-tag key="contour_ext" value="elevation_medium" zoom-appear="12" />
      <osm-tag key="contour_ext" value="elevation_major" zoom-appear="12" />
   </ways>

#Custom [render].xml
#added lines:

                <!-- contour lines -->
            <rule e="way" k="contour_ext" v="*">
               <rule e="way" k="contour_ext" v="elevation_major">   
                  <line stroke="#0a630c" stroke-width="1.1" stroke-dasharray="7,5"/>
               </rule>
               <rule e="way" k="contour_ext" v="elevation_medium">   
                  <line stroke="#0a630c" stroke-width="0.8" stroke-dasharray="7,5"/>
               </rule>
               <rule e="way" k="contour_ext" v="elevation_minor">   
                  <line stroke="#0a630c" stroke-width="0.4" stroke-dasharray="7,5"/>
               </rule>
            </rule>

I will try to test.

Have a good day.

[limex]

@geekitus: that would be great, if it works. Because being also coded in python would make a better integration than with the srtm2osm executable. Please keep us update about the results of your test. BR Gü

[treee111]

Hi @geekitus, did you have any success / intermediate result? I would work on top of them or tryout by myself with the information in this issue. greetings!

[Ebe66]

Did some testing today, but I got stuck with PHYGHTMAP on windows with the latest matplotlib not having the _contour function anymore for a long time now it seems. So I got a phyton import error on _contour.

I did a test with srtm2osm and got a osm file out of it to test further with. Maybe later this week.

[vanyasvl]

Did some testing today, but I got stuck with PHYGHTMAP on windows with the latest matplotlib not having the _contour function anymore for a long time now it seems. So I got a phyton import error on _contour.

I did a test with srtm2osm and got a osm file out of it to test further with. Maybe later this week.

Just install matplotlib 3.4.3 via pip

[Ebe66]

Thanks! It's starting without error (after pip install bs4 too)

[Ebe66]

@vanyasvl I'm am at work right now but I ran into an error yesterday evening (late). Don't have the error message at hand but I'm hoping you know what I'm running into... I used a variation of this line: phyghtmap -a 14.0:48.0:16.5:48.5 -o D:\maps\Mapsforge\at_test\srtm6 --source=view1,view3,srtm3 -s 25 -c 500,100 --start-node-id=2000000000 --start-way-id=1000000000 --write-timestamp --max-nodes-per-tile=0 --max-nodes-per-way=0. First I got an error that I needed a license for the srtm3 data. Ok, registered and tried again. Now I get an error on a textfile in the cache directory that does not exist yet. From memory something about an idex file. There are other files in there after the first run btw. Hope this means something to you, if not I post the error message tonight and try further. This elevation stuff is pretty new to me.

[vanyasvl]

@vanyasvl I'm am at work right now but I ran into an error yesterday evening (late). Don't have the error message at hand but I'm hoping you know what I'm running into... I used a variation of this line: phyghtmap -a 14.0:48.0:16.5:48.5 -o D:\maps\Mapsforge\at_test\srtm6 --source=view1,view3,srtm3 -s 25 -c 500,100 --start-node-id=2000000000 --start-way-id=1000000000 --write-timestamp --max-nodes-per-tile=0 --max-nodes-per-way=0. First I got an error that I needed a license for the srtm3 data. Ok, registered and tried again. Now I get an error on a textfile in the cache directory that does not exist yet. From memory something about an idex file. There are other files in there after the first run btw. Hope this means something to you, if not I post the error message tonight and try further. This elevation stuff is pretty new to me.

Check phyghtmap command-line arguments in my build script for Garmin here https://github.com/vanyasvl/cyprus-map-sid/blob/master/build.sh

[Ebe66]

Thanks, but here you seem to manually download the elevation tiles. I thought phyghtmap was supposed to do that for you? I'll, try to, look further into it. If it works it seems to be pretty easy to incorporate into the Wahoo map maker

[Ebe66]

@vanyasvl phyghtmap -a 6:44:9:47 -o high_alps --source=view1 (from the man page example) gives FileNotFoundError: [WinError 2] The system cannot find the file specified: 'hgt\viewfinderHgtIndex_1.txt' view3 results in FileNotFoundError: [WinError 2] The system cannot find the file specified: 'hgt\viewfinderHgtIndex_3.txt' and srtm3 in FileNotFoundError: [WinError 2] The system cannot find the file specified: 'hgt\hgtIndex_3_v3.0.txt'

Maybe I need to do something with --rewrite-indices, but how? Something like below gives: PS D:\Windows-Wahoo-Map-Creator-Osmosis> phyghtmap -a 6:44:9:47 -o high_alps --source=srtm3 --rewrite-indices generating index in hgt\hgtIndex_1_v2.1.txt ... Traceback (most recent call last): File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\Scripts\phyghtmap-script.py", line 33, in sys.exit(load_entry_point('phyghtmap==2.23', 'console_scripts', 'phyghtmap')()) File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\site-packages\phyghtmap-2.23-py3.9.egg\phyghtmap\main.py", line 659, in main File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\site-packages\phyghtmap-2.23-py3.9.egg\phyghtmap\main.py", line 247, in parseCommandLine File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\site-packages\phyghtmap-2.23-py3.9.egg\phyghtmap\NASASRTMUtil.py", line 496, in rewriteIndices File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\site-packages\phyghtmap-2.23-py3.9.egg\phyghtmap\NASASRTMUtil.py", line 473, in makeIndex File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\site-packages\phyghtmap-2.23-py3.9.egg\phyghtmap\NASASRTMUtil.py", line 373, in makeNasaHgtIndex File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 214, in urlopen return opener.open(url, data, timeout) File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 523, in open response = meth(req, response) File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 632, in http_response response = self.parent.error( File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 561, in error return self._call_chain(args) File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain result = func(args) File "C:\Users\ebodd\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 641, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found

I'm sure I'm doing something stupid, but what?

[vanyasvl]

phyghtmap has an issue with hsgt data downloading. Download files manually like I do in my script

[Ebe66]

@vanyasvl Just got it to work (at least it downloads and does something) Turned out all I needed to do was a pip install lxml...

[Ebe66]

Still running into some troubles, but also some success! While rendering Scotland osmosis errors out halfway through at tile 30 of 72 and there are some weird diagonal(ish) lines which I don't understand yet. Just proof of concept, if working would also need somebody with a newer unit than a bolt 1 to look how it is displayed on the device(s) The screenshot is of cruiser on my pc with a modified roam 1 mapsforge theme.

[Ebe66]

And finally success. Had one of those rare bright moment ;-). I had taken over the start node and way number of the posted example. But, that was in 2012 so changed --start-node-id=2000000000 to --start-node-id=3000000000 and did the same for ways and voila. The complete map set for bonnie Scotland rendered just fine and no artifacts (here) anymore.

[limex]

@Ebe66 : I don't go into every details of the tread above. And even if I did: In some month I will not remember the solution any more. As many might experience the same: Any chance that you create a summary of the additons/changes needed to get contour lines added? Or even add it to the repo readme? Thanks in advance.

[Ebe66]

@limex I don't know if Treee111 wants to incorporate this, but sure I can make a list of the things I ran into (on windows). The method used was the phyghtmap way as described above near the top (or more precise the later version of it posted in the mentioned thread).

• The first issue I ran into was that matplotlib does not have the _contour function/functionality anymore. Solution: install an older version with pip install matplotlib==3.4.3
• Then it turns out you now need to register to use the srtm3 data. Solution: make a free account here https://ers.cr.usgs.gov/login?redirectUrl=https://ers.cr.usgs.gov/ and add the command line arguments for it --earthexplorer-user= and --earthexplorer-password=
• Now phyghtmap ended with a runtime error. Took me some time but... Solution: All I needed to do was pip install lxml which was not mentioned on the programs manual page.

Basically it was running now and I could generate my tile maps. But...

• Strange diagonal(ish) lines in the output. See my first screenshot above. Solution: the --start-node-id and --start-way-id used in the example script near the top where probably fine in 2012 when it was posted but are now somewhere in the used range of id's. So id's were conflicting with already existing map data. Upping them to a higher value resulted in the tile from the screenshot rendering good.
• But the next tile again had some lines, less but lines. Solution: Find out what the actual highest node and way id's used are and use start id's above that number. I found them on this page: https://osmstats.neis-one.org/?item=elements

Mind, I'm most certainly not a expert in this field but my current command line looks something like this: phyghtmap -a left:bottom:right:topcoordinates -o outFile -s 10 -c 100,50 --source=view1,view3,srtm3 --jobs=8 --viewfinder-mask=1 --start-node-id=20000000000 --max-nodes-per-tile=0 --start-way-id=2000000000 --write-timestamp --no-zero-contour --earthexplorer-user= --earthexplorer-password=

Hope it helps out someone.

Edit: Have had feedback that the matplotlib _countour problem is also on macOS but fortunately the fix is also the same

[masc4ii]

@Ebe66 helped me to get it working and integrated into @treee111 's code. Thanks again! I tested on macOS BigSur and it works fine so far. I use python3.10 via brew. Additionally to before I had to do:

pip3 install bs4 pip3 install lxml

Download and unpack matplotlib-3.4.3.tar.gz from https://pypi.org/project/matplotlib/3.4.3/#files sudo python3 setup.py install

Download and unpack http://katze.tfiu.de/projects/phyghtmap/phyghtmap_2.23.orig.tar.gz sudo python3 setup.py install

And here you find a working version (some things can be made better). In osm_maps_functions.py, line 551, insert your account data from https://ers.cr.usgs.gov. Here it would be cool to use arguments/parameters to not have the account data in source code.

https://github.com/masc4ii/wahooMapsCreator

[masc4ii]

More testing and optimization on theme with elevation lines code done. Here an example (Bolt v2, Gran Canaria). Find code in my fork - as always.

[treee111]

Hi you both - @Ebe66 and @masc4ii, thanks a lot for your effort in getting it running! Very nice and as more or less always some geniuses getting it to run and I integrate it into the repo, make it equal to the existing coding, have validations etc.

That's done from my side for contour lines. @masc4ii it would be nice if you could test it on your end if my solution is also working for you! I also switched back to python 3.10 to get matplotlib 3.4.3 running. I write a solid PR now, if you are faster you can see all changes (including setting everything up, might not be needed for you anyway) in the PR #188

Please make sure that shapely is installed via conda-forge as written in this link! Tests can go on with branch contour-lines or alpha version installed via pip install wahoomc==4.0.0a3 pip install wahoomc==4.0.0a5 pip install wahoomc==4.0.0a8 If you have any comments, don't hesitate to use the PR review functionality!

[masc4ii]

@treee111 : Thank you for implementing! Now I found the time to test. I tried to create canary islands. python3 -m wahoomc cli -co canary_islands -con

The processing stops with this error:

INFO:# Generate land for each coordinate
INFO:+ (tile 1 of 17) Coordinates: 114,106
ERROR:subprocess:'File "/Users/masc/Desktop/wahooMapsCreator-contour-lines/wahoomc/resources/shape2osm.py", line 197'
ERROR:subprocess:'print("<osm version=\'0.6\'>", file=open_file)'
ERROR:subprocess:'^'
ERROR:subprocess:'SyntaxError: invalid syntax'
ERROR:! Error creating land.osm for tile: 114,106

When commenting this line, I get a very similar error in line 215. Something I missed to install? I use the macOS brew python 3.10. Installed shapely is 1.8.5. In my fork this line looks identical and no such error pops up.

Edit: found it. "python" in code always has to be "python3" on my macOS. Changed -> works. Cool! Where do you save the user data needed for contour lines?

[treee111]

Hi @masc4ii! Sorry for the late reply..

thanks a lot for testing! Nice, that you found that out by yourself!

I assume you have your installation in your main/standard Python environment and not via Anaconda. That might be the reason for you to have python3. In earlier times I had this in the coding as well but it turned out to be not needed.

I'd recommend using Anaconda and the setup I described in the docs, its very easy to setup and you have it isolated from the rest of your system.

Anyway, I am glad that it worked also in your setup 👍

I'll save the userdata in ~/wahooMapsCreatorData/.config.json file where other tool-related settings are stored.

[treee111]

implemented with #188

[masc4ii]

Thanks for your answer. Found the file now! I found some areas where the contour lines failed. One example is near Plauen (136/86). In the middle of the tile the lines are missing. In the near of this, towards Czech Republic that happens often.

[Ebe66]

I have the same in my maps I saw. At first I thought it might be a fault with phyghtmap because of the square cutout but at the Czech border it follows the border. When I load the generated height .pbf file in maperitive for this tile you can clearly see the voids.

I think I will manually try some different source settings.

[Ebe66]

Well, trying srtm opens the python module hell for me. It needs ogr and osr. After changing the source to use "from osgeo import ogr" and the same for osr I get loads of other errors. Switching to view3 does give a complete picture. But the viewfinder site confuses me and sometime contradicts itself it seems. But if this part is still valid it might even be better to use the view3 data: "NASA has only released 1" SRTM data for US territory. I have created HGT files with a resolution of 1" for some parts of Europe from topographic maps. For most of the Alps and the most rugged parts of the Pyrenees, these were created from the most detailed sources, and should be accurate. Elsewhere, I have 1" DEM data for Great Britain, Ireland, France (excluding Corsica), Germany, Benelux, Denmark, Norway, Sweden and north east Spain. But these were created from Russian maps of scale 1:100,000, which are less accurate than SRTM. Consequently, for these areas, where 3" DEM data is available from this page or from SRTM, they should be generally better than the 1" data, despite the lower resolution. "

[treee111]

nice finding! So can there be applied a general coding/config change to get better results over all? Or is it specific to certains regions/countries?

[Ebe66]

Hard to say for me. That is what I mean by the contractionary information. The obvious choice would be to use srtm1 data (if someone can make it work) but the info on the viewfinder site on the one hand says that only the US is available but on the other hand at a different place the whole planet excluding north/south of 60". As long as srtm is not working the best option appears to be using view3. You, having a gui, could consider making it a checkbox option? On the viewfinder homepage it says "Last update 15 May 2022 with addition of more and improved 1" DEM for Europe." which is fairly recent considering other dates on there but also explicitly implies it is not complete.

[Ebe66]

Got srtm1 to work. turns out that I also had to update my gdal installation to 3.6.3 And, the Czech part now present

No real suggestion on what elevation source should be used. I'm no expert on this too, but reading the viewfinder site it seems all have their pros and cons. Completeness can only be checked visually and accuracy...??? Best option is probably put something in the readme and make it a checkbox on what to use

[masc4ii]

What do we have to change, to get this working? Looks great!

[Ebe66]

@masc4ii Not sure if it's still needed (or in all cases) but I had to change my gdal.py in folder C:\Users\ebodd\AppData\Local\Programs\Python\Python39\Lib\site-packages\osgeo (in my case) at line 1984 to from osgeo import ogr from osgeo import osr

I also needed to install a newer version of gdal (that's why I'm not sure the above is still needed).

The, best, choice of elevation data sources seems to be region dependent. atm I have it set to --source=srtm1,view1,view3,srtm3 So the example above uses srtm1 data.

[masc4ii]

@Ebe66 thank you. The gdal.py seems to be different on my system. The line looks not as I would expect it. When searching "from osgeo import" in this file I get many results - no idea which of those lines to change. So I did not change it for now.

What version of gdal do you have installed? I have v3.6.2.

Then I tried just to add the "srtm1" into the call in osm_maps_functions.py, but then I get maaaany errors.

Edit: how bad... it is not just because of this change. Elevation line generating is completely broken for me now. It seems elevation data is not downloaded. zip file has 0Byte.

[Ebe66]

I just shut down my computer but the lines where import ogr and osr which I changed to from osgeo import etc. If you look on my GitHub at my (very outdated) readme there is a link for gdal. I just had to update to the latest there. Sorry but don't know the version from the top of my mind. 3.7?? But also remember I'm not using tree's version, so no idea how that translates to the anaconda setup

[masc4ii]

Seems there was a problem with the elevation data server. Now it is working just fine. All I had to do was changing to this "--source=srtm1,view1,view3,srtm3" and I got the srtm1 data. My gdal is (as written) 3.6.2. No more change needed here. Quality of view1 data often looks better than srtm1. Therefor this elevation "hole" is fixed. Thank you!

[treee111]

Hi you both - thanks for digging into that and clarifiying! 👍 I implemented both changes in #203.

[masc4ii]

Hi @treee111 , your change looks like we use srtm1 always now. I am not sure if this is good. view1 offers better quality for contour lines in general, apart from the few missing/faulty areas. srtm1 here and there looks very rough compared to view1. Maybe the idea from @Ebe66 , having a checkbox in GUI (or option in commandline) could solve this.

[treee111]

Thanks for the hint @masc4ii! Now I got it. I thought in the argument would be some kind of ordering what suits better but it isn't that clever ;-)

Option is available via CLI and GUI. I just realized now that generating contour lines from GUI doesn't work for the first time, it crashed when we ask the user for credentials. But that's another topic ;-)

[masc4ii]

Great! Thank you very much. What parameter do I have to set via CLI, if I like to create with srtm1? I just found the already known "-con" in the docs.

[masc4ii]

Found it: "-con -srtm1" seems to be it.

On my testruns I still get a problem. Tile 134/90 fails with 'java.lang.OutOfMemoryError: Java heap space'.

I noticed that the elevationXXX.osm file is huuuge for this tile: for view1 it is 4.6GB, for srtm1 it is 76GB. merged.osm.pbf is around 1.1GB.

I tried before the wmc command (16GB because I think we need a lot of memory here): set JAVACMD_OPTIONS=-server -Xmx16G

But this doesn't help. For neighbor tiles it works. For 134/91 the elevationXXX.osm is just 1.1GB and merged.osm.pbf is 92MB. Something seems to be very wrong for 134/90. @Ebe66 : for your maps this tile run through without errors?

Edit: I had luck with view3. But this looks ugly. Will test more.

[treee111]

Hi @masc4ii, you'r right, the CLI argument is -srtm1. You can always use -h in CLI to get an overview of available arguments. For completeness, I added it to the docs, missed that!

Thanks for the hint & sorry for the late reply.

Concerning the behaviour of 134/90 you did what I also mentioned in the FAQ, setting the RAM higher. 76GB for elevation data does not really look OK in comparison of the others. I don't think Mount Everest is there which would be a valid reason ;-)

you guys are awesome 👍

[masc4ii]

I found a solution. You can tell osmosis to use HD instead of RAM, using parameter "type=hd". Might be a little slower... but no more crash.

def create_map_files(self, save_cruiser, tag_wahoo_xml):
....
      # Non-Windows
            else:
                cmd = ['osmosis', '--rb', merged_file,
                       '--mw', 'file='+out_file_map, 'type=hd']

Resoluting .map file is 74MB. Without elevation it is "a little" smaller, 14MB. 🤣