search

agrenott / pyhgtmap

Generate OSM contour lines from NASA SRTM (or other digital elevation model sources) data
GNU General Public License v2.0
13 stars 3 forks source link

Node ids are not sorted (on big dataset) #20

Closed agrenott closed 1 year ago

agrenott commented 1 year ago 
          Well I now have another problem with the newest version - seems there are problems with order - when trying to split the output with mkgmap splitter - I get the error:

Error: Node ids are not sorted. Use e.g. osmosis to sort the input data. This is not supported with keep-complete=true or --problem-list uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted

Phygtmap produced data never had that problem. Running osmosis on this data to sort it - would again fail due to size constraints.

I tried if it's related to a map being south of the Equator - and used Fiji which was fine except for getting this error message although I don't see a problem with the data output on first glance:

hgt file /home/contourlines/hgt/SRTM1v3.0/S20W180.hgt: 3601 x 3601 points, bbox: (-180.00000, -20.00000, -179.00000, -19.00000), checking polygon borders /usr/local/lib/python3.10/dist-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection return lib.intersection(a, b, **kwargs)

Not sure how what this means. Maybe input data is problematic on that tile?

I then tried Ecuador to have a map passing the equator. Also no problem. France neither. Seems to affect only very big countries - and I think that problem already has been there with pyhgtmap v 3.1 (but I don't know right now - just know that splitting Europe with mkgmap splitter failed which was produced by v 3.1

Originally posted by @extremecarver in https://github.com/agrenott/pyhgtmap/issues/16#issuecomment-1573955720

agrenott commented 1 year ago

I've been able to reproduce on Africa dataset. It indeed looks like an integer overflow:

java -jar /mnt/g/git/garmin_mtb/tools/splitter-r653/splitter.jar africa_lon-27.26_66.72lat-60.32_37.78_view1,view3,srtm3v3.0.osm.pbf
...
2,120,000,000 nodes parsed... id=2129999999
2,130,000,000 nodes parsed... id=2139999999
2,140,000,000 nodes parsed... id=-2144967297
2,150,000,000 nodes parsed... id=-2134967297
...
2,110,000,000 nodes parsed... id=2119999999
2,120,000,000 nodes parsed... id=2129999999
2,130,000,000 nodes parsed... id=2139999999
Error: Node ids are not sorted. Use e.g. osmosis to sort the input data.
This is not supported with keep-complete=true or --problem-list
uk.me.parabola.splitter.SplitFailedException: Node ids are not sorted
        at uk.me.parabola.splitter.MultiTileProcessor.storeCoord(MultiTileProcessor.java:497)
        at uk.me.parabola.splitter.MultiTileProcessor.processNode(MultiTileProcessor.java:126)
        at uk.me.parabola.splitter.AbstractMapProcessor.consume(AbstractMapProcessor.java:82)
        at uk.me.parabola.splitter.OSMFileHandler.execute(OSMFileHandler.java:157)
        at uk.me.parabola.splitter.ProblemLists.calcMultiTileElements(ProblemLists.java:255)
        at uk.me.parabola.splitter.Main.useProblemLists(Main.java:503)
        at uk.me.parabola.splitter.Main.start(Main.java:127)
        at uk.me.parabola.splitter.Main.main(Main.java:81)
extremecarver commented 1 year ago

Can confirm, best push a new stable version. This would be the first stable version that works correctly with big areas! And yes it's a big improvment over the original phygtmap that was much slower, and used more RAM. Both version 3.0 and 3.1 were still affected by this (and other) bugs.