search

osmandapp / OsmAnd-tools

OsmAnd tools to generate new maps & to test OsmAnd files on PC
osmand.net
Other
94 stars 82 forks source link

OsmAndMapCreator IndexOutOfBoundsException for big extracts #687

Open feludwig opened 11 months ago

feludwig commented 11 months ago

Hello! I have been trying to create .obf maps of .osm.bz2 extracts, and OsmAndMapCreator generate-obf works well for the following: Switzerland.osm.bz2 (osm.bz2 692MB, obf 783MB) But it crashes while writing route index when the input file is bigger than about 1GB (I am giving it -Xmx 25G of memory to use) on the following: (germany) baden-wuerttemberg-latest.osm.bz2 (osm.bz2 854MB, unfinished obf 696MB) (germany) bayern-latest.osm.bz2 (osm.bz2 1.2G, unfinished obf 889MB) Here is the end of the log for baden-wuerttemberg:

INFO: Done 98.965 %.                                                                                                                                                         
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: Task index_lo_level_ways: work total has changed to -1                                                                                                                 
Oct 07, 2023 6:15:34 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: pack_rtree_map started after 2071665 ms - -1                                                                                                                           
Pack.packTree : Size of elmts: 10458811                                               
Pack.packTree : Size of elmts: 3171796                                                
Pack.packTree : Size of elmts: 1758434
Pack.packTree : Size of elmts: 168685                                                                                                                                        
Pack.packTree : Size of elmts: 2345214                                                                                                                                       
Pack.packTree : Size of elmts: 184318                                                                                                                                        
Oct 07, 2023 6:18:17 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: pack_rtree_transp started after 162578 ms - -1               
Pack.packTree : Size of elmts: 36359                                                  
Oct 07, 2023 6:18:18 PM net.osmand.impl.ConsoleProgressImplementation print                                                                                                  
INFO: Writing map index to binary file... started after 666 ms - -1                                                                                                          
Oct 07, 2023 6:55:14 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex                        
INFO: MAP level SIZE : 469256321                                                                                                                                             
Oct 07, 2023 7:14:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex           
INFO: MAP level SIZE : 157953833                                                                                                                                             
Oct 07, 2023 7:26:33 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex           
INFO: MAP level SIZE : 87773842                                                                                                                                              
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex         
INFO: MAP level SIZE : 8657172                                                                                                                                               
Oct 07, 2023 7:28:29 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapIndex 
INFO: MAP INDEX SIZE : 723805292                                                                                                                                             
Oct 07, 2023 7:28:29 PM net.osmand.impl.ConsoleProgressImplementation print
INFO: Writing route index to binary file... started after 4211731 ms - -1
Oct 07, 2023 7:28:48 PM net.osmand.obf.preparation.IndexCreator generateIndexes
SEVERE: Log exception
java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -26077 out of bounds for length 53395
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)

(IndexRouteCreator.java:1594 that line does appear 3 consecutive times) The two extracts from Germany are geofabrik extracts, because I was suspecting that the Bayern input I had was corrupted (it had crashed, at the same point). I have just split Bayern along 49 latitude and will start converting the two halves separately and report how it went. These extracts take more than 10h to convert so it was a bit of an adventure. I'm running the latest version from github:

$ git log |head
commit e5b6027c2485a39adf5aadcf2761d2f0200906a4
Author: ivanPyrohivskyi <ivan.pirogovsky@gmail.com>
Date:   Wed Oct 4 14:24:28 2023 +0300

Do you also want logs for other and/or successful extracts ? Or the complete log ? Should I run the conversion, or compile MapCreator, again with some specific flags ?

feludwig commented 11 months ago

The Bayern split accross 49th parallel was successful and the two files are Bayern-above49.obf 625MB Bayern-below49.obf 751MB So the isssue is with input size...

vshcherb commented 11 months ago

Did you try pbf as input file cause it might work better as it tested more

feludwig commented 11 months ago

No, but I'll go try that as well

feludwig commented 11 months ago

Yes, it also crashes: input: bayern-latest.osm.pbf from geofabrik size: 712M incomplete output: Bayern-latest.obf size: 890M

Oct 16, 2023 5:26:23 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapLevelIndex                                                                                
INFO: MAP level SIZE : 12645757                                                                                                                                              
Oct 16, 2023 5:26:23 PM net.osmand.obf.preparation.BinaryMapIndexWriter endWriteMapIndex                                                                                     
INFO: MAP INDEX SIZE : 927596053                                                                                                                                             
INFO: Writing route index to binary file... started after 5174106 ms - -1                                                                                                    
Oct 16, 2023 5:26:41 PM net.osmand.obf.preparation.IndexCreator generateIndexes                                                                                              
SEVERE: Log exception                                                                                                                                                        
java.lang.IndexOutOfBoundsException: Index -28761 out of bounds for length 58308                                                                                             
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)                                                                                      
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)                                                                            
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)                                                                                      
        at java.base/java.util.Objects.checkIndex(Objects.java:372)                                                                                                          
        at java.base/java.util.ArrayList.get(ArrayList.java:459)                                                                                                             
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)                                                                                      
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)                                                                                    
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)                                                                                    
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)                                                                                                      
        at net.osmand.MainUtilities.main(MainUtilities.java:127)                                                                                                             

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -28761 out of bounds for length 58308
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.get(ArrayList.java:459)
        at net.osmand.osm.MapRoutingTypes.getTypeByInternalId(MapRoutingTypes.java:335)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.decodeNames(IndexRouteCreator.java:1337)
        at net.osmand.obf.preparation.IndexRouteCreator$RouteWriteContext.retrieveObject(IndexRouteCreator.java:1382)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1532)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryMapBlock(IndexRouteCreator.java:1594)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndexBlocks(IndexRouteCreator.java:1498)
        at net.osmand.obf.preparation.IndexRouteCreator.writeRouteSections(IndexRouteCreator.java:743)
        at net.osmand.obf.preparation.IndexRouteCreator.writeBinaryRouteIndex(IndexRouteCreator.java:704)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:631)
        at net.osmand.obf.preparation.IndexCreator.generateIndexes(IndexCreator.java:484)
        at net.osmand.MainUtilities.generateObf(MainUtilities.java:297)
        at net.osmand.MainUtilities.main(MainUtilities.java:127)
vshcherb commented 11 months ago

We don't build bayern separately cause it might be too big, we build as separate regions but the crash is strange. It happens on the last stage...

feludwig commented 10 months ago

Oh I see, so the workaround is something like

vshcherb commented 10 months ago

Yes that's what we do actually... processing large file is just takes too much memory and we even can run in parallel using 1 mapcreator batch utility and local docker