java.lang.OutOfMemoryError: GC overhead limit exceeded while trying to create maps for europe.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. download from svn
2. download osm file from http://wiki.openstreetmap.org/wiki/Planet.osm
3. Run OSMandroidConverter with 29G europe file.

What is the expected output? What do you see instead?
   1. osm_android/maps - the map's directory containing binary map tiles
   2. osm_android/names - directory containing street names and coordinates 

What version of the product are you using? On what operating system?
revision 34

Please provide any additional information below.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
    at java.util.Arrays.copyOfRange(Arrays.java:3221)
    at java.lang.String.<init>(String.java:233)
    at org.apache.xerces.utils.UTF8DataChunk.toString(UTF8DataChunk.java:275)
    at org.apache.xerces.utils.StringPool.toString(StringPool.java:687)
    at org.apache.xerces.parsers.SAXParser.startElement(SAXParser.java:1364)
    at
org.apache.xerces.validators.common.XMLValidator.callStartElement(XMLValidator.j
ava:1284)
    at
org.apache.xerces.framework.XMLDocumentScanner.scanElement(XMLDocumentScanner.ja
va:1806)
    at
org.apache.xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDoc
umentScanner.java:1182)
    at
org.apache.xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java
:381)
    at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1098)
    at org.apache.xerces.framework.XMLParser.parse(XMLParser.java:1139)
    at
com.google.code.osmandroidconverter.main.MapBuilder.parseOsmFile(MapBuilder.java
:60)
    at
com.google.code.osmandroidconverter.main.OSMconverter.main(OSMconverter.java:17)

This error was recieved with default settings in Eclipse.

Original issue reported on code.google.com by WLigtenb...@gmail.com on 4 Oct 2009 at 10:31

[GoogleCodeExporter]

Increasing xmx and xms results in out of heap space error.
This clearly indicates that the OSMandroidConverter cannot handle large input 
files
on normal computers. Probably due to the SAX xml handling. Maybe this can be 
done in
another way.

Original comment by WLigtenb...@gmail.com on 4 Oct 2009 at 7:08

[GoogleCodeExporter]

Hi there, and thanks for trying the app.

Regarding the issue...I don't think that parsing such a large file is a good 
idea in
this stage of the project, because (just to name a few):

1) all the labels (street names, parks, lakes, etc) are stored in a centralized
database ... you can imagine how slow querying it will be on a mobile device

2) at this time, data is read into memory, converted into the binary format and 
then
spliced into smaller tiles; this could be improved

3) there is no current easy way to determine to which city a street belongs;
osm-android currently makes a list of all settlements found in a file and then 
takes
each map-item and tries to find the best match; this requires storing all the 
map
items in memory before writing them to files; this also could be improved

My feeling is that we should approach a "country-based" solution, either by 
parsing
each country individually or by separating countries in a larger file; usually 
the
data from a country can be managed by a mobile device and I see no real benefit 
in
mixing stuff from different countries. In this way, we'll achieve a further 
split of
the two-dimensional space, which results in less data to be handled 
unnecessarily by
the mobile device.  

Original comment by iulian.b...@gmail.com on 6 Oct 2009 at 7:28

[GoogleCodeExporter]

Regarding a country-based solution. This might "help" in the European situation,
however, the same would hold for the US. (Then you might argue a State-based 
solution)
However, I would really like to replace my TomTom device with this software in 
the
future. And many people like me use their navigation device on holiday as well.
Actually, then it can really matter!
How would you want to tackle the route planning from one country into the other?

So on to solutions on how to enable continent wide navigation (because it is 
possible
(TomTom runs on my older WinMo phone).

1) Make a distinction between streets and other names, like parks etc. 
Parks etc. should be POI's, placed into categories.
Streets are organised to be found after the selection of a village. (this is 
also how
TomTom does it) These things should improve database speed.

2) As you say, this could (should really) be improved. And is most of the times
"easy". (Might require coding hours, but is no rocket science)

3) If I understand you correctly, this problem arises from the OSM datafile. 
This is
not a limitation of the osm-android, but of the OSM file itself, right?
Then this is actually quite strange that this information is not in OSM. 
However,
this procedure should (I assume) not require to have everything loaded into 
memory.
One could divide the map into squares of reasonable size and look at nine of 
them at
a time. And moving the center of the nine squares along the entire map. (I 
don't know
if you understand my description, but this is the best I can do now.)

As I said before, I really want to be able to ditch my TomTom, so I would be 
glad to
help if there are some bite size things to do.

Original comment by WLigtenb...@gmail.com on 6 Oct 2009 at 7:42

[GoogleCodeExporter]

Even a small country like Slovakia took up 1.3GB of memory to process.  That's 
also coincidently the size of the processed .OSM file. Or it's no coincidence 
and the whole .OSM file needs to be pulled to the memory? The theory for 
SAXParser is, that it's event driven and should be able to process files larger 
then the memory available.

Original comment by tibor.arpas on 16 Sep 2010 at 6:02