citygml4j / citygml-tools

Collection of tools for processing CityGML files
Apache License 2.0
115 stars 18 forks source link

Large gml file conversion to Cityjson #23

Closed Saji70 closed 2 years ago

Saji70 commented 2 years ago

Dear Citygml team

I want to convert large CityGML file (~4GB) to CityJson but I encontered following Error: any advice to resolve it?

[19:49:35 ERROR] The following unexpected error occurred during execution. java.lang.OutOfMemoryError: GC overhead limit exceeded at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source) at sun.misc.FloatingDecimal.parseDouble(Unknown Source) at java.lang.Double.parseDouble(Unknown Source) at com.sun.xml.bind.DatatypeConverterImpl._parseDouble(DatatypeConverterImpl.java:207) at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$21.parse(RuntimeBuiltinLeafInfoImpl.java:749) at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$21.parse(RuntimeBuiltinLeafInfoImpl.java:747) at com.sun.xml.bind.v2.runtime.reflect.ListTransducedAccessorImpl.processValue(ListTransducedAccessorImpl.java:110) at com.sun.xml.bind.v2.runtime.reflect.ListTransducedAccessorImpl.parse(ListTransducedAccessorImpl.java:125) at com.sun.xml.bind.v2.runtime.unmarshaller.ValuePropertyLoader.text(ValuePropertyLoader.java:37) at com.sun.xml.bind.v2.runtime.unmarshaller.StructureLoader.text(StructureLoader.java:261) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.text(UnmarshallingContext.java:560) at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.text(InterningXmlVisitor.java:63) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.processText(StAXStreamConnector.java:308) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleEndElement(StAXStreamConnector.java:186) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:155) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:385) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:356) at org.citygml4j.builder.jaxb.xml.io.reader.JAXBSimpleReader.nextFeature(JAXBSimpleReader.java:133) at org.citygml4j.tools.command.ToCityJSONCommand.call(ToCityJSONCommand.java:107) at org.citygml4j.tools.command.ToCityJSONCommand.call(ToCityJSONCommand.java:50) at picocli.CommandLine.executeUserObject(CommandLine.java:1953) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2422) at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2424) at picocli.CommandLine$RunAll.handle(CommandLine.java:2419) at picocli.CommandLine$RunAll.handle(CommandLine.java:2383) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at org.citygml4j.tools.CityGMLTools.process(CityGMLTools.java:131) at org.citygml4j.tools.CityGMLTools.main(CityGMLTools.java:87) [19:49:35 WARN] citygml-tools execution failed.

Thanks

clausnagel commented 2 years ago

Well, in order to write a CityJSON file, most of its contents must be held in main memory. In your case, this requires more main memory than available for the JVM. That's why you face a java.lang.OutOfMemoryError exception.

The question is whether it makes senso to create such a large CityJSON file. Target applications that you want to load the CityJSON with will most likely run into main memory issues as well. Thus, my recommendation is to split the huge CityGML file into smaller files and convert each of them separately.

If, however, you still want to create one huge CityJSON file, you need to give the citygml-tools process more main memory. To do so, you can either open the start script (citygml-tools for Linux, citygml-tools.bat for Windows) with a text editor, search for the line DEFAULT_JVM_OPTS="-Xms1G" and adapt the memory limit. I guess you will need at least 8GB or 10GB, so try to start with DEFAULT_JVM_OPTS="-Xms1G -Xmx8G".

If you do not want to fiddle around with the start scripts, you can also set the environment variable JAVA_OPTS to the value "-Xms1G -Xmx8G" before running the tool.

Saji70 commented 2 years ago

Thank you Claus for the response.

I set the JVM_OPTS up to 12 GB but the result was the same and it seems need more memory!

Due to your recommendation; How should I split the huge CityGML file into smaller files? Is it possible with citygml-tools or other open-sources?

clausnagel commented 2 years ago

citygml-tools does not offer a way to split CityGML files.

If you feel comfortable with Java programming, I recommend using the citygml4j library itself. It's super simple to read any CityGML file in chunks and to write the chunks to separate files.

I can also recommend using the 3D City Database. You can easily load your CityGML into this database and then export the data using different filters. The easiest way would be to export the data in tiles, which would automatically create smaller files. And the 3DCityDB export tool can already export the data as CityJSON. The 3DCityDB and the required tools are also available as Docker. A comprehensive user manual is available at: https://3dcitydb-docs.readthedocs.io/en/latest/

clausnagel commented 2 years ago

Closing due to lack of feedback. Feel free to re-open.