Error message when converting .cityjson file back to citygml using Windows command line

[Brabanter]

I am trying to convert a .json file back to city gml so that I can upload it into the Autodesk suite (Infraworks). I am using data downloaded from the Dutch national 3D files https://www.pdok.nl/3d-basisvoorziening

Unfortunately it doesn't work and I receive the following error message:

[10:47:23 ERROR] The following unexpected error occurred during execution.java.lang.OutOfMemoryError: GC overhead limit exceeded

I understand that Java throws this error when most computing time is spent doing garbage collection but am not quite sure how to solve the issue.

Please let me know what further information you need, if any. And thanks in advance.

[clausnagel]

Hi @Brabanter, citygml-tools performs an in-memory conversion from CityJSON to CityGML. Your CityJSON input file is simply too large and the Java process runs out of memory.

My suggestion is to use cjio to split your large CityJSON file into smaller ones. cjio offers a subset command for this purpose. It's also available on Docker Hub. Alternatively, you could try and load the dataset into the 3DCityDB and export it as CityGML.

I will check whether I can improve citygml-tools to better handle large CityJSON files. This will require some work and time though, so no quick solution. I know that the CityJSON folks are working on better supporting a chunk-wise processing of CityJSON data. I think this is a good example why it is required.

[clausnagel]

I optimized the from-cityjson command a bit (044e4bf971a3ccb64a7aedd9729a254bf154fca0). It must still read the CityJSON file completely into memory. But unlike before, converting to CityGML and writing to disk does not use significantly more memory anymore. To benefit from this optimization, clone the master branch and build the tool from source following this guide.

Processing the 1 GB 44an1.json file with the updated citygml-tools requires 7 GB of memory. To avoid the java.lang.OutOfMemoryError exception, you need to adapt the amount of memory that citygml-tools is allowed to allocate using the Java parameter -Xmx.

There are two possibilities:

1. Adapt the citygml-tools start script. Under Windows, open citygml-tools.bat with your favourite text editor and replace the line setting the variable DEFAULT_JVM_OPTS with something like:

   set DEFAULT_JVM_OPTS="-Xmx7G"

2. Set the CITYGML_TOOLS_OPTS environment variable before executing citygml-tools. For Windows:

   > set CITYGML_TOOLS_OPTS="-Xmx7G"
   > citygml-tools from-cityjson 44an1.json

   You may also use JAVA_OPTS instead.

For this to work, your machine must of course have more than 7 GB of memory and must support the allocation of 7 GB to a Java process. On my Windows machine it worked fine, and I was able to convert the 44an1.json file to CityGML. Be careful, the process takes some time and the output file size will be 12 GB. I am curious whether you can load this file with Infraworks. Let me know :-) You can at least easily load it into the 3DCityDB and export smaller files using a bbox filter.

After all, 7 GB memory for loading a 1 GB file is obviously quite a lot. I'm using Gson for parsing CityJSON with a pretty simple Java object mapping. Most of the memory is consumed by the CityJSON vertices array. I suspect that loading it into a file-based cache instead would make a big difference, but might also slow down performance. I'll take a look at it later.

[Brabanter]

Hi Claus Thank you for altering the code and for your detailed instructions.

I have now successfully generated an enormous citygml file using the citygml-tool. I did this on a remote server with a 32 gb memory. The next challenge will be to try to upload it into Infraworks. I will be happy to keep you posted on this.

Depending on my other commitments, I would like to explore the cjio tool in a windows environment. Am I correct in believing that the citygml database would work best in a Linux environment? I've done a quick google on it but would also appreciate more information (or links to information) on how to get a bbox filter to work.

[clausnagel]

Great to hear the conversion worked for you.

The 3DCityDB is a relational schema for managing CityGML data and can be setup on both PostgreSQL/PostGIS and Oracle. So, you need to install one of two database systems. They both run either on Windows or Linux. In addition, there is a small Java-based tool called Importer/Exporter that lets you import and export data from the 3DCityDB. The Importer/Exporter can be installed on your desktop computer.

I recommend to download the latest version of the 3DCityDB Importer/Exporter from here. And then simply follow all the installation steps as described in this online documentation.

This documentation also contains a chapter on how to use the Importer/Exporter tool. Here you will find all the details on how to export content from the 3DCityDB using a bbox filter.

The Importer/Exporter is able to handle large CityGML files. So, you can directly load the enormous CityGML files from your conversions into the database. Have fun :-)

[Brabanter]

Hi Claus Am still on the trail. Here is what the latest version of Infraworks can upload

and here is the message your receive when you try to do so.

I am presuming that the CityGML importer indexes the CityGML files in a similar manner to Recap, a program that prepares (indexes) pointcloud files so that they can be imported into Autocad.

[Brabanter]

Hi Claus How we got the sample Frankfurt citygml data set to look in Infraworks. The first few images had the co ordinate system set to RD new (so the Netherlands) but when we tried 3 degree Gauss zone 3 we ended up with the third image. The fourth image is what it should look like. We have assumed that the dataset is back to front/inside out because of the co ordinate system. It may, however, have occurred in the conversion process (citygml=>imx) or when loading the data into Infraworks.

[Brabanter]

Four images of the same terrain using/incorporating the 3D foundation file ( basisbestand )

1. In Qgis
2. In Infraworks with a pointcloud and aerial photo superimposed
3. The foundation file itself
4. Unfortunately roads and buildings have been filtered out by the program.

I needed to use both cjio and the citygml tools in addition to a virtual machine with a 32 gb memory to get this far. And for anyone who may be following this discussion, Infraworks prefers RD to RD new :)