Import a subset with a polygon/bbox filter

[raymondnijssen]

Cityjson files can be very large and hit the max RAM during the import. It would be nice to select your desired area by drawing a polygon or bounding box on the screen before starting the import. Maybe even showing the cityjson data extent on the map?

[liberostelios]

This is a great idea, actually. You could use cjio to do this through command line, but we have been thinking for a while to incorporate cjio's functionality in QGIS toolbox. I guess this is a good moment to do so (based on the huge tiles of 3D topografie).

[rduivenvoorde]

Yes, PLEASE :-)

See for example the QGIS related comments here: https://tweakers.net/nieuws/169816/het-kadaster-stelt-3d-bestand-van-nederland-beschikbaar-als-open-data.html People really have issues with current PDOK datasets

[liberostelios]

I've uploaded a new version that supports filter by extent and object type. Turns out, for the specific files the biggest issue is the terrain. So, filtering by "Building" works in QGIS.

Keep in mind, this doesn't work through the original window, but you'll have to run the CityJSON loader's algorithm in the Processing Toolbox, which is a bit unstable for now.

The new version should get approved in a few hours so you can get it through QGIS. Otherwise, you can install it from the link above.

[rduivenvoorde]

@liberostelios note that if you download the fixes by zip from here, you need the 'develop' branch?

I'm now trying to extract my house from 25za1.json (only buildings):

how much time should I count on? Which dataset did you use? Swap is filling up again :-) But CPU is doing nothing (probably OS is moving stuff?):

[liberostelios]

Indeed, I haven't pushed the changes to main. But you should be able to download the zip from the release, no need to download all source code.

I tried with the 25hz1 file. Loading took roughly about 10'-15', I think. I guess at that point the biggest bottleneck is your disk IO and not CPU, so not having 100% CPU consumption should be normal. I'll try with your dataset as well to make sure it work.

[rduivenvoorde]

Yes, it would be supercool, both for QGIS and PDOK if we could make it work in a way to make these datasets usable for QGIS users.

I see (with another small demo cityjson of DenHaag), that the layers will be loaded in a Memory layer EPSG:7415 (being Amersfoort+RDNAP?).

What would be a good (3d-enabled) dataformat to save it in a way you can more easily distribute parts of the large files?

[rduivenvoorde]

After about 10 minutes: I think it would be good to give some more feedback, so the user get's the idea that you actually make progress... same for cjio?

After 15 minutes:

Yess:

/me wondering: would it be more efficient to do the extent filtering earlier in the proces? So not all features would be loaded first?

[raymondnijssen]

It got "stuck" here for hours, but things seems to keep on running. There has been a jump from 45% to 48% Should be finished in 192 minutes. I'm trying to figure out how to optimize the parsing.

[liberostelios]

What would be a good (3d-enabled) dataformat to save it in a way you can more easily distribute parts of the large files?

@rduivenvoorde In the past I've used geojson to store 3D data (using ogr2ogr from a PostGIS database) and it worked fine in QGIS (although, technically, geojson is not a 3D format). I know there is a GSOC project for several QGIS 3D improvements, involving an export to obj (a format that we use often for visualisation). Not sure how that exporter will work, though.

I think it would be good to give some more feedback, so the user get's the idea that you actually make progress... same for cjio?

That could be done, but not for the "Loading city model" part. This is plain JSON parsing by Python, actually, and there is nothing we can do to monitor its progress. Maybe we could add a progress bar for the filtering process.

/me wondering: would it be more efficient to do the extent filtering earlier in the proces? So not all features would be loaded first?

That's impossible, because the only step before that is simply to parse the file itself, as I said before. So, filter is done immediately after the file has been loaded.

I am looking into ways to improve the parsing of JSON by using another library, but that's a bit tricky because adding Python libraries that are not part of the original QGIS installation is tricky.

It got "stuck" here for hours, but things seems to keep on running. There has been a jump from 45% to 48% Should be finished in 192 minutes. I'm trying to figure out how to optimize the parsing.

@raymondnijssen As I mentioned before, the part of file loading could be improved, but I am not sure what could be done for the actually part that you are showing here (the transformation of objects).

Your file seems to have relatively few objects, though (about 3.7k objects). It's weird that it takes so much. Is your computer that slow or are there complicated geometries in there? It could be that terrain slows down the process. Maybe you could consider loading only buildings to check?

[raymondnijssen]

@liberostelios My laptop is old but not super slow, (Intel® Core™ i7-4500U CPU @ 1.80GHz × 4 ) with 8gb ram. Should not take this long. I did try buildings only and it was fast. But it resulted in a layer not showing any polygons. After changing it to 28992 though, they are showing up. But by then I already started a new import with every object. I will check it tomorrow morning :)

[raymondnijssen]

Loading city model... Loaded 3724 objects. No CRS selected! Looking for CRS definition in metadata... CRS found: 7415. Transforming city objects... Execution completed in 43609.59 seconds Results: {'STATUS': 'SUCCESS'}

\o/

[liberostelios]

That's great! This is still extremely slow, but at least it worked. Which file is this? Is it a whole tile?

[rduivenvoorde]

Mine was indeed a whole tile (25az1.json == haarlem) but I did a bbox selection in it. @raymondnijssen I think he took one of the northern tile with mostly water :-)

I tried to do a whole tile with all objects in but that fails.

Only selecting one (Buildings) succeeds.

Would be great though if you could select several objects (for example the all but the terrain?), OR have a checkbox to NOT load the terrain (as that is apparently the big deal)?

[liberostelios]

Yes, that's a great idea! It's actually part of the plan. I think I'll implement that soon.

[liberostelios]

@rduivenvoorde Version 0.6.2 is out with the option to select multiple object types.