search

boku-ilen / geodot-plugin

Godot plugin for loading geospatial data
GNU General Public License v3.0
109 stars 19 forks source link

Loading from higher resolution geoimages / extracting larger-scale textures is slower #15

Closed kb173 closed 4 years ago

kb173 commented 4 years ago

With nearest neighbor sampling, the scale of the desired image / the resolution (pixels / meter) of the raw geodata should have close to no effect on performance because only single points are sampled either way. Only the resolution of the output texture should have an effect.

For some reason though, the resolution / scale does have a pretty significant impact that is especially noticeable with very high-resolution images and large-scale (overview) extraction. It seems to stem from the warp operation. We should investigate whether additional samples or a different algorithm is used somewhere there. Perhaps #4 could come into play here too!

kb173 commented 4 years ago

Seems like I misread the logs there: It doesn't only happen in the warp operation, but also before it. The time this takes could be related to reading the metadata of the dataset or something else that happens when first opening it. We could cache it, it's not a problem if it only happens the first time. Maybe not though, since a larger size in meters of the request also increases this waiting time... Gotta figure out more exactly where it gets stuck.

Weird - the warp operation is only slow with a larger output pixel size (which is expected), but the amount it gets slower seems to be different depending on the raw data. It's also not consistently slow but it gets stuck - seemingly after each quarter of warping. Funnily, an image size of 3x3 gets slow after each third of warping, whereas a 2x2 image warps instantly even when requesting 5x5km.

I might want to test this with a different hard drive, that could be the reason too... or maybe it depends on how well the file system handles large files like that? (The TIF I'm testing with is over 100GB large)

kb173 commented 4 years ago

No, GDALOpen is not the problem - It seems like it gets stuck before the warp, but apparently it's just stuck at 0% of the warp.

I guess this makes it somewhat easier since the warp is the only problem.

kb173 commented 4 years ago

ChunkAndWarpMulti is much faster than ChunkAndWarpImage. Makes sense that it's somewhat faster, but it doesn't exhibit the stuck behavior at all at scales where ChunkAndWarpImage does. However, it does also start getting stuck at similar points (quarters of progress) starting at a certain scale/resolution.

In order to handle images too large to hold in RAM, the warper needs to segment large images. This is the responsibility of the GDALWarpOperation class.

This could be the reason... but why is it loading everything into RAM, not just the required parts?

kb173 commented 4 years ago

Ok, to some extent it's definitely a hard drive thing because loading it the first time takes long, and each time the same raster is requested afterwards is much faster.

I tried fixing #4 and this solves this issue because we can get rid of the whole warping stuff. It needs to be cleaned up though because something is not quite right with the coordinate transformation.

kb173 commented 4 years ago

4 does indeed fix this! https://github.com/boku-ilen/geodot-plugin/commit/bb065dbd609e91226bf540c022f9fdef76bd8878 is a proof of concept where this issue is gone, but not everything is reimplemented without GDALWarp yet. Further discussion will be in #4.