pgeleg
commented
11 months ago Thanks for this. Will look more into it.
There is a lot of performance left on the table as it stands. There is no multithreading at all at the moment, which both fetch_tiles and build_vrt can benefit from.
I did a quick crude implementation of multithreading for fetch_tiles which would be I/O bound. Fetching all 3602 currently available tiles, my personal machine went from 45 minutes to 18 minutes with that implementation.
I think there would be significant performance gains for build_vrt as well. A lot of CPU is left on the table.
aaime
GlenRice-NOAA
The generation of the top level UTM VRTs is taking hours (at least on my hardware). I'm wondering if there's anything that can be done to speed it up.
On one side, I see there is a large histogram in all UTM file, taking some 20% of the file size, mostly filled with zeroes. I'd guess that in order to compute it, GDAL has to read all actual source pixels (the other VRTs do not seem to have it). Is the histogram actually important? Can it be skipped?
The unified RAT of course takes time, but it's kind of required, and the algorithm to build it appears to be efficient already (single pass with a lookup containing the already encountered rows, if I read correctly). Although, I don't see the RATs in the intermediate VRT files, which could maybe help to avoid opening all the source files once more (not sure, just guessing).
The last bit is the overviews, which also feels quite slow, I can see the script process as the ovr file grows. I've done a bit of debugging:
1) Using CPL_DEBUG=on, I've verified that the process is indeed opening every single source TIF file, e.g.:
2) Using pyrasite-shell I've verified that while that happens, the process is doing this:
So it looks like the culprit is indeed overview creation... however, the other VRTs also have overviews, I would have expected the "add overviews" process to just build on the overviews of the single block VRT file... and yet, it does not seem to be happening, which is a pity. For reference, a "normal" gdaladdo builds the first overview from the actual pixels, but the second overview is build using the first overview pixels, and so on, speeding up the process dramatically.