north-road / qgis-processing-saga-nextgen

GNU General Public License v3.0
21 stars 11 forks source link

Natural-Neighbour and Shapefile size limit #41

Open cpWWS opened 5 months ago

cpWWS commented 5 months ago

I am trying to run the natural-neighbour module in Saga 9.2 and ran into a block, which I assume is due to the size of the shapefile limit:

input is a set of points in gpkg format (about 200 million points) output desired is a grid of 12000x12000 pixels.

The program (natural neighbour - same for thin plate) first saves the gpkg file into a shapefile into a temp directory. I note that the program seems to proceed well but stays put at 100% complete and nothing seems to happen, even though the CPU and memory continue to operate - no error message is indicated in the log.

I tried to conver the gpkg file to a shapefile, but QGIS gives an error as the shapefile exceeds 4Gb.

Is there a solution? I wish Natural Neighbour could take multiple input point files, but that does not seem to be possible. Perhaps I need to break up the problem into smaller grids and join them later... but that is not ideal (edges will need to be smoothed etc).

Perhaps eventually the gridding routines could be improved to handle such large datasets.

Any suggestion welcome.

baswein commented 5 months ago

Adding this reference to an old QGIS issue with some details for reference. https://github.com/qgis/QGIS/issues/47100

cpWWS commented 5 months ago

Thanks @baswein for the comment. The reference mentioned does not really address the issue. In my case the vector file has only one field, with a short name (ELEV). The problem seems to be simply that the conversion from gpkg to esri shapefile does not work in my case since the vector file has more than 200 million points and is a large file (over 15 Gb).

I propose that the simpler solution would be to enable the gridding programs (thin plate, natural neighbour, etc) to receive a file list as input, this way I can prepare several smaller point files (in gpkg or shp format) that would fit the size limits of shapefiles.