Generate GFS name earlier and remove old file if it exists

nlextract / NLExtract

Convert (ETL) and visualize free Dutch geo-datasets.

https://nlextract.nl

GNU General Public License v3.0

149 stars 84 forks source link

Generate GFS name earlier and remove old file if it exists #373

Closed fsteggink closed 4 months ago

fsteggink commented 5 months ago

Small PR. When processing a small DKK dataset it appears that the old GFS file is being reused when OGRInfo is being executed. Therefor the name of the GFS file should be determined earlier, and if this file already exists, it should be removed.

justb4 commented 4 months ago

I noticed that behaviour (old GFS reused) when testing the modified BRK GFS, but now I am puzzled with the use of GfsPreparationFilter:

AFAICS

only used in BGT and BRT Top10, 50, 100 ETL Stetl configs
thus not in BRK ETL config etl-brk.cfg

So this PR will have no effect on BRK ETL?

'Puzzled' in the sense: I was under the impression that with a well-prepared GFS file, no further preparation is needed. Or is this a performance issue? I know e.g. ogr2ogr generates a GFS file, and that it may not be complete if not all attribute occurrences are present in the data file,... or something else.

fsteggink commented 4 months ago

The GFSPreparationFilter was added in order to improve the loading speed of large datasets. I think it would be beneficial for the BRK as well, because it's a large dataset. However, since it's not used there, this PR won't have any impact on the BRK.

In this issue VNuhaan gave us hints how to improve the load speed with OGR2OGR, and this is the original PR.

justb4 commented 4 months ago

Ok, now I understand better, also the use of the Filter: to (XSLT-)render a GFS file with only the Features present in the dataset file and its feature counts. So I will merge this PR and open an issue to add the GFS Prep filter for BRK.