search

evetion / GeoDataFrames.jl

Simple geographical vector interaction built on top of ArchGDAL
https://www.evetion.nl/GeoDataFrames.jl/dev/
MIT License
67 stars 6 forks source link

Make write 30 times faster #69

Closed maxfreu closed 2 months ago

maxfreu commented 3 months ago

I just copied the ArchGDAL code for chunked writes here. Makes writing sqlite files 30 times faster for me :)

evetion commented 3 months ago

That's great stuff, thanks! FWIW, do we need these config options, as in, can we make it just the default?

maxfreu commented 3 months ago

We can make it the default, but if I recall correctly, there were issues with some vector drivers and some corner cases. That's why this flag exists. But of course we can set the default to false and just say set it to true if you run into troubles.

evetion commented 2 months ago

I think I will adopt chunking to be default for the applicable drivers. You could disable that by setting the chunksize to zero? That makes more sense to me than several kwargs?

maxfreu commented 2 months ago

I think you mean setting the chunksize to one. But that could still result in different behaviour in corner cases, because then we don't use GDAL's gdalcreatecopy. However, we might dare to set chunking to be the default. I wrote some extensive tests here. So it should work at least for the drivers listed at the top of that file.