search

yeesian / ArchGDAL.jl

A high level API for GDAL - Geospatial Data Abstraction Library
https://yeesian.github.io/ArchGDAL.jl/stable/
Other
141 stars 27 forks source link

How to multithreaded read/write? #293

Closed bjoern234 closed 2 years ago

bjoern234 commented 2 years ago

Hey, do you have an example how to read or write multithreaded? I've set flags=ArchGDAL.OF_SHARED in ArchGDAL.read or ArchGDAL.readraster and then looped over windows or blocks using Threads.@threads. But the threaded loop needs twice as much time as a simple loop.

Thanks! Bjoern

visr commented 2 years ago

I haven't tried this before, but wouldn't IO be the bottleneck for such an operation, unless you do a very expensive computation on each window? If IO is the bottleneck, I can image it being slower, especially if it is from a hard drive.

bjoern234 commented 2 years ago

Yes, I want to apply time consuming computations on each block. Maybe I did not asked correctly. What I would like to know if is the way I did it correctly? Because sometimes I got strange errors like 

    nested task error: GDALError (CE_Failure, code 1):
        TIFFReadEncodedStrip:Seek error at scanline 4294967295, strip 7613

which I don't get if I run a simple loop on only one thread.

visr commented 2 years ago

I understand. I don't know if that is the correct way to do it. If you have a reproducable example of this issue maybe somebody can have a look at it. Does it happen with all tiff files, or only some?

For reference here is the GDAL_OF_SHARED docs:

Shared mode: GDAL_OF_SHARED. If set, it allows the sharing of GDALDataset handles for a dataset with other callers that have set GDAL_OF_SHARED. In particular, GDALOpenEx() will first consult its list of currently open and shared GDALDataset's, and if the GetDescription() name for one exactly matches the pszFilename passed to GDALOpenEx() it will be referenced and returned, if GDALOpenEx() is called from the same thread.

yeesian commented 2 years ago

To my understanding, the same dataset should not be accessed by two or more threads at the same time; see https://trac.osgeo.org/gdal/wiki/FAQMiscellaneous and https://rasterio.groups.io/g/main/topic/read_using_multithreading/73348785.

You might have to find other approaches for concurrent processing.

On Thu, Apr 21, 2022, 8:52 AM Martijn Visser @.***> wrote:

I understand. I don't know if that is the correct way to do it. If you have a reproducable example of this issue maybe somebody can have a look at it. Does it happen with all tiff files, or only some?

For reference here is the GDAL_OF_SHARED docs:

Shared mode: GDAL_OF_SHARED. If set, it allows the sharing of GDALDataset handles for a dataset with other callers that have set GDAL_OF_SHARED. In particular, GDALOpenEx() will first consult its list of currently open and shared GDALDataset's, and if the GetDescription() name for one exactly matches the pszFilename passed to GDALOpenEx() it will be referenced and returned, if GDALOpenEx() is called from the same thread.

— Reply to this email directly, view it on GitHub https://github.com/yeesian/ArchGDAL.jl/issues/293#issuecomment-1105169929, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJMRZ7ZAENCQ5SOT4IWVDDVGFFRXANCNFSM5T37K34A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

visr commented 2 years ago

Ah yes I read that FAQ too but wondered if it is still up to date, since I cannot find it on gdal.org. But it seems that statement does not contradict the GDAL_OF_SHARED explanation, which only means that the handle itself can be shared, not neccesarily accessed at the same time.

rafaqz commented 2 years ago

Yes, parallel read is unlikely to improve anything. What will work better is loading one file or section of it to memory for all threads, and then using a threaded loop over it or LoopVectorization.jl, or some JuliaFolds package, depending what you are doing.

bjoern234 commented 2 years ago

Thanks for the clarifications. @visr I've tried different files which all produces the same errors. @rafaqz yes, that's the normal way I do this, but in some cases (e.g. large raster stacks) the data fits not into memory. It seems that I have to open a dataset for each worker separately in that case.

rafaqz commented 2 years ago

If your workers are threads in a single cpu, its probably still better to load memory sized chunks of the data and use all threads on the same chunk. You can get chunk sizes for RasterDataset using the DiskArrays.jl api.

We should work out how to integrate DiskArrays.jk with threaded loops so that just works automatically.