geospatial-jeff / aiocogeo

Asynchronous cogeotiff reader
MIT License
73 stars 10 forks source link

parallelize tag reads #88

Closed geospatial-jeff closed 3 years ago

geospatial-jeff commented 4 years ago

With a slightly smarter reader we could easily read all tags for an ifd in parallel.

https://github.com/geospatial-jeff/aiocogeo/blob/master/aiocogeo/ifd.py#L42-L45

kylebarron commented 4 years ago

Amounts to basically removing that await?

geospatial-jeff commented 3 years ago

Kind of. It's a little more difficult mostly because the offset which keeps track of where we are in the file is incremented inside the call to .read(). The same file reader instance is shared across each coroutine (in this case each Tag we are trying to read) so we can't try to read all tags at once because the offset will get messed up, especially as we seek into other parts of the file to search for large tag values.

The coupling of the http request (the thing we want to await) and the offset increment isn't a great design pattern for this reason, and something I intend to replace. Although it made initial development of the library much easier.

geospatial-jeff commented 3 years ago

Scratch that, the requests which reach to other parts of the file to fetch tag values don't call .read() so this is easy to implement.

https://github.com/geospatial-jeff/aiocogeo/blob/master/aiocogeo/tag.py#L86