Closed dabhicusp closed 8 months ago
Hello @alxmrs I tried with just making the requests in a loop.
but it takes much more time(~21x) as compare to threadpool
for the calculating in the image which user provided.
Code:
tiles_threadpool = [None for _ in range(total_tile)]
tiles_forloop = [None for _ in range(total_tile)]
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor() as pool:
for i, arr in pool.map(
self._get_tile_from_EE, list(zip(data, cycle([coordinate_type])))
):
tiles_threadpool[i] = (
arr.tolist() if coordinate_type == 'longitude' else arr.tolist()[0]
)
print("threadpool_time: ", coordinate_type,time.time() - start_time)
start_time = time.time()
for i in range(total_tile):
_, res = self._get_tile_from_EE((data[i], coordinate_type))
tiles_forloop[i] = (res.tolist() if coordinate_type == 'longitude' else res.tolist()[0])
print("for_loop_time : ", coordinate_type, time.time() - start_time)
Image which tested:
ee_class = EarthEngineBackendEntrypoint()
geom = ee.Geometry.Rectangle([-61, -7, -50, -18])
county = ee.Feature(geom)
image = ee.Image('projects/mapbiomas-workspace/TRANSVERSAIS/ZONACOSTEIRA6/mosaic_1985').clip(county)
coll = ee.ImageCollection(image)
ds = ee_class.open_dataset(coll.select(['NDVI']), crs='EPSG:3857',
scale=30, geometry=geom,
io_chunks={'time':24,'X': 512, 'Y': 512}
)
Result:
threadpool_time: longitude 4.9227306842803955
for_loop_time : longitude 95.86925983428955
threadpool_time: latitude 4.389440298080444
for_loop_time : latitude 74.4054172039032
Also tested on other images too and got the same result as above(taken ~21X more times).
Fixed: #85.
The current code gives an error(
Pixel grid dimensions (40818x1) must be less than or equal to 32768
) when creating an lat, lon array for theimage_to_array
conversion, for certain lengthy images. To address this, I've made updates to the code. Now, when generating an lat, lon array for certain lengthy images, I break it down into smaller array based on Users preferred size or default size. This adjustment prevents the above error.