opendatacube / datacube-wps

Web Processing Service running on opendatacube

Other

7 stars 3 forks source link

Dask pixel drill #62

Closed uchchwhash closed 3 years ago

uchchwhash commented 3 years ago

Deprecate @Kirill888's (wonderful) hand-crafted pixel-drilling code in favor of datacube load with unsigned s3 access.

Load pixels using Dask with workers initialized to use configure_s3_access with AWS no sign request
Remove old code

The performance is a bit slower, especially for all time queries (~1.5x). But on the flip side, de-duplicated pixels!

Edit: With @Kirill888's suggestion of 1 worker N threads implemented, the performance is now on par :smile: .

codecov[bot] commented 3 years ago

Codecov Report

Merging #62 (30351b6) into master (e0ecef5) will increase coverage by 0.53%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #62      +/-   ##
==========================================
+ Coverage   73.72%   74.26%   +0.53%     
==========================================
  Files           4        4              
  Lines         373      373              
==========================================
+ Hits          275      277       +2     
+ Misses         98       96       -2

Impacted Files	Coverage Δ
datacube_wps/processes/__init__.py	`61.72% <100.00%> (+0.82%)`	:arrow_up:

Kirill888 commented 3 years ago

I just want to point out that reading a 1x1 image, possibly with reprojection, is not the same as pixel drill that I have prototyped ages ago. Previous pixel drill read one native pixel at the location specified by the user, no reprojection was involved. I guess one can claim that with nearest resampling the two should be equivalent, but reprojection is still going to be costlier and more prone to failure.