Closed djhoese closed 9 months ago
Attention: 11 lines
in your changes are missing coverage. Please review.
Comparison is base (
6a8afc0
) 94.11% compared to head (b0a2579
) 94.13%.
Files | Patch % | Lines |
---|---|---|
pyresample/future/geometry/_subset.py | 90.00% | 8 Missing :warning: |
pyresample/_caching.py | 96.10% | 3 Missing :warning: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Timings using Satpy main
and this PR. Using gradient_search
to load, resample and save 10 composites (some Day/Night, some normal) for FCI L1c to 3868 x 3918 EPSG:3035 area.
reduce_data=True
:
As a comparison, with reduce_data=False
it takes ~59.7 s (C) to run the same script.
Dask graphs:
A
B
C
Thanks @pnuu , this looks good! So I'm merging this.
Thanks for testing @pnuu!
While profiling some Satpy computations (ABI full disk -> nearest neighbor resampling) I noticed that a decent amount of time at the beginning of processing was spent outside of dask computations and was using a single core. After some print-statement debugging I discovered it was Satpy's
reduce_data
functionality and theAreaDefinition.get_area_slices
that was taking the most time. The majority of that time is spent in the polygon intersection operation to see if the two areas being used intersect and where.This PR adds a decorator and a couple configuration settings for caching the results of
AreaDefinition.get_area_slices
to on-disk JSON files. For my testing case I was seeing about ~10-12 seconds being used forget_area_slices
per area definition pair. I was using 2 resolutions of ABI data and one target area so that was ~22 seconds. With this caching enabled that time basically disappears.This PR is only a proof of concept at this point and I will continue to improve it. I just wanted to get the initial commits up on github for others to see.
git diff origin/main **/*py | flake8 --diff