Open gerritholl opened 10 months ago
From my experimenting with my ABI full disk -> "mercator subset" area yesterday, I have concluded that the reduce_data
step (and specifically the AreaDefinition.get_area_slices
step, which is specifically the polygon intersection calculations) takes about 10-12 seconds on my machine per area combination. How many unique areas does FCI have? I'm guessing at least 3 for the 3 resolutions involved here? So that's 3 source areas 3 target areas = 9 polygon intersection checks. Not to mention, I don't think the polygon creation is cached in any way so that is also repeated for each one of those. So 9 intersections 10-12 seconds = 90 to 108 seconds before we even get out of the .resample
block of the code.
I mentioned yesterday, but I'd like to add JSON on-disk caching of the get_area_slices
function while we wait for polygon operations to be made faster in pyresample.
That means the dask overhead penalty is disproportionately high in this case.
I disagree with the idea that this is "dask overhead". This isn't building the dask graph. This is Satpy and Pyresample wasting time doing things that:
I didn't use any HRFI data in this test, so the source data is just 2 areas. If I understand you correctly, that means the overhead will get even more once I include HRFI (which includes two channels at 500 metre resolution, unlike FDHSI, which has only 1 km and 2 km resolutions).
Sorry, I thought it was all dask overhead, but I see it's "our fault" in this case.
You'd have to add prints inside Scene._resample_scene around the reduce data logic to see how much time it's spending there and if it seems reasonable. I also assume the cached resampling was pre-cached in a previous execution.
I profiled it with cProfile
and found that it spends 160.4 seconds in _resampled_scene
(3 calls), 133 seconds in _reduce_data
(66 calls), including 132 seconds in intersection
(141792 calls). By comparison, compute_writer_results
is only 37 seconds:
Yes, this uses cached resampling.
If I run with reduce_data=False
, the overall processing is much faster, despite compute_writer_results
increasing from 37s to 54s. Bokeh plot:
cProfile result:
Do your target areas cover a majority of the input data? Let's say more than 75%? If so then I'd expect similar processing time. My case was an ABI case where I think I had cut off quite a bit of the full disk input data and when I did reduce_data=False the processing took much longer. I wasn't seeing so much of a slow down from resampling/reduce_data though. Let me look closer at this cprofile output...
Oh wow, your per-call for the get_area_slices
is almost twice as long as mine was (22s versus my 10-12s). That would explain the large dead zone in your original plot for only 2 source areas at 3 targets: 6 * ~22s => 132s.
Do your target areas cover a majority of the input data? Let's say more than 75%?
No, not in this case, but some of them do in the operational case (which takes much longer to run).
@gerritholl with pyresample main
you should be able to use:
https://pyresample.readthedocs.io/en/latest/howtos/configuration.html#cache-geometry-slices
This should almost completely remove the slow down of reduce_data
.
Describe the bug
When saving multiple composites to multiple areas, there is a very large overhead before dask gets to work using all nodes.
To Reproduce
The following might seem convoluted, but is a pretty close approximation to what trollflow2 does in practice and therefore to what happens operationally. @pnuu may correct me if I'm wrong.
Expected behavior
I would expect a small overhead and then many dask workers working along nicely.
Actual results
The dask graph shows the clearest illustation of the problem:
Once dask gets to work, it's fine. But most of the time is overhead.
The full text log output is 391 KB and cannot be included here. Some key moments:
This reveals that it spends about 7 seconds reading the files a first time for the headers, about 8 seconds reading the variables from the files, then about 90+ seconds with "resampling" messages, which probably means building dask graphs, and another 3 seconds doing the same for the enhancements. It seems to build those resampling dask graphs for all composites/channels and all pixel qualities, even the ones we end up not using. That means the dask overhead penalty is disproportionately high in this case.
Maybe this should in part be a trollflow2 issue, as it's trollflow2 that's asking satpy to build dask graphs that it ends up not computing.
The final computation is quick.
Environment Info:
Not sure how to label what is essentially a performance problem.