djhoese
commented
10 months ago From my experimenting with my ABI full disk -> "mercator subset" area yesterday, I have concluded that the reduce_data step (and specifically the AreaDefinition.get_area_slices step, which is specifically the polygon intersection calculations) takes about 10-12 seconds on my machine per area combination. How many unique areas does FCI have? I'm guessing at least 3 for the 3 resolutions involved here? So that's 3 source areas 3 target areas = 9 polygon intersection checks. Not to mention, I don't think the polygon creation is cached in any way so that is also repeated for each one of those. So 9 intersections 10-12 seconds = 90 to 108 seconds before we even get out of the .resample block of the code.
I mentioned yesterday, but I'd like to add JSON on-disk caching of the get_area_slices function while we wait for polygon operations to be made faster in pyresample.
That means the dask overhead penalty is disproportionately high in this case.
I disagree with the idea that this is "dask overhead". This isn't building the dask graph. This is Satpy and Pyresample wasting time doing things that:
- Take longer than we assume they should (bad algorithms written in python instead of C or other compiled language).
- Aren't cached.
gerritholl
Describe the bug
When saving multiple composites to multiple areas, there is a very large overhead before dask gets to work using all nodes.
To Reproduce
The following might seem convoluted, but is a pretty close approximation to what trollflow2 does in practice and therefore to what happens operationally. @pnuu may correct me if I'm wrong.
Expected behavior
I would expect a small overhead and then many dask workers working along nicely.
Actual results
The dask graph shows the clearest illustation of the problem:
Once dask gets to work, it's fine. But most of the time is overhead.
The full text log output is 391 KB and cannot be included here. Some key moments:
This reveals that it spends about 7 seconds reading the files a first time for the headers, about 8 seconds reading the variables from the files, then about 90+ seconds with "resampling" messages, which probably means building dask graphs, and another 3 seconds doing the same for the enhancements. It seems to build those resampling dask graphs for all composites/channels and all pixel qualities, even the ones we end up not using. That means the dask overhead penalty is disproportionately high in this case.
Maybe this should in part be a trollflow2 issue, as it's trollflow2 that's asking satpy to build dask graphs that it ends up not computing.
The final computation is quick.
Environment Info:
Not sure how to label what is essentially a performance problem.