Closed pkgw closed 2 years ago
Merging #89 (f58ef47) into master (cd662ec) will increase coverage by
0.82%
. The diff coverage is93.69%
.
@@ Coverage Diff @@
## master #89 +/- ##
==========================================
+ Coverage 75.61% 76.43% +0.82%
==========================================
Files 24 24
Lines 3809 3981 +172
==========================================
+ Hits 2880 3043 +163
- Misses 929 938 +9
Impacted Files | Coverage Δ | |
---|---|---|
toasty/pyramid.py | 93.07% <92.47%> (-1.58%) |
:arrow_down: |
toasty/merge.py | 100.00% <100.00%> (+7.50%) |
:arrow_up: |
toasty/samplers.py | 87.57% <100.00%> (ø) |
|
toasty/toast.py | 93.63% <100.00%> (+0.02%) |
:arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
Docs build still failing just because dotnetfoundation.org
has an expired HTTPS cert.
Here we add a new
toasty.pyramid.Pyramid
class for better managing operations on tile pyramids. I've worked out some formalisms and ways to talk about the pyramid that I think will simplify a lot of things going forward. In this particular PR, we convert the "merge" (image cascade) functionality to use the new approach.There is new functionality here in that we can now process "subpyramids", where we can do a reduction cascade over only a piece of a larger pyramid. This was necessary for DECaPS2 processing, where the full level=14 TOAST pyramid was going to take too long to process within a single process, even with a lot of CPUs. This is a slight generalization of our existing "tile filter" approach, but the complexity of implementing it within the earlier framework motivated me work out this new system.
One of the key improvements is the "reduction iterator" system for thinking about a depth-first pyramid walk as a reduction operation. The key thing is that it's not easy to parallelize reductions, so we can (should) only use the new iterator for relatively cheap computations. Image cascading must be done with the "walk" framework, which offers the possibility of parallelization but cannot "reduce". The basic problem with parallelizing reductions is transmitting data between processes, which is definitely solvable, but not something I want to deal with right now.
We use the new reduction iterator to speed up the initialization of parallelized, filtered TOAST cascades, because the previous implementation was basically walking the tree several times to compute all of the stuff that it needed to have to manage the parallelized cascade.
A nice extension would be to add a
visit_leaves()
method and port a bunch of our other operations over to use it. Should be pretty easy to do that now that the framework is in place.