Open rafaqz opened 8 months ago
I largely support the idea. However, I think this raises a few questions:
DimStack
return the dimensional values, as it does with a DimTable
?DimStacks
going to be stored as an array of NamedTuples
, or is this simply a type-level abstraction as in StructArrays.jl? I'm concerned that the former would be highly inefficient for many operations.NamedTuple
has extremely long compile times for DimStacks
with many layers. If we choose to represent a standard stack as an AbstractArray{<:NamedTuple}
, perhaps we could represent a wide stack as an AbstractArray{<:AbstractDict}
. Indexing into a stack would then be analogous to a Tables.rowtable
or Tables.dictrowtable
, respectively.RasterStack
representing a satellite image and a Raster
containing a DEM layer and take their intersection to produce a new RasterStack
with an extent equal to the area covered by both data sources. Similarly, a union operation could be used to combine two satellite images covering different geographic regions. One problem is that NamedTuples
are immutable, so adding new layers on the fly is difficult. Is there any way we could support dynamically adding layers to an existing stack?broadcast
operate over NamedTuples
or their individual fields? I feel like the former would be tedious for many operations, like scaling all values by some constant. Also, what would happen if the broadcast function returns a NamedTuple
with different fields than the input? Would we get a new DimStack
with the same layers as the names in the output NamedTuple
? So broadcasting the function f(x) = (sum = x.a + x.b, product = x.a * x.b)
would take a stack with the layers a
and b
and return a new stack with layers sum
and product
?getindex
. We would never switch completely to Dict (or Dictionary) as it's much slower for small stack use cases.crop
or extend
. If we had a mode switch to Dict we could do in-place adding layers. But what are the issues with using merge
to make a new stack (besides wide stacks...)?AbstractDimArray
of your return values. You could do things like:A = (x -> x.a * x.b).(st) .* C
Which is different to
A = st.a .* st.b .* C
Because it will handle mixed-size layers. But yes its a little clunky accessing nested values in broadcasts.
Probably a Dictionaries.jl Dictionary
would have nicer properties as a mode switch from NamedTuple.
See: https://github.com/andyferris/Dictionaries.jl/pull/43
We really need getproperty
on dictionary keys for this to work seamlessly.
For InferenceObjects.jl, we for the most part ignore that getindex
returns a NamedTuple
and primarily use DimStack
as a NamedTuple
of DimArray
s with the extra check that Dim
s are consistent. The only exception is the Tables interface implementation. So I don't have a strong feeling either way, as I don't think it will affect my users much. My only concern is whether making an AbstractDimStack
an AbstractArray
might cause annoying defaults to be hit in other packages or method ambiguities.
- This is mostly already how it works! NamedTuple returned from indexing is not a new idea Im adding. But in Julia 1.11 NamedTuples should compile a lot faster and reduce your wide stack problem. If we still want Dicts we need a new stack mode that we can switch to. We could just change the new AbstractArray T in AbstractDimArray{T} from NamedTuble to Dict (with matching internals) and dispatch on that in
getindex
. We would never switch completely to Dict (or Dictionary) as it's much slower for small stack use cases.
It would be a huge relief if 1.11 can address the compilation issue with NamedTuples
, as it's been one of my major performance gripes. I wasn't suggesting that we replace NamedTuple
with a Dictionary
entirely, only that we make it possible to substitute one for the other in order to accommodate wide stacks. However, perhaps Julia 1.11 will solve this issue?
- Broadcast would return an
AbstractDimArray
of your return values. You could do things like:A = (x -> x.a * x.b).(st) .* C
Which is different to
A = st.a .* st.b .* C
Because it will handle mixed-size layers. But yes its a little clunky accessing nested values in broadcasts.
That makes sense and would work well enough in general. However, would it be possible to return an AbstractDimStack
if the broadcasted function returns a NamedTuple
and an AbstractDimArray
if it returns a scalar?
The idea came up trying to write
rand
samplers forAbstractDimStack
.Why isn't an
AbstractDimArray
anAbstractArray{<:NamedTuple}
?With
rand
, we would always want the sampler to return aNamedTuple
of values from each layer. But what does that mean with mixed-size layers? Currently indexing aDimStack
with a linearInt
doesn't do something sensible - it just indexes every layer at the same index inmap
. We should instead treat aDimStack
exactly like theDimTable
it can be unrolled into. Linear indexing aDimStack
should always give the same value as indexing a row of itsDimTable
- that is it really indexes with:I had actually forgotten that it doesn't do that already and consider it almost a bug. When all layers are the same size there will be no difference, and we can still just use regular linear indexing on all layers.
With that change
DimStack
can basically be anAbstractArray
. This meansrand
will work already without adding a sampler, andbroadcast
and iteration will also "just work", closing #412.The main breaking change is to
map
that currently works over stackArray
layers, notNamedTuple
values. We will need to replace it with e.g.maplayers
. This will need a lot of changes here and also in Rasters.jl. So probably in other packages too.Any thoughts @sethaxen @felixcremer @JoshuaBillson ?