Open mbauman opened 9 years ago
Great ideas, Matt!
Allow (or maybe even encourage) the use of an ordered hashmap for categorical axes to enable O(1) index lookup.
Julia could definitely use a perfect-hashing "dictionary" type. If anyone tackles this, please make it a standalone package rather than burying it in some other package. There would be many users.
Allow (or maybe even encourage) the use of an ordered hashmap for categorical axes to enable O(1) index lookup. NamedArrays now uses an OrderedDict for its axis names. Profiling showed that when subsetting such a NamedArray, much of the time went into making a new OrderedDict for the new NamedArray. I have a PR (#211) over at DataStructures that speeds this up quite a bit, but this PR is on hold until I can make it backwards compatible with julia 0.4. (Any suggestions would be most welcome.)
Allow (or maybe even encourage) the use of an ordered hashmap for categorical axes to enable O(1) index lookup.
+1
I'm sure the linear search will outperform hashing for small N (particularly with symbols), but what's the cutoff? 10? 100? What about strings? Chances are that folks won't be using categorical vectors to enumerate more than 100 elements.
FWIW in biology it is not uncommon to talk about categories of thousands of species, tens of thousands of gene families. In medicine O(10^5) identifiers for diagnoses, similar for patients.
See also work on new packages inspired by AxisArrays https://github.com/JuliaCollections/AxisArraysFuture/issues/1
We're getting to the point where the core API is stabilizing. We can now start making things fancier and building out interfaces to base methods and other packages (like #5). Here's a current brain-dump of some of my thoughts. Additions, critiques and comments are very welcome.
Remaining core infrastructure
setindex!
(mirroringgetindex
's capabilities). (#11)Axis
types to specify the dimension names. It would be an interesting experiment to store the dimension names asAxis
types instead of symbols. I'm not sure if that'd make things simpler or more convoluted. It may be a mixed bag.Possible additions to the core infrastructure
Add a third flavor of axis trait for Dimensional axes with elements of a discrete step-like type. The key defining characteristic of this element type is that their StepRanges must enumerate all values between the endpoints, allowing us to provide sensible indexing directly with a StepRange. This also means that there's no issues along the lines of floating point instability, so we can also allow indexing directly with single-elements of this type (and don't need to force the use of Intervals). The main use-case I see here is forDate
. Are there other types that satisfy these criteria? What is a sensible name for this trait?Axis
type (#15). Ensuring that large, non-Range Dimensional axes are monotonically increasing can take a long time (we may be able to speed this up some with a special Ordering type, but it's still O(n)). Similarly, ensuring that elements in categorical vectors are unique requires hashing all elements (which could be used for the above hashmap).eachslice
iterator would be nice and useful in and of itself, and if we allow the same sort of syntax and semantics asmapslices
it can serve as the building block for augmenting that Base function.Signals.window
. I was thinking of allowing windowing as an indexing operation (https://github.com/mbauman/Signals.jl/issues/10), but constructing vectors of interval types with deferred promotion (to allow, e.g., windows specified in time about integer indices) has been very challenging.Extensions to Base
sum
,mean
,maximum
,mapslices
, etc. We can also return AxisArrays with the properly reduced axis set, dropping a dimension and eliminating the type-unstable squeezes.Extensions to other packages