search

JuliaArrays / ArraysOfArrays.jl

Efficient storage and handling of nested arrays in Julia
Other
43 stars 9 forks source link

getindex(.) return type does not match eltype #2

Open colinxs opened 5 years ago

colinxs commented 5 years ago

First off, awesome idea/package! It's been quite useful in my own work.

I was really excited to use your package in conjunction with StructArrays but ran into an issue (that I attempted to fix with a PR to StructArrays) as explained here. The owner of that package pointed out, however, that the real issue is the following:

julia> x=nestedview(rand(2,3,10), 2);
julia> x[1] isa eltype(x)
false

What are your thoughts on modifying eltype to be consistent with the true return type of getindex?

-Colin

colinxs commented 4 years ago

@oschulz @bramtayl you happen to see my message on Slack?

bramtayl commented 4 years ago

Just commented there. Sorry, I don't check slack very often.

oschulz commented 4 years ago

No, sorry, missed it - I'm an intermittent slacker ... :-)

oschulz commented 4 years ago

@colinxs, looks like a bug in ArraysOfArrays, can you open an issue an repost your comment from slack?

colinxs commented 4 years ago

@oschulz done!

Concerning the merging of SpecialArrays/JuliennedArrays: Given that we all appear to be fairly busy, I think it's easiest for everyone if SpecialArrays is merged into JuliennedArrays.

ArraysOfArrays has many moving parts and is quite different, while SpecialArrays/JuliennedArrays are effectively the same thing, just implemented a bit differently. Given that @bramtayl has been working on the challenges of type stable "dimension surgery" (Brandon I think I've read damn near every issue you've commented on), he may also be able to better maintain/push forward the whole mapslices story.

@oschulz, given that you have many packages depending on ArraysOfArrays, this may also make things easier for you.

If everyone is alright with this I'll make a PR this weekend. There's two registered packages depending on JuliennedArrays, both of which have compat entries upper bounding it to v0.2, so I think we can just include this in the next minor release without a deprecation cycle. @bramtayl could you add me as a collaborator?

bramtayl commented 4 years ago

I think I've read damn near every issue you've commented on

Im sorry you had to suffer through them!

could you add me as a collaborator?

Sure!

oschulz commented 4 years ago

Given that we all appear to be fairly busy, I think it's easiest for everyone if SpecialArrays is merged into JuliennedArrays.

Sure, I guess that's maybe the fastest route for you

@oschulz, given that you have many packages depending on ArraysOfArrays, this may also make things easier for you.

Not that many, but those do need it - I'd be fine to adapt to API changes, though. I just needed the statistics support, but I believe that's now fully compatible with Statistics/StatsBase, semantically.

Maybe longer term, we can still converge? Until then, it would be nice if we could avoid conflicts between exported functions (e.g. flatview, nestedview), though.

colinxs commented 4 years ago

@bramtayl no worries, it was educational! I'll go ahead and make one big PR then and we can disect from there.

I definitely see things converging in the future! Right now there's a ton of packages with similar goals (RecursiveArrayTools, SliceMap, StarSlice, etc.) but I think the diversity has lead to better ideas (e.g. I may not have made SpecialArrays if both ArraysOfArrays and JuliennedArrays didn't already exist).

flatview and the inner_* are the only conflicts right now. Perhaps we can discuss how to work around that in the PR to JuliennedArrays.

Sorry for all the trouble!

oschulz commented 4 years ago

"RecursiveArrayTools" is actually almost fully orthogonal - from what I understand, it's purpose access nested array with a flat-array-like API. So kinda the reverse from what he do, actually.

colinxs commented 4 years ago

Yup, it's purpose is definitely nested-->flat, but that's just the inverse of flat->nested. As the JuliaennedArrays/SpecialArrays code shows they share quite a bit in common.

oschulz commented 4 years ago

I definitely see things converging in the future!

@colinxs , as a first step in that direction, how would you feel about publishing your FlattenedArray implementation as a lightweight, standalone "FlattenedArrays.jl" or so? I'm sure it would be useful on it's own for many use cases anyhow.

colinxs commented 4 years ago

@oschulz I think it's easiest to keep it with SlicedArray gives that they are inverses of each-other. There's very few dependencies, however:

(SpecialArrays) pkg> st
Project SpecialArrays v0.1.1
Status `~/workspace/juliadev/SpecialArrays/Project.toml`
  [79e6a3ab] Adapt v1.0.1
  [ffbed154] DocStringExtensions v0.8.1
  [ae029012] Requires v1.0.1
  [c4a57d5a] UnsafeArrays v1.0.1
  [e88e6eb3] Zygote v0.4.20

julia> @time using SpecialArrays
  0.253651 seconds (421.11 k allocations: 22.848 MiB)

Note that Zygote is loaded with Requires.

Still haven't made the PR into JuliennedArrays, I've been cleaning things up on my end before :)

oschulz commented 4 years ago

Sure, sounds good. Maybe ArraysOfArray can depend on JuliennedArrays then for a while, until we come up with a plan to converge everything.

colinxs commented 4 years ago

Sounds good! I haven't yet made the PR to JuliennedArrays, but did just finish cleaning/simplifying a bunch of things in https://github.com/Lyceum/SpecialArrays.jl/pull/2. I'll make the PR this weekend and we can chat specifics there :)

cc: @bramtayl

oschulz commented 4 years ago

I think that efficient arrays-of-arrays data structure could become very relevant in the future, since with Julia v1.5, views become very cheap. Here are some benchmarks: JuliaArrays/UnsafeArrays.jl#8

Moelf commented 3 years ago

could become very relevant in the future

bump, the future is now! Let's do Jim's (future) work for him image

oschulz commented 3 years ago

I wrote to Jim last night, letting him know that we already have it :-)

Moelf commented 3 years ago

AK supports n-jaggedness so we're still some distance away, btw, I definitely reinvented StructArray in UnROOT.jl:

https://github.com/JuliaArrays/StructArrays.jl/blob/ce6da2920d6004ade4dc9d2e07dabe2ff89a321a/src/lazy.jl#L35-L36

https://github.com/tamasgal/UnROOT.jl/blob/5d44d7c640adb858f1c132b6d6b9375b8e62fe09/src/iteration.jl#L296-L297

oschulz commented 3 years ago

AK supports n-jaggedness

Do you mean vectors of vectors of [...] of arrays? ArraysOfArrays.VectorOfArrays can do that, actually.

oschulz commented 3 years ago

I definitely reinvented StructArray in UnROOT.jl

Do you think it would be possible to use StructArray then, instead, longer term? It's a very well supported package.

Moelf commented 3 years ago

let's see if we survive into longer-term... but yes that's definitely something worth trying