search

meggart / DiskArrays.jl

Other
73 stars 13 forks source link

Fix zip and other cases of simultaneous iteration #103

Closed rafaqz closed 1 year ago

rafaqz commented 1 year ago

zip is used in array equality, and breaks chunking order when arrays with different chunks are used.

We should at minimum fix it so that the chunks of the first disk array found are used for all others (and for regular arrays). This can be achieved with a RechunkDiskArray wrapper that simply forces a different chunk pattern. It may be slow in some cases but will at least be correct.

Later we could optimise the combined iteration pattern of all diskarrays passed to zip.

rafaqz commented 1 year ago

It seems to be the == comparison calling zip that causes the broken tests for cat

(I have a fix for this finished, will just wait until #102 is merged for simplicity)