search

meggart / DiskArrays.jl

Other
73 stars 13 forks source link

add lazy concatenating disk arrays and Base.cat method #102

Closed rafaqz closed 1 year ago

rafaqz commented 1 year ago

This is half code modified from DiskArrayTools.jl

It adds a ConcatDiskArray wrapper, and a cat method that applies it.

We could possibly try to merge if a ConcatDiskArray is passed to the ConcatDiskArray constructor?

meggart commented 1 year ago

We could possibly try to merge if a ConcatDiskArray is passed to the ConcatDiskArray constructor?

This would be a nice optimization. So far in my YAXArrays code I was usually using the direct constructor passing an array of arrays. However, since it is a common pattern to use things like reduce(cat,array_of_diskarrays) we should make sure to not have 1000 nested ConcatDiskArrays.

rafaqz commented 1 year ago

I might leave the unwrapping ConcatDiskArray part to another time, but I implemented writeblock! and tested mixed chunk sizes

meggart commented 1 year ago

Ok, thanks a lot. The nested ConcatDIskArrays problem I can take care of as well. So feel free to merge and tag a new version.