Open tlambert03 opened 1 month ago
nooo...
my naive approach hit a snag... need to be directly updating the 3d texture objects
asynchronous 3d chunked loading!
https://github.com/pyapp-kit/ndv/assets/1609449/c6d4121e-4cd8-423b-be8c-e8973ff256ed
note to self... some array types (such as tensorstore and dask arrays) provide their own async chunk retrieval and futures patterns. There should be a way to defer to those for array types that support it
@tlambert03 finally back from a workshop and checking this out properly - it's wonderful, you're my favorite :)
:) thanks after some encouraging proofs of principle, i've gone back to thinking harder about the request/response model. it's all getting a little blurrier in my head. I definitely think there's a nice generalizable slicing abstraction to be had here, that doesn't overcomplicate things, but it's slightly elusive at the moment. would love to chat about it
this sounds like fun, let's find a time to chat - will ping
this draft PR moves much of the logic for indexing into data into a Chunker class that takes care of the asynchronous requests to the data store, and yields chunks back to the viewer, which then need only worry about putting them at the appropriate place/channel in the canvas. It's working pretty well (note that in the demo, I've added an artificial sleep of 20 ms to every individual chunk request, and chunked it overly dramatically)
This demo shows:
(64, 32)
, to show that square chunks are not necessary. Obviously, the performance benefit here would depend on aligning chunk requests with the chunks of the storage.https://github.com/pyapp-kit/ndv/assets/1609449/fe4f1980-5373-42bc-a2b6-3b0858c14325
the implementation should extend to 3d relatively straightforwardly. Ultimately, all that the chunker cares about is that you tell it the full extent of the data you want, and how you'd like to chunk it up, and all the viewer cares about is receiving the final chunks, and the offset required to place them on the canvas
lots of cleanup needed, and need to make sure that it doesn't affect performance of the simple cases