mehta-lab / recOrder

3D quantitative label-free imaging with phase and polarization
BSD 3-Clause "New" or "Revised" License
12 stars 4 forks source link

Chunk transfer functions #463

Closed talonchandler closed 9 months ago

talonchandler commented 9 months ago

Fixes #460. Supersedes #461. Tested on @dsundarraman's reconstructions.

This PR chunks the transfer functions along YX, matching the reconstruction chunking.

Details: Writing to a .zarr with __setitem__ (i.e. dataset['group_name'] = myarray) uses a single chunk, so requesting a large reconstruction failed when the chunk exceeded 2147483647 bytes. The fix is to use create_image instead of __setitem__ so that we can set the chunk size.

Notes on tests: my first commit was a test on a large transfer function that failed locally, and my second commit fixed the problem and made the test pass locally. Unfortunately, the github action fails on this test, likely because it uses too much memory(?). My third commit reverts the test.

codecov[bot] commented 9 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (d31ad91) 9.45% compared to head (1c6f81e) 9.45%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #463 +/- ## ===================================== Coverage 9.45% 9.45% ===================================== Files 29 29 Lines 4592 4592 ===================================== Hits 434 434 Misses 4158 4158 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.