search

radio-astro-tools / spectral-cube

Library for reading and analyzing astrophysical spectral data cubes
http://spectral-cube.rtfd.org
BSD 3-Clause "New" or "Revised" License
97 stars 65 forks source link

Avoid memory issues when computing statistics #752

Closed astrofrog closed 2 years ago

astrofrog commented 3 years ago

For some reason, reshaping the array then running map_blocks leads to memory issues when dealing with Numpy memory mapped arrays, e.g. with FITS files. This new approach avoids reshaping the array at all.

codecov-commenter commented 3 years ago

Codecov Report

Merging #752 (a230448) into master (66c013f) will increase coverage by 0.07%. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #752      +/-   ##
==========================================
+ Coverage   77.48%   77.55%   +0.07%     
==========================================
  Files          24       24              
  Lines        5557     5775     +218     
==========================================
+ Hits         4306     4479     +173     
- Misses       1251     1296      +45
Impacted Files Coverage Δ
spectral_cube/dask_spectral_cube.py 85.17% <100.00%> (-0.10%) :arrow_down:
spectral_cube/lower_dimensional_structures.py 80.62% <0.00%> (-1.83%) :arrow_down:
spectral_cube/spectral_cube.py 76.44% <0.00%> (-1.22%) :arrow_down:
spectral_cube/utils.py 100.00% <0.00%> (ø)
spectral_cube/stokes_spectral_cube.py 92.00% <0.00%> (ø)
spectral_cube/base_class.py 86.33% <0.00%> (+0.48%) :arrow_up:
spectral_cube/io/fits.py 85.33% <0.00%> (+1.00%) :arrow_up:
spectral_cube/cube_utils.py 81.44% <0.00%> (+1.27%) :arrow_up:
spectral_cube/conftest.py 92.48% <0.00%> (+1.94%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 66c013f...a230448. Read the comment docs.