Large Cube Loading Error

[jmangum]

Encountered an error (rather than a warning, that I had become accustomed to seeing) with trying to create moments from an image cube which is apparently too large for SpectralCube:

cerussite:CubeLineMoment jmangum$ ipython ~/Python/mangum_galaxies-master/CubeLineMoment.py HCN32.yamlWARNING: StokesWarning: Cube is a Stokes cube, returning spectral cube for I component [spectral_cube.io.core]
WARNING: StokesWarning: Cube is a Stokes cube, returning spectral cube for I component [spectral_cube.io.core]
WARNING: PossiblySlowWarning: This function (<function BaseSpectralCube.argmax at 0x196eb5280>) requires loading the entire cube into memory and may therefore be slow. [spectral_cube.utils]
WARNING: PossiblySlowWarning: This function (<function BaseSpectralCube.max at 0x196eb3ee0>) requires loading the entire cube into memory and may therefore be slow. [spectral_cube.utils]
/Users/jmangum/anaconda3/envs/python39/lib/python3.9/site-packages/spectral_cube/spectral_cube.py:441: RuntimeWarning: All-NaN slice encountered
  out = function(self._get_filled_data(fill=fill,
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/Python/mangum_galaxies-master/CubeLineMoment.py:1026, in <module>
   1022     return locals()
   1025 if __name__ == "__main__":
-> 1026     new_locals = main()
   1027     locals().update(new_locals)

File ~/Python/mangum_galaxies-master/CubeLineMoment.py:970, in main()
    928     '''
    929 #        params['sample_pixel'] = ast.literal_eval(params['sample_pixel'])
    930         # Check to make sure that sample pixexl regions file exists.  Open it if
   (...)
    938             raise ValueError("Sample pixel file {0} does not exist.".format(params['sample_pixel']))
    939     '''
    941         # NOTE: The following is necessary to provide WCS information for interpreting sample_pixel values
    942         #       Try moving to later after cube is opened, avoiding multiple opens of cutoutcube...
    943         # ===========
   (...)
    966
    967     # Read parameters from dictionary
    969     (cube, spatialmaskcube, spatial_mask, noisemap, noisemapbright,
--> 970      centroid_map, width_map, max_map, peak_velocity, sample_pixel) = cubelinemoment_setup(**params)
    972     params.pop('cube')
    973     params.pop('sample_pixel')

File ~/Python/mangum_galaxies-master/CubeLineMoment.py:268, in cubelinemoment_setup(cube, cuberegion, cutoutcube, cutoutcuberegion, vz, target, brightest_line_name, brightest_line_frequency, width_line_frequency, velocity_half_range, noisemapbright_baseline, noisemap_baseline, spatial_mask_limit, sample_pixel, min_width, mask_negatives, **kwargs)
    265         raise ValueError("noisemapbright_baseline ({0},{1}) out of range ({2},{3})".format(low,high,0,noisecubebright.header['NAXIS3']))
    267 # need to use an unmasked cube
--> 268 noisemapbright = noisecubebright.with_mask(mask[:,None,None]).std(axis=0)
    269 print("noisemapbright peak = {0}".format(np.nanmax(noisemapbright)))
    271 # Create noisemapbright_baseline mask for plotting

File ~/anaconda3/envs/python39/lib/python3.9/site-packages/spectral_cube/spectral_cube.py:102, in aggregation_docstring.<locals>.wrapper(*args, **kwargs)
    100 @wraps(func)
    101 def wrapper(*args, **kwargs):
--> 102     return func(*args, **kwargs)

File ~/anaconda3/envs/python39/lib/python3.9/site-packages/spectral_cube/utils.py:32, in warn_slow.<locals>.wrapper(self, *args, **kwargs)
     30 warn_how = (kwargs.get('how') == 'cube') or 'how' not in kwargs
     31 if self._is_huge and not self.allow_huge_operations and warn_how:
---> 32     raise ValueError("This function ({0}) requires loading the entire "
     33                      "cube into memory, and the cube is large ({1} "
     34                      "pixels), so by default we disable this operation. "
     35                      "To enable the operation, set "
     36                      "`cube.allow_huge_operations=True` and try again.  "
     37                      "Alternatively, you may want to consider using an "
     38                      "approach that does not load the whole cube into "
     39                      "memory by specifying how='slice' or how='ray'.  "
     40                      "See {bigdataurl} for details."
     41                      .format(str(function), self.size,
     42                              bigdataurl=bigdataurl))
     43 elif warn_how and not self._is_huge:
     44     # TODO: add check for whether cube has been loaded into memory
     45     warnings.warn("This function ({0}) requires loading the entire cube into "
     46                   "memory and may therefore be slow.".format(str(function)),
     47                   PossiblySlowWarning
     48                  )

ValueError: This function (<function BaseSpectralCube.std at 0x196eb3b80>) requires loading the entire cube into memory, and the cube is large (103612041 pixels), so by default we disable this operation. To enable the operation, set `cube.allow_huge_operations=True` and try again.  Alternatively, you may want to consider using an approach that does not load the whole cube into memory by specifying how='slice' or how='ray'.  See https://spectral-cube.readthedocs.io/en/latest/big_data.html for details.
cerussite:CubeLineMoment jmangum$

I tried setting cube.allow_huge_operations=True in cubelinemoment_setup but that resulted in an attribute error:

AttributeError: 'str' object has no attribute 'allow_huge_operations'

[keflavich]

put cube.allow_huge_operations=True after cube = SpectralCube.read(cube).with_spectral_unit(u.Hz)

[jmangum]

Thanks @keflavich . Tried that. Same error.

[keflavich]

ok try adding it also after https://github.com/keflavich/cube-line-extractor/blob/master/CubeLineMoment.py#L142 (i.e., after the last time cube = <something> is done