search

aperiosoftware / astropy

Repository for the Astropy core package
www.astropy.org
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Added support for ZBLANK #43

Closed astrofrog closed 1 year ago

astrofrog commented 1 year ago

This was a bit tricky to test out but I managed to get started by using fpack to create a real-life compressed file with a NaN. I think we didn't actually previously support this properly in astropy or fitsio. With the reference file included here produced by fpack:

fitsio:

In [1]: from fitsio import FITS

In [2]: FITS('compressed_with_nan.fits', 'r')[1].read()
Out[2]: 
array([[  0.24800134,   0.93837476,   1.86717916,   2.7572279 ,
          3.8365202 ,   4.81325245,   6.07874541,   7.19016685,
          8.25755919,   8.94436647,  10.21816534,  10.96331429],
       [ 11.85326338,  -5.32149553,  13.81193948,  15.10613155,
         15.78522205,  16.83257818,  17.81239832,  19.09749043,
         19.89572431,  21.02495848,  22.16447399,  23.12375106],
       [ 24.11236739,  24.82116628,  26.16324592,  27.07616186,
         28.25843024,  29.11530542,  29.81170208,  30.84318564,
         31.8651949 ,  33.00300841,  34.18679967,  34.95438466],
       [ 36.13891673,  36.87695789,  38.01172495,  38.83706474,
         40.02535057,  40.85193944,  42.00145825,  42.88512335,
         44.13053724,  44.81240045,  45.81124977,  47.16511866],
       [ 47.8302145 ,  48.75466275,  49.81585455,  51.12784076,
         52.08974266,  53.09945488,  53.84582118,  55.18520071,
         55.7960822 ,  56.83639076,  57.83776176,  59.11492913],
       [ 59.85114574,  61.26117682,  62.04878569,  62.82218981,
         63.88607383,  64.83932519,  66.15294444,  66.80629131,
         68.15489802,  69.21407623,  70.22950176,  70.79220729],
       [ 72.03176049,  73.24666723,  73.84131311,  75.10758251,
         76.15610049,  76.74972581,  78.121889  ,  78.85475872,
         80.08750787,  81.25353301,  82.20820131,  83.24902675],
       [ 83.92690852,  85.20415628,  86.25493305,  87.07144213,
         88.18303637,  89.00969058,  89.79208753,  90.98874658,
         92.06370719,  93.09941351,  93.80499528,  95.06551634],
       [ 96.03795214,  96.8279785 ,  97.87266748,  99.11046648,
        100.23236876, 100.80277421, 102.24092209, 103.00673964,
        103.93654425, 104.86408338, 105.98073709, 106.92482162],
       [108.23702145, 109.19076106, 109.82427748, 111.26009878,
        112.0067016 , 112.88903692, 114.07103467, 114.82678465,
        115.86056711, 117.25534537, 118.01094506, 119.13983734],
       [120.09037049, 121.03239009, 121.77888891, 123.00993662,
        123.88564231, 125.02300276, 126.0655005 , 127.03287914,
        127.83126459, 128.88703329, 130.25815717, 130.78218727],
       [132.22038248, 132.75691605, 133.87444555, 134.86106918,
        136.21591094, 137.23214199, 138.23995742, 138.78825754,
        140.05056295, 141.04726505, 141.95683582, 143.00620657]])

astropy before this PR:

In [1]: from astropy.io import fits

In [2]: fits.open('compressed_with_nan.fits')[1].data
Out[2]: 
array([[  0.24800134,   0.93837476,   1.86717916,   2.7572279 ,
          3.8365202 ,   4.81325245,   6.07874541,   7.19016685,
          8.25755919,   8.94436647,  10.21816534,  10.96331429],
       [ 11.85326338,  -5.32149553,  13.81193948,  15.10613155,
         15.78522205,  16.83257818,  17.81239832,  19.09749043,
         19.89572431,  21.02495848,  22.16447399,  23.12375106],
       [ 24.11236739,  24.82116628,  26.16324592,  27.07616186,
         28.25843024,  29.11530542,  29.81170208,  30.84318564,
         31.8651949 ,  33.00300841,  34.18679967,  34.95438466],
       [ 36.13891673,  36.87695789,  38.01172495,  38.83706474,
         40.02535057,  40.85193944,  42.00145825,  42.88512335,
         44.13053724,  44.81240045,  45.81124977,  47.16511866],
       [ 47.8302145 ,  48.75466275,  49.81585455,  51.12784076,
         52.08974266,  53.09945488,  53.84582118,  55.18520071,
         55.7960822 ,  56.83639076,  57.83776176,  59.11492913],
       [ 59.85114574,  61.26117682,  62.04878569,  62.82218981,
         63.88607383,  64.83932519,  66.15294444,  66.80629131,
         68.15489802,  69.21407623,  70.22950176,  70.79220729],
       [ 72.03176049,  73.24666723,  73.84131311,  75.10758251,
         76.15610049,  76.74972581,  78.121889  ,  78.85475872,
         80.08750787,  81.25353301,  82.20820131,  83.24902675],
       [ 83.92690852,  85.20415628,  86.25493305,  87.07144213,
         88.18303637,  89.00969058,  89.79208753,  90.98874658,
         92.06370719,  93.09941351,  93.80499528,  95.06551634],
       [ 96.03795214,  96.8279785 ,  97.87266748,  99.11046648,
        100.23236876, 100.80277421, 102.24092209, 103.00673964,
        103.93654425, 104.86408338, 105.98073709, 106.92482162],
       [108.23702145, 109.19076106, 109.82427748, 111.26009878,
        112.0067016 , 112.88903692, 114.07103467, 114.82678465,
        115.86056711, 117.25534537, 118.01094506, 119.13983734],
       [120.09037049, 121.03239009, 121.77888891, 123.00993662,
        123.88564231, 125.02300276, 126.0655005 , 127.03287914,
        127.83126459, 128.88703329, 130.25815717, 130.78218727],
       [132.22038248, 132.75691605, 133.87444555, 134.86106918,
        136.21591094, 137.23214199, 138.23995742, 138.78825754,
        140.05056295, 141.04726505, 141.95683582, 143.00620657]])

with this PR

In [1]: from astropy.io import fits

In [2]: fits.open('compressed_with_nan.fits')[1].data
Out[2]: 
array([[  0.24800134,   0.93837476,   1.86717916,   2.7572279 ,
          3.8365202 ,   4.81325245,   6.07874541,   7.19016685,
          8.25755919,   8.94436647,  10.21816534,  10.96331429],
       [ 11.85326338,          nan,  13.81193948,  15.10613155,
         15.78522205,  16.83257818,  17.81239832,  19.09749043,
         19.89572431,  21.02495848,  22.16447399,  23.12375106],
       [ 24.11236739,  24.82116628,  26.16324592,  27.07616186,
         28.25843024,  29.11530542,  29.81170208,  30.84318564,
         31.8651949 ,  33.00300841,  34.18679967,  34.95438466],
       [ 36.13891673,  36.87695789,  38.01172495,  38.83706474,
         40.02535057,  40.85193944,  42.00145825,  42.88512335,
         44.13053724,  44.81240045,  45.81124977,  47.16511866],
       [ 47.8302145 ,  48.75466275,  49.81585455,  51.12784076,
         52.08974266,  53.09945488,  53.84582118,  55.18520071,
         55.7960822 ,  56.83639076,  57.83776176,  59.11492913],
       [ 59.85114574,  61.26117682,  62.04878569,  62.82218981,
         63.88607383,  64.83932519,  66.15294444,  66.80629131,
         68.15489802,  69.21407623,  70.22950176,  70.79220729],
       [ 72.03176049,  73.24666723,  73.84131311,  75.10758251,
         76.15610049,  76.74972581,  78.121889  ,  78.85475872,
         80.08750787,  81.25353301,  82.20820131,  83.24902675],
       [ 83.92690852,  85.20415628,  86.25493305,  87.07144213,
         88.18303637,  89.00969058,  89.79208753,  90.98874658,
         92.06370719,  93.09941351,  93.80499528,  95.06551634],
       [ 96.03795214,  96.8279785 ,  97.87266748,  99.11046648,
        100.23236876, 100.80277421, 102.24092209, 103.00673964,
        103.93654425, 104.86408338, 105.98073709, 106.92482162],
       [108.23702145, 109.19076106, 109.82427748, 111.26009878,
        112.0067016 , 112.88903692, 114.07103467, 114.82678465,
        115.86056711, 117.25534537, 118.01094506, 119.13983734],
       [120.09037049, 121.03239009, 121.77888891, 123.00993662,
        123.88564231, 125.02300276, 126.0655005 , 127.03287914,
        127.83126459, 128.88703329, 130.25815717, 130.78218727],
       [132.22038248, 132.75691605, 133.87444555, 134.86106918,
        136.21591094, 137.23214199, 138.23995742, 138.78825754,
        140.05056295, 141.04726505, 141.95683582, 143.00620657]])

I didn't yet implement the ability to write per-tile different values of ZBLANK but as far as I can tell we didn't expose that before anyway so I don't think it needs to be done (I'm not even sure how one would have written any file out with ZBLANK before).

Cadair commented 1 year ago

Why would you even want different tile ZBLANKS?

astrofrog commented 1 year ago

Why would you even want different tile ZBLANKS?

I don't know but the FITS standard allows it :laughing: