Combine fits.fz fails with hdu argument

[KerryPaterson]

Ccdproc.combine will automatically combine the first hdu with data unless the hdu argument is passed. Trying to combine a list of fpacked fits files (i.e. file.fits.fz), the ccdproc.combine works fine with the default hdu, but if you specify an hdu it fails.

E.g: files = ['file1.fits.fz','file2.fits.fz'] ccdproc.combine(files) works ccdproc.combine(files,hdu=1) fails

The problem seems to be with astropy.io.fits?

Error:

ValueError Traceback (most recent call last)

in () ----> 1 bias_master = ccdproc.combine(bias_list,hdu=1,unit=u.adu) /Users/Kerry/anaconda2/lib/python2.7/site-packages/ccdproc-1.2.dev916-py2.7.egg/ccdproc/combiner.pyc in combine(img_list, output_file, method, weights, scale, mem_limit, clip_extrema, nlow, nhigh, minmax_clip, minmax_clip_min, minmax_clip_max, sigma_clip, sigma_clip_low_thresh, sigma_clip_high_thresh, sigma_clip_func, sigma_clip_dev_func, **ccdkwargs) 642 643 # Trim image --> 644 ccd_list.append(trim_image(imgccd[x:xend, y:yend])) 645 646 # Create Combiner for tile /Users/Kerry/anaconda2/lib/python2.7/site-packages/ccdproc-1.2.dev916-py2.7.egg/ccdproc/log_meta.pyc in wrapper(_args, *_kwd) 90 91 for k, v in six.iteritems(meta_dict): ---> 92 result._insert_in_metadata_fits_safe(k, v) 93 return result 94 /Users/Kerry/anaconda2/lib/python2.7/site-packages/ccdproc-1.2.dev916-py2.7.egg/ccdproc/ccddata.pyc in _insert_in_metadata_fits_safe(self, key, value) 551 # Shorten, sort of... 552 short_name = _short_names[key] --> 553 self.meta[key] = (short_name, "Shortened name for ccdproc command") 554 self.meta[short_name] = value 555 else: /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/compressed.pyc in **setitem**(self, key, value) 138 return 139 --> 140 super(CompImageHeader, self).**setitem**(key, value) 141 142 /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in **setitem**(self, key, value) 193 # If we get an IndexError that should be raised; we don't allow 194 # assignment to non-existing indices --> 195 self._update((key, value, comment)) 196 197 def __delitem__(self, key): /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/compressed.pyc in _update(self, card) 247 return 248 --> 249 super(CompImageHeader, self)._update(card) 250 251 if keyword in Card._commentary_keywords: /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in _update(self, card) 1676 else: 1677 # A new keyword! self.append() will handle updating _modified -> 1678 self.append(card) 1679 1680 def _cardindex(self, key): /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/compressed.pyc in append(self, card, useblanks, bottom, end) 191 192 super(CompImageHeader, self).append(card=card, useblanks=useblanks, --> 193 bottom=bottom, end=end) 194 195 remapped_keyword = self._remap_keyword(card.keyword) /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in append(self, card, useblanks, bottom, end) 1272 # this will just be 80 (Card.length) but it may be longer for 1273 # CONTINUE cards -> 1274 self._useblanks(len(str(card)) // Card.length) 1275 1276 self._modified = True /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in _useblanks(self, count) 1835 for _ in range(count): 1836 if self._cards[-1].is_blank: -> 1837 del self[-1] 1838 else: 1839 break /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/hdu/compressed.pyc in **delitem**(self, key) 154 155 if isinstance(key, int): --> 156 keyword, index = self._keyword_from_index(key) 157 elif isinstance(key, tuple): 158 keyword, index = key /Users/Kerry/anaconda2/lib/python2.7/site-packages/astropy/io/fits/header.pyc in _keyword_from_index(self, idx) 1744 1745 keyword = self._cards[idx].keyword -> 1746 repeat = self._keyword_indices[keyword].index(idx) 1747 return keyword, repeat 1748 ValueError: 68 is not in list

[MSeifert04]

@KerryPaterson Thank you for reporting the bug.

I'm actually surprised that it works at all because fits.fz is not automatically recognized as fits-file.

Is it possible to share one "affected" file so we can confirm and debug the issue?

[KerryPaterson]

Hi Michael

Thanks for looking at this.

I was under the impression that python did recognise fits.fz as fits file (open, getdata, getheader from astropy all work as well as CCDData.read and combine [without the hdu argument] from ccdproc) but can not compress fz files.

The compressed fits file can be found here: https://www.dropbox.com/s/n3ope5mj8kzuux8/test1.fits.fz?dl=0

Setting fz_list = ['test1.fits.fz','test1.fits.fz','test1.fits.fz'] Running ccdproc.combine(fz_list,unit=u.adu) works fine, but ccdproc.combine(fz_list,hdu=2,unit=u.adu) fails. Modules imported: import ccdproc import astropy.units.astrophys as u

Regards Kerry

[crawfordsm]

@KerryPaterson -- with the most recent version of ccdproc and astropy, I'm not currently able to reproduce this. Can you send along the versions that you are using for both of those?

[KerryPaterson]

Astropy: 1.3.dev Ccdproc: 1.2.dev916

[MSeifert04]

@KerryPaterson I'm confused. The file you shared only has 2 extensions:

from astropy.io import fits

with fits.open('C:/-/Downloads/test1.fits.fz') as hdus:
    hdus.info()

Filename: C:/-/Downloads/test1.fits.fz
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU       8   ()              
1    COMPRESSED_IMAGE  CompImageHDU     13   (1211, 9232)   int16   

So the hdu=2 does not exist (note that python uses 0-based-indexing, so the second hdu is hdu=1). But ccdproc by default ignores data-less extensions (see log-messages):

import ccdproc
fz_list = ['C:/-/Downloads/test1.fits.fz'] * 3
ccdproc.combine(fz_list, unit='adu')

INFO: first HDU with data is extension 1. [ccdproc.ccddata]

WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]

INFO: splitting each image into 1 chunks to limit memory usage to 16000000000.0 bytes. [ccdproc.combiner]
INFO: first HDU with data is extension 1. [ccdproc.ccddata]

WARNING: VerifyWarning: Keyword name 'trim_image' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]

INFO: first HDU with data is extension 1. [ccdproc.ccddata]
INFO: first HDU with data is extension 1. [ccdproc.ccddata]

CCDData([[ 2145,  2143,  2144, ...,  2139,  2136,  2134],
         [ 2000,  1995,  1990, ...,  1797,  2782, 65528],
         [ 1982,  1983,  1983, ...,  1841,  2548, 65529],
         ..., 
         [ 1866,  1857,  1862, ...,  1723,  2459, 65529],
         [ 1857,  1863,  1860, ...,  1733,  2443, 65529],
         [ 1857,  1861,  1860, ...,  1725,  2439, 65529]])

which also works for hdu=1

fz_list = ['C:/-/Downloads/test1.fits.fz'] * 3
ccdproc.combine(fz_list, hdu=1, unit='adu')

INFO: splitting each image into 1 chunks to limit memory usage to 16000000000.0 bytes. [ccdproc.combiner]

WARNING: VerifyWarning: Keyword name 'trim_image' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]

CCDData([[ 2145,  2143,  2144, ...,  2139,  2136,  2134],
         [ 2000,  1995,  1990, ...,  1797,  2782, 65528],
         [ 1982,  1983,  1983, ...,  1841,  2548, 65529],
         ..., 
         [ 1866,  1857,  1862, ...,  1723,  2459, 65529],
         [ 1857,  1863,  1860, ...,  1733,  2443, 65529],
         [ 1857,  1861,  1860, ...,  1725,  2439, 65529]])

[KerryPaterson]

Sorry about that, the original file has 32 extensions, the test1.fits was a file I wrote out with a single extension to save on space. Here is a link to the original file: https://www.dropbox.com/s/4w7orq06ss2z5a6/kmts.20151202.030382.fits.fz?dl=0 . It seems it may be an issue with one of the keywords in the header? Adding more extensions to the test1.fits.fz, ccdproc.combine didn't seem to have an issue with the hdu argument (new test1 file here: https://www.dropbox.com/s/0iwqqs088c0abx0/test1.fits.fz?dl=0)

[MSeifert04]

Thank you, now I can reproduce the issue. I'll have a look.

[crawfordsm]

@KerryPaterson can you update to the most recent development version of both astropy and ccdproc and try again?

[KerryPaterson]

@crawfordsm I get the same error (ccdproc version = 1.2.dev921, astropy version = 1.3.dev16028).

[MSeifert04]

@KerryPaterson Could you check if #393 fixes the problem?

However that will just fix one symptom of this issue (which should probably be found and fixed in astropy) so it's likely one would encounter it in several functions of ccdproc. But for those functions one can disable logging to the Header (which caused the Exception) by setting add_keywords=None.

FYI: Another way to avoid the exception is to "reparse" the Header (weird but it works):

import ccdproc
from astropy.io import fits
ccd = ccdproc.CCDData.read('name.fits.fz', hdu=1, unit='adu')
ccd.meta = fits.Header(ccd.meta)
ccdproc.combine([ccd, ccd, ccd])

[MSeifert04]

Another question: Am I allowed to link to your data file (the big one) if I open an issue in astropy? Because it may be that someone there knows how to find and eliminate the root of the problem and I wasn't able to create a file from scratch to reproduce this issue.

[crawfordsm]

@MSeifert04 Yes, I've tracked down the problem to the Header.append function in astropy.io.fits.header such that the following fails with the same error:

ccd = ccdproc.CCDData.read('kmts.20151202.030382.fits.fz', hdu=2, unit='adu')
key = 'trim_image'
short_name = 'trimim'
card = (key, short_name, "Shortened name for command")
ccd.meta.append(card)

Still need to look into that a little more detail to figure out what is going on.

[crawfordsm]

@KerryPaterson We need to merge #393, but once that is, it should fix your problem with combine. For using other ccdproc tasks, you should include add_keyword=False in the keywords for the task and then those tasks should work until there is a fix in astropy.io.fits for this problem.

[KerryPaterson]

393 fixes the problem, thanks! @MSeifert04 Sure, you can copy the link over to astropy.

[MSeifert04]

@KerryPaterson Ok, thanks :+1:

[crawfordsm]

For reference, this problem was deep into how header treated HIERARCH keywords and hopefully fixed in https://github.com/astropy/astropy/pull/5321