Mangled import of long strings in header keys when read by `AstroImage(filename)`

[EastEriq]

E.g., this file

imagefile='/raid/enrico/PTF/2ndTake/images/level1/proc/2011/10/11/f11/c2/p5/v2/PTF_201110114398_i_p_scie_t103316_u010636821_f11_p100037_c02.fits'

contains this section of the header:

ORIGNAME= '/data/PTF_default_24854.fits' / Filename as written by the camera    
FILENAME= 'PTF201110114398_11_o_24854.fits' / Filename of delivered camera image
PROCORIG= 'IPAC-PTF pipelines' / Processing origin                              
PROCDATE= 'Sun Dec 18 21:44:36 2011' / Processing date/time (Pacific time)      
PTFVERSN=                   5. / Version of PTFSCIENCEPIPELINE program          
PMASKPTH= '/ptf/pos/archive/fallbackcal/pmasks/' / Pathname of pixel mask       
PMASKFIL= '70sOn35s_pixmask_chip2.trimmed.v4.fits' / Filename of pixel mask     
SFLATPTH= '/ptf/pos/sbx2/2011/10/11/f11/c2/cal/p4/cId62146/' / Pathname of super
SFLATFIL= 'PTF_201110110000_i_s_flat_t120000_u000062146_f11_p000000_c02.fits'   
SBIASPTH= '/ptf/pos/sbx2/2011/10/10/f2/c2/p13/v1/' / Pathname of super bias     
SBIASFIL= 'PTF_201110100000_i_s_bias_t120000_u000050818_f00_p000000_c02.fits'   

If I read this file with AI=AstroImage(imagefile) I get inside the header, shown here for clarity in format short

>> AI.Header

ans =

  327×3 cell array

...

    {'ORIGNAME'}    {'/data/PTF_default_…'}    {' Filename as writt…'}
    {'FILENAME'}    {'PTF201110114398_11…'}    {' Filename of deliv…'}
    {'PROCORIG'}    {'IPAC-PTF pipelines' }    {' Processing origin' }
    {'PROCDATE'}    {'Sun Dec 18 21:44:3…'}    {' Processing date/t…'}
    {'PTFVERSN'}    {[                  5]}    {' Version of PTFSCI…'}
    {'PMASKPTH'}    {'/ptf/pos/archive/f…'}    {'pmasks/' / Pathnam…'}
    {'PMASKFIL'}    {'70sOn35s_pixmask_c…'}    {' Filename of pixel…'}
    {'SFLATPTH'}    {'/ptf/pos/sbx2/2011…'}    {'11/f11/c2/cal/p4/c…'}
    {'SFLATFIL'}    {'PTF_201110110000_i…'}    {'flat_t120000_u0000…'}
    {'SBIASPTH'}    {'/ptf/pos/sbx2/2011…'}    {'10/f2/c2/p13/v1/' …'}
    {'SBIASFIL'}    {'PTF_201110100000_i…'}    {'bias_t120000_u0000…'}

...

Note that the content of PMASKPTH, SFLATPTH, SBIASPTH, SBIASFIL is messed up. Not that I really need these particular fields, but the problem is carried on. In more detail:

>> AI.Header(find(strcmp(AI.Header,'PMASKPTH')),2)

ans =

  1×1 cell array

    {'/ptf/pos/archive/fallbackcal/pmasks/' / Pathname of pixel mask'}

>> AI.Header(find(strcmp(AI.Header,'PMASKPTH')),3)

ans =

  1×1 cell array

    {'pmasks/' / Pathname of pixel mask'}

>> AI.Header(find(strcmp(AI.Header,'SFLATPTH')),2)

ans =

  1×1 cell array

    {'/ptf/pos/sbx2/2011/10/11/f11/c2/cal/p4/cId62146/' / Pathname of super'}

>> AI.Header(find(strcmp(AI.Header,'SFLATPTH')),3)

ans =

  1×1 cell array

    {'11/f11/c2/cal/p4/cId62146/' / Pathname of super'}

>> AI.Header(find(strcmp(AI.Header,'SFLATFIL')),2)

ans =

  1×1 cell array

    {'PTF_201110110000_i_s_flat_t120000_u000062146_f11_p000000_c02.fits''}

>> AI.Header(find(strcmp(AI.Header,'SFLATFIL')),3)

ans =

  1×1 cell array

    {'flat_t120000_u000062146_f11_p000000_c02.fits''}

Note the pieces carried from the content field to the comment field, the doubled apices, etc.

[EastEriq]

Further processing of the images adds to the mangling: At a later stage we have

ORIGNAME= '/data/PTF_default_24854.fits' /  Filename as written by the camera   
FILENAME= 'PTF201110114398_11_o_24854.fits' /  Filename of delivered camera imag
PROCORIG= 'IPAC-PTF pipelines' /  Processing origin                             
PROCDATE= 'Sun Dec 18 21:44:36 2011' /  Processing date/time (Pacific time)     
PTFVERSN=                   5. /  Version of PTFSCIENCEPIPELINE program         
PMASKPTH= '/ptf/pos/archive/fallbackcal/pmasks/'' / Pathname of pixel mask' / pm
PMASKFIL= '70sOn35s_pixmask_chip2.trimmed.v4.fits' /  Filename of pixel mask    
SFLATPTH= '/ptf/pos/sbx2/2011/10/11/f11/c2/cal/p4/cId62146/'' / Pathname of su&'
CONTINUE  'per     '           / 11/f11/c2/cal/p4/cId62146/' / Pathname of super
LONGSTRN= 'OGIP 1.0'           / The HEASARC Long String Convention may be used.
COMMENT   This FITS file may contain long string keyword values that are        
COMMENT   continued over multiple keywords.  The HEASARC convention uses the &  
COMMENT   character at the end of each substring which is then continued        
COMMENT   on the next keyword which has the name CONTINUE.                      
SFLATFIL= 'PTF_201110110000_i_s_flat_t120000_u000062146_f11_p000000_c02.fits''' 
SBIASPTH= '/ptf/pos/sbx2/2011/10/10/f2/c2/p13/v1/'' / Pathname of super bias'   
SBIASFIL= 'PTF_201110100000_i_s_bias_t120000_u000050818_f00_p000000_c02.fits''' 

I think that the pattern is: "long mangled fields are split as CONTINUE (within reason), long comments are truncated"

[EastEriq]

Since the risk of breaking the reading of FITS file with modifications of the code is very high, I have created a branch FITSheaderWithLongStrings which needs to be vetted before merging. My first commit, https://github.com/EranOfek/AstroPack/commit/c219a71c74e4eb7c00f0cf0cc0d3f7168ccb98c1, modifies FITS.readHeader1 so that it deals correctly with single line header key-value-comment, in which value is a string which may be longer than 20 char, may contain slashes, and comments may be absent all together. Moreover, it fixes a small bug of a trailing space in comments.

Apparently, AstroImage(filename) makes use of this method, so the header should be retrieved correctly.