Problems with the L1 fits files on pub099

[paolomassa]

I tried to use the L1 fits files on pub099 for reconstructing the images, but I found the following issues in the 'stix_l1_read' procedure:

1) line 10 doesn't work properly; I think that changing "energy = mrdfits(sci_file, 3, energy_header)" with "energy = mrdfits(sci_file, 4, energy_header)" should fix this. 2) line 34: "cccc" has dimension [32, 8, 32, 45] instead of [32, 12, 32, 45]. Was the content of the 4 small pixels removed in these fits files? 3) if I fix 1) and 2) the routine doesn't crash anymore, but it gives unreasonable values for the count rates. I suspect that the problem is in the 'triggers32' variable (defined in line 21 of 'stix_l1_read'). Indeed, the entries of this variable contain values close to -9.8e18, which are quite strange.

If I have understood correctly the problems in 1) and 2) the modifications to be done are very simple. However, I don't know how to fix 3).

[samaloney]

So for 1. we added an additional fits header I would suggest using the extension names rather then number.

2. seem like a bug could you give me the exact file name?
3. is a recurring issue idl can't read fits standard file

I think an general solution is that we create a six_reader file which can read / correct the fits files and return expected structures

[paolomassa]

Thanks for your clarifications. 1) seems to be fixed using the extension names; 2) the file I was trying to read is

solo_L1_stix-sci-xray-cpd-1178428688_20200607T213709_20200607T215208_V01_49155.fits

that I found at

http://pub099.cs.technik.fhnw.ch/data/fits/L1/2020/06/07/SCI/

3) do you have any idea on how to read these fits files in IDL in order to avoid the problem of the trigger values?

Thanks again!

[samaloney]

So just had a look an the request (https://pub023.cs.technik.fhnw.ch/view/ior/overview/225) it was only for large pixels so that's why the small pixels are not in the data.

[samaloney]

Ok so if you look in the fits headers for that data you will see this.

data = mrdfits('/var/folders/r1/6cz6bl3156s18tp0wqggp6mc0000gn/T/astropy-download-47857-_p1_7y99', 2, header)

TFORM8  = '16K     '
TZERO8  =  9223372036854775808

so the correct triggers are obtained by data.triggers+9223372036854775808

so for the first time

(data.triggers + 9223372036854775808)[*,0]
                126975                118783                139263                139263                155647                139263                110591                172031
                110591                102399                 86015                126975                221183                 94207                110591                102399

The exact value of TZERO8 may change I'm don't know why mrdfits isn't applying the proper corrections.

[paolomassa]

Ok thanks! I can implement a correction by adding the quantity in saved in TZERO8. Maybe not the best way to do it, but at least it should work

[paolomassa]

I've made some tests with the correction obtained by summing TZERO8. I think it doesn't work properly, indeed:

1) now the values in 'triggers32' (defined in 'stix_l1_read', line 21) are larger than the ones we had from the "old" L1 fits files of the same event 2) because of 1), the live time values contained in 'live32'(defined in 'stix_l1_read', line 23) are negative when I read the L1 fits file of the background

Maybe, I'll try some other IDL routines for reading the fits file

[samaloney]

What values do you expect the triggers for the first time interval to be? It could be the compression scheme bug fix issue which is known i4Ds/STIXCore#125

[paolomassa]

When I used the "old" science fits file that is on the STIX website, the values in 'triggers32' range between 1055 and 3391. Instead, if I use the fits file on pub099 and I correct by adding TZERO8, the values in 'triggers32' range between 2.4e5 and 2.7e5.

[samaloney]

Please have a look at the most recent processed file here http://pub099.cs.technik.fhnw.ch/data/fits_new/L1/2020/06/07/SCI/solo_L1_stix-sci-xray-cpd-1178428688_20200607T213709_20200607T215208_V01_49155.fits

I looked at the triggers and get the same values as you mentioned (once the TZERO value is taken into account) also not note the count and trigger errors are calculated and the primary header has spacecraft position info.

[paolomassa]

Thanks! I've checked and if I use this new fits file (an I correct for TZERO) I get exactly the same triggers as with the corresponding fits file on the website

[grazwegian]

It seems mrdfits might need TSCAL to be set to apply TZERO would it be possible to make a test file with TSCAL8 = 1 to see if we could read them correctly automatically?

from https://idlastro.gsfc.nasa.gov/mrdfits.html :

• Automatic mapping into IDL unsigned data types (/UNSIGNED) when the appropriate BSCALE(TSCAL) and BZERO(TZERO) keywords appear in the header

[samaloney]

Yea so there is a comment in the mrdfits code that suggests its was fixed for uint64 but data but doesn't seem to be the case works for 16 and 32 bit so we might be ok I think the reason I changed to 64bit was because of a roll over issue the triggers

; V2.6 February 15, 2002 Fix error in handling unsigned numbers, $ ; and 64 bit unsigneds. (Thanks to Stephane Beland)

python

<QTable length=4>
  d      t         c
  dm     ds        ct
uint32 uint16    uint64
------ ------ -----------
     1      0  8589934592
     2     10 17179869184
     3     20 34359738368
     4     30 68719476736

ColDefs(
    name = 'd'; format = 'J'; unit = 'dm'; bscale = 1.0; bzero = 2147483648
    name = 't'; format = 'I'; unit = 'ds'; bscale = 1.0; bzero = 32768
    name = 'c'; format = 'K'; unit = 'count'; bscale = 1.0; bzero = 9223372036854775808
)

ssw

data = mrdfits('test3.fits', 1, header, /un)
MRDFITS: Binary table.  3 columns by  4 rows.
IDL> data
[
    {
        "D": 1,
        "T": 0,
        "C": -9223372028264841216
    },
    {
        "D": 2,
        "T": 10,
        "C": -9223372019674906624
    },
    {
        "D": 3,
        "T": 20,
        "C": -9223372002495037440
    },
    {
        "D": 4,
        "T": 30,
        "C": -9223371968135299072
    }
]

[samaloney]

@paolomassa could you check what version of mrdfits you have mrdfits(/version)

[paolomassa]

I've checked and I'm using this version:

MRDFITS: Version 2.23 April 24, 2014

[samaloney]

So it seem if we upgrade (we can check in software and warn the users) mrdfits from here and I set bscale to 1 we should be able to read the fits files in both IDL and python.