i4Ds / STIXCore

STIX Core functionalities
BSD 3-Clause "New" or "Revised" License
3 stars 3 forks source link

aux file batch mode: will loose entire batch in case of spice error #340

Closed nicHoch closed 1 year ago

nicHoch commented 1 year ago

in the preprocessing of the AUX file generation after returning from IDL the batch loop needs an error handler on file level.

2023-04-12 02:41:31,926 calling IDL for hk files:
2023-04-12 02:41:31,926     /data/stix/out/fits_v1.0.0/L1/2023/02/12/HK/solo_L1_stix-hk-maxi_20230212_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/27/HK/solo_L1_stix-hk-maxi_20230227_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/11/HK/solo_L1_stix-hk-maxi_20230211_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/26/HK/solo_L1_stix-hk-maxi_20230226_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/19/HK/solo_L1_stix-hk-maxi_20230219_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/22/HK/solo_L1_stix-hk-maxi_20230222_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/25/HK/solo_L1_stix-hk-maxi_20230225_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/28/HK/solo_L1_stix-hk-maxi_20230228_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/09/HK/solo_L1_stix-hk-maxi_20230209_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/02/HK/solo_L1_stix-hk-maxi_20230202_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/13/HK/solo_L1_stix-hk-maxi_20230213_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/15/HK/solo_L1_stix-hk-maxi_20230215_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/02/04/HK/solo_L1_stix-hk-maxi_20230204_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/01/HK/solo_L1_stix-hk-maxi_20230401_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/08/HK/solo_L1_stix-hk-maxi_20230408_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/05/HK/solo_L1_stix-hk-maxi_20230405_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/07/HK/solo_L1_stix-hk-maxi_20230407_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/06/HK/solo_L1_stix-hk-maxi_20230406_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/03/HK/solo_L1_stix-hk-maxi_20230403_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/02/HK/solo_L1_stix-hk-maxi_20230402_V01.fits
2023-04-12 02:41:31,927     /data/stix/out/fits_v1.0.0/L1/2023/04/04/HK/solo_L1_stix-hk-maxi_20230404_V01.fits
2023-04-12 03:04:44,991 returning from IDL
2023-04-12 03:04:44,992 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/12/HK/solo_L1_stix-hk-maxi_20230212_V01.fits'
2023-04-12 03:04:46,576 No soops found for time: 2023-02-12 00:00:54.140061 - 2023-02-12 23:59:49.541111
2023-04-12 03:04:46,662 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/27/HK/solo_L1_stix-hk-maxi_20230227_V01.fits'
2023-04-12 03:04:48,217 No soops found for time: 2023-02-27 00:00:44.785339 - 2023-02-27 23:59:40.177517
2023-04-12 03:04:48,301 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/11/HK/solo_L1_stix-hk-maxi_20230211_V01.fits'
2023-04-12 03:04:49,847 No soops found for time: 2023-02-11 00:00:54.805505 - 2023-02-11 23:59:50.136547
2023-04-12 03:04:49,931 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/26/HK/solo_L1_stix-hk-maxi_20230226_V01.fits'
2023-04-12 03:04:51,478 No soops found for time: 2023-02-26 00:00:45.384045 - 2023-02-26 23:59:40.786233
2023-04-12 03:04:51,561 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/19/HK/solo_L1_stix-hk-maxi_20230219_V01.fits'
2023-04-12 03:04:53,097 No soops found for time: 2023-02-19 00:00:49.655100 - 2023-02-19 23:59:45.046656
2023-04-12 03:04:53,179 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/22/HK/solo_L1_stix-hk-maxi_20230222_V01.fits'
2023-04-12 03:04:54,724 No soops found for time: 2023-02-22 00:00:47.730467 - 2023-02-22 23:59:43.132656
2023-04-12 03:04:54,808 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/25/HK/solo_L1_stix-hk-maxi_20230225_V01.fits'
2023-04-12 03:04:56,341 No soops found for time: 2023-02-25 00:00:45.924782 - 2023-02-25 23:59:41.386968
2023-04-12 03:04:56,424 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/28/HK/solo_L1_stix-hk-maxi_20230228_V01.fits'
2023-04-12 03:04:57,964 No soops found for time: 2023-02-28 00:00:44.179217 - 2023-02-28 23:59:39.581405
2023-04-12 03:04:58,047 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/09/HK/solo_L1_stix-hk-maxi_20230209_V01.fits'
2023-04-12 03:04:59,590 No soops found for time: 2023-02-09 00:00:56.066624 - 2023-02-09 23:59:51.478268
2023-04-12 03:04:59,674 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/02/HK/solo_L1_stix-hk-maxi_20230202_V01.fits'
2023-04-12 03:05:01,218 No soops found for time: 2023-02-02 00:01:01.030385 - 2023-02-02 23:59:56.361663
2023-04-12 03:05:01,302 IDL postprocessing HK file: b'/data/stix/out/fits_v1.0.0/L1/2023/02/13/HK/solo_L1_stix-hk-maxi_20230213_V01.fits'
2023-04-12 03:05:01,692 error
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/stixcore/STIXCore/stixcore/processing/L1toL2.py", line 106, in process_type
    all_files.extend(idlprocessor.process())
  File "/home/stixcore/STIXCore/stixcore/processing/sswidl.py", line 205, in process
    files.extend(task.run(self.fits_processor))
  File "/home/stixcore/STIXCore/stixcore/processing/sswidl.py", line 153, in run
    self._results = self.postprocessing(results, fits_processor)
  File "/home/stixcore/STIXCore/stixcore/products/level2/housekeepingL2.py", line 289, in postprocessing
    orient, dist, car, heeq = Spice.instance.get_auxiliary_positional_data(date=d)
  File "/home/stixcore/STIXCore/stixcore/ephemeris/manager.py", line 303, in get_auxiliary_positional_data
    cmat, _ = spiceypy.ckgp(SOLAR_ORBITER_STIX_ILS_FRAME_ID, sc, 1.0, 'SOLO_SUN_RTN')
  File "/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/spiceypy/spiceypy.py", line 138, in with_errcheck
    res = f(*args, **kwargs)
  File "/home/stixcore/STIXCore/venv/lib/python3.9/site-packages/spiceypy/spiceypy.py", line 158, in wrapper
    raise NotFoundError(
spiceypy.utils.exceptions.NotFoundError: Spice returns not found for function: ckgp
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/stixcore/STIXCore/stixcore/processing/L1toL2.py", line 65, in process_fits_files
    new_files = job.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
spiceypy.utils.exceptions.NotFoundError: Spice returns not found for function: ckgp

the loop stops after the error and did not go on with the next file

samaloney commented 1 year ago

Yea so there seems to be something wrong with the spice kernels but only for on specific time 2023-02-13T01:58:13.479. We can catch these error with something like this and put nans in files to stop processing from crashing

with spicey.no_found_check():
     for t in times:
         et = spicey.utc2et(t.isot)
         sc = spicey.sce2t(-144, et)
         try:
             cmat, *_ = spicey.ckgp(-144851, sc, 1000.0, 'SOLO_SUN_RTN')
             print(t, cmat)
         except Exception as e:
             print(t, e)

Minimal code to reproduce the problem

import spiceypy as spicey
spicey.furnsh('solo_ANC_soc-flown-mk.tm')
et = spicey.utc2et('2023-02-13T01:58:13.479')
sc = spicey.sce2t(-144, et)
cmat, *_ = spice.ckgp(-144851, sc, 1.0, 'SOLO_SUN_RTN')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[82], line 1
----> 1 cmat, *_ = spice.ckgp(-144851, sc, 1.0, 'SOLO_SUN_RTN')

NameError: name 'spice' is not defined

In [83]: cmat, *_ = spicey.ckgp(-144851, sc, 1.0, 'SOLO_SUN_RTN')
---------------------------------------------------------------------------
SpiceNOFRAMECONNECT                       Traceback (most recent call last)
<ipython-input-83-c8e1f3a98c30> in ?()
----> 1 cmat, *_ = spicey.ckgp(-144851, sc, 1.0, 'SOLO_SUN_RTN')

~/.virtualenvs/stixpy/lib/python3.11/site-packages/spiceypy/spiceypy.py in ?(*args, **kwargs)
    138             res = f(*args, **kwargs)
    139             check_for_spice_error(f)
    140             return res
    141         except BaseException:
--> 142             raise

~/.virtualenvs/stixpy/lib/python3.11/site-packages/spiceypy/spiceypy.py in ?(f)
    118         explain = getmsg("EXPLAIN", 100).strip()
    119         long = getmsg("LONG", 1841).strip()
    120         traceback = qcktrc(200)
    121         reset()
--> 122         raise dynamically_instantiate_spiceyerror(
    123             short=short, explain=explain, long=long, traceback=traceback
    124         )

SpiceNOFRAMECONNECT:
================================================================================

Toolkit version: CSPICE_N0067

SPICE(NOFRAMECONNECT) --

At epoch 7.2952556266407E+08 TDB (2023 FEB 13 01:59:22.664 TDB), there is insufficient information available to transform from reference frame -144991 (SOLO_SUN_RTN) to reference frame -144001 (SOLO_FOF). Frame SOLO_SUN_RTN could be transformed to frame 1 (J2000). Frame SOLO_FOF could be transformed to frame -144000 (SOLO_SRF). The latter is a CK frame; a CK file containing data for instrument or structure -144000 at the epoch shown above, as well as a corresponding SCLK kernel, must be loaded in order to use this frame. Failure to find required CK data could be due to one or more CK files not having been loaded, or to the epoch shown above lying within a coverage gap or beyond the coverage bounds of the loaded CK files. It is also possible that no loaded CK file has required angular velocity data for the input epoch, even if a loaded CK does have attitude data for that epoch. You can use CKBRIEF with the -dump option to display coverage intervals of a CK file.

ckgp_c --> CKGP --> REFCHG

================================================================================
samaloney commented 1 year ago

So ESA confirmed this is due to TM gaps a list of such TM gaps is maintained on the SO wiki: https://issues.cosmos.esa.int/solarorbiterwiki/display/SOSP/Gaps+in+Attitude+Data

samaloney commented 1 year ago

Closed by #356