SOIT error for specific date range

[danielmwatkins]

I ran into a new error with SOIT while running the IFT-Pipeline. It worked perfectly for the same region for all years except one. Using the this specification file

,location,center_lat,center_lon,top_left_lat,top_left_lon,lower_right_lat,lower_right_lon,left_x,right_x,lower_y,top_y,startdate,enddate
0,fram_strait-20130331-20130501,76.0015,-18.4315,83.7087,-23.7427,68.3809,-16.9041,247326.41049684677,1115678.4104968468,-2089839.4681516434,-635759.4681516434,2013-03-31,2013-05-01

the IFT is able to download all the imagery without an issue, but it runs into a bug in SOIT. Perhaps there is missing data from the satellite overpass website?

[danielmwatkins]

Here's the traceback error: Traceback (most recent call last): File "/usr/local/bin/pass_time_cylc.py", line 482, in <module> main() File "/usr/local/bin/pass_time_cylc.py", line 477, in main get_passtimes(**vars(args)) File "/usr/local/bin/pass_time_cylc.py", line 136, in get_passtimes aqua_tleline1 = aquaData[aqua_i]["TLE_LINE1"] IndexError: list index out of range 2024-06-28T10:06:52-04:00 CRITICAL - failed/ERR

[cpaniaguam]

@mmwilhelmus @danielmwatkins @mkkim400 It appears that this one error is not due to missing data. The program fails at the following block:

        # Update aqua index. If the epoch of the current TLE is more than a day away from the area of interest,
        # move to a newer TLE. This ensures the TLE epoch remains close to the date of the pass, reducing error in the orbital
        # mechanics algorithm.
        while ((t0 - aqua.epoch) > 1):
            aqua_i += 1
            aqua_tleline1 = aquaData[aqua_i]['TLE_LINE1']
            aqua_tleline2 = aquaData[aqua_i]['TLE_LINE2']
            aqua = EarthSatellite(aqua_tleline1, aqua_tleline2, 'AQUA', ts)

The code iterates through aquaData but exhausts all entries without meeting an exit condition. In such cases, retaining the original aqua value before attempting an update might be a solution. What do you think?

A similar block follows for terra, and presumably, the same solution would apply there as well.

[danielmwatkins]

To clarify, is the original aqua.epoch value a time stamp or an index? Could it be that the issue is that the time is slightly before the original value, i.e., should it be looking at the absolute value of the time difference instead?

[cpaniaguam]

To clarify, is the original aqua.epoch value a time stamp or an index?

Here is the state of the relevant variables right after they are defined for the first time:

(Pdb) today
['03', '31', '2013']
(Pdb) t0
<Time tt=2456382.5007775924>
(Pdb) aqua.epoch
<Time tt=2456383.336764223>
(Pdb) t0 - aqua.epoch
np.float64(-0.8359866301218668)
(Pdb) t0 - terra.epoch
np.float64(-0.183604130215115)

The program crashes when processing the last day in the date range:

(Pdb) today
['05', '01', '2013']

Here is the state of the variables for every iteration in the while loop for this day:

aqua_i=01 t0=<Time tt=2456384.5007775924> aqua.epoch=<Time tt=2456383.336764223> t0-aqua.epoch > 1=np.True_
aqua_i=02 t0=<Time tt=2456385.5007775924> aqua.epoch=<Time tt=2456384.3667170326> t0-aqua.epoch > 1=np.True_
aqua_i=03 t0=<Time tt=2456387.5007775924> aqua.epoch=<Time tt=2456385.6576076723> t0-aqua.epoch > 1=np.True_
aqua_i=04 t0=<Time tt=2456388.5007775924> aqua.epoch=<Time tt=2456387.177644643> t0-aqua.epoch > 1=np.True_
aqua_i=05 t0=<Time tt=2456389.5007775924> aqua.epoch=<Time tt=2456388.412880712> t0-aqua.epoch > 1=np.True_
aqua_i=06 t0=<Time tt=2456391.5007775924> aqua.epoch=<Time tt=2456389.6528376127> t0-aqua.epoch > 1=np.True_
aqua_i=07 t0=<Time tt=2456393.5007775924> aqua.epoch=<Time tt=2456392.001727293> t0-aqua.epoch > 1=np.True_
aqua_i=08 t0=<Time tt=2456394.5007775924> aqua.epoch=<Time tt=2456393.0277301623> t0-aqua.epoch > 1=np.True_
aqua_i=09 t0=<Time tt=2456395.5007775924> aqua.epoch=<Time tt=2456394.195131843> t0-aqua.epoch > 1=np.True_
aqua_i=10 t0=<Time tt=2456397.5007775924> aqua.epoch=<Time tt=2456395.919763922> t0-aqua.epoch > 1=np.True_
aqua_i=11 t0=<Time tt=2456398.5007775924> aqua.epoch=<Time tt=2456396.9427232523> t0-aqua.epoch > 1=np.True_
aqua_i=12 t0=<Time tt=2456399.5007775924> aqua.epoch=<Time tt=2456397.9773492725> t0-aqua.epoch > 1=np.True_
aqua_i=13 t0=<Time tt=2456400.5007775924> aqua.epoch=<Time tt=2456398.9996751524> t0-aqua.epoch > 1=np.True_
aqua_i=14 t0=<Time tt=2456401.5007775924> aqua.epoch=<Time tt=2456400.3018955127> t0-aqua.epoch > 1=np.True_
aqua_i=15 t0=<Time tt=2456406.5007775924> aqua.epoch=<Time tt=2456404.974249553> t0-aqua.epoch > 1=np.True_
aqua_i=16 t0=<Time tt=2456408.5007775924> aqua.epoch=<Time tt=2456407.306328273> t0-aqua.epoch > 1=np.True_
aqua_i=17 t0=<Time tt=2456409.5007775924> aqua.epoch=<Time tt=2456408.3321948927> t0-aqua.epoch > 1=np.True_
aqua_i=18 t0=<Time tt=2456411.5007775924> aqua.epoch=<Time tt=2456409.6577914828> t0-aqua.epoch > 1=np.True_
aqua_i=19 t0=<Time tt=2456412.5007775924> aqua.epoch=<Time tt=2456411.020555853> t0-aqua.epoch > 1=np.True_
aqua_i=20 t0=<Time tt=2456413.5007775924> aqua.epoch=<Time tt=2456412.3162051626> t0-aqua.epoch > 1=np.True_

As you can see, t0 - aqua,epoch is always greater than 1 for 2013-05-01 so there is no exit condition.

... should it be looking at the absolute value of the time difference instead?

In this case it doesn't matter as the difference t0 - aqua.epoch is always positive ¯\_(ツ)_/¯.

[danielmwatkins]

I suggest the following solution: When a timestamp is missing, use the date e.g. YYYY-MM-DD 0:00:00 Real data won't have exactly 0 for hour minute second, so we can flag those times later in the pipeline.

[danielmwatkins]

In the pipeline later on, we can use interpolation to check for outliers, and fill the times with 0's.