Salvage more data

[skymandr]

📋 Info

If a file fails due to a raised error in the current pipeline, the entire file is invalidated and put back into the queue, without any data being saved. The most common reason for this to happen is if platform data is missing for the time interval in question. This happens even if some of the data in a file might be matchable, thereby potentially leading to some data loss.

For CCD data, which is partitioned by hour, the maximum data loss is one hour, which we have hitherto deemed acceptable. PM data, on the other hand, is partitioned by day, so the potential data loss there is one day. This may lead to problems in a later stage, when L1B data for CCD and PM are matched. If PM data is missing for a day, but CCD data is available for some hours of that day, this will lead to this stage unnecessarily invalidating combined CCD/PM data (L1C?).

I suggest that we:

• update the covers check so that the error is not re-raised immediately
• try to make an interpolation for some of the data, fill with nan where appropriate – if none are matchable, re-raise
• otherwise proceed to HTR (use a similar scheme)
• save data
• raise new error, with statistics of how many records where failed and how many saved, so that we get an overview and message is put back in queue (and eventually DLQ)

🏁 DoD

• [ ] update cover check
• [ ] interpolate or nan fill
• [ ] check if any data is matched, raise if not
• [ ] proceed with HTR
• [ ] save as usual, but raise new error after

[skymandr]

This is actually done, I think.