search

spedas / bleeding_edge

IDL-based Space Physics Environment Data Analysis Software (bleeding edge)
http://www.spedas.org
Other
7 stars 0 forks source link

Gap fill needed in SOSMAG/KOMPSAT summary plots? #107

Closed jameswilburlewis closed 10 months ago

jameswilburlewis commented 11 months ago

Vassilis is seeing some gaps in the SOSMAG summary plots. We should check data availability for any missing plots -- and if possible, try to determine the lag between the original processing date for the gap, and the date where data became available. We might need a scheme like we do for the THEMIS summary plots, where we have a nightly cron job that looks back 30 days and reprocesses that date's summary plots, to account for late-arriving data, calibration changes, etc.

nickssl commented 11 months ago

It turns out there is a problem with the ESA HAPI server. The real-time data is complete, but the calibrated data is cut off at a random time for many dates. I have informed David Fischer about this, and they will try to find and solve the problem. After they fix this issue on their end, we need to reprocess the plots.

Example of missing data:

Real time data for 2023-01-20, 1pm-2pm, exists: https://swe.ssa.esa.int/hapi/data?id=spase://SSA/NumericalData/GEO-KOMPSAT-2A/esa_gk2a_sosmag_1m&time.min=2023-01-20T13:00:00.000Z&time.max=2023-01-20T14:00:00.000Z&format=json

But the recalibrated data for the same time interval is empty: https://swe.ssa.esa.int/hapi/data?id=spase://SSA/NumericalData/GEO-KOMPSAT-2A/esa_gk2a_sosmag_recalib&time.min=2023-01-20T13:00:00.000Z&time.max=2023-01-20T14:00:00.000Z&format=json

nickssl commented 11 months ago

Unrelated to the above, David Fischer informed me that ESA is currently working on including the data of the Korean particle instrument on GK-2A.

nickssl commented 10 months ago

We are not sure when the ESA server problem will be fixed. We changed the summary plots code so that, from now on, only real-time data will be used. We reprocessed all plots from 2022-12-26 to 2023-10-16. They now use real-time data and have no gaps. It seems that earlier dates do not have cut-off problems. The title of the summary plot indicates which data set was used, real-time or recalibrated. In the future, if ESA solves the problem with the recalibrated data cut-offs, we will have to reprocess all plots.