During the processing of daily fits files (HK, QL) the files are already available on pub99 (also datacenter?) as soon some data was processed.
Normally it take up to 3 days until a daily file is complete as the data comes in 6h batches.
If a user downloads that files with just half a day content it changes again on the server some time later but without any updated version number.
So it is a bit risky not to inform the user about the incompleteness of the data.
The SO meta data standard defines a way how to label "not fully ready" data.
The version of the file, formatted as capital V + 2-character incremental number padded with 0. Again, version numbers in L0 files are more flexible. For Low latency data, a “C”, “I” or “U” shall be appended to the version field for files that are known to be complete, incomplete or with an unknown level of completeness, respectively. See [LLCDFICD] and [LLFITSICD] for details.
We should implement this:
All automatically processed files will get the "I" label first-hand. While publishing to ESA (SOAR) we will revoke the "incomplete" label.
an example file name would be:
solo_L1_stix-ql-lightcurve_20230203_V01I.fits
we could also use that waiting time from processing to publishing to update spice pointing information and therefore start processing incoming TM right away: see also: #346
nicHoch
commented
10 months ago
During the processing of daily fits files (HK, QL) the files are already available on pub99 (also datacenter?) as soon some data was processed. Normally it take up to 3 days until a daily file is complete as the data comes in 6h batches.
If a user downloads that files with just half a day content it changes again on the server some time later but without any updated version number.
So it is a bit risky not to inform the user about the incompleteness of the data.
The SO meta data standard defines a way how to label "not fully ready" data.
We should implement this:
All automatically processed files will get the "I" label first-hand. While publishing to ESA (SOAR) we will revoke the "incomplete" label.
an example file name would be:
solo_L1_stix-ql-lightcurve_20230203_V01I.fits
we could also use that waiting time from processing to publishing to update spice pointing information and therefore start processing incoming TM right away: see also: #346