GFTS: tm files missing

[nicHoch]

with the latest rerun (/data/stix/out/test/run_08_30) of all TM data we good some daily HK files that are short in data entries not the usual 1350 time bins ...

here are some candidate: 2021-01-25, 2021-01-27, 2021-03-02, 2021-03-03, 2021-03-04, 2021-03-06, 2021-03-07

i where looking into the time period of: 2021-01-25 to 2021-01-27

p_aux = Product(Path('/data/stix/out/test/run_08_30/L2/2021/01/27/AUX/solo_L2_stix-aux-auxiliary_20210127_V01.fits'))

Hualin tracked back all input files for this day as follows:

/* 1 */
{
    "_id" : 499,
    "filename" : "LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.108.BApn@2021.025.05.00.02.125.1.xml"
}
/* 2 */
{
    "_id" : 500,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.21.737.KGwY@2021.025.06.00.01.801.1.xml"
}
/* 3 */
{
    "_id" : 501,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.49.03.010.NwWJ@2021.025.06.00.02.873.1.xml"
}
/* 4 */
{
    "_id" : 502,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.50.27.005.VXSW@2021.025.13.00.01.391.1.xml"
}
/* 5 */
{
    "_id" : 503,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.45.219.bClG@2021.025.13.00.01.423.1.xml"
}
/* 6 */
{
    "_id" : 504,
    "filename" : "LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.111.JuWR@2021.025.17.00.00.570.1.xml"
}
/* 7 */
{
    "_id" : 506,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.49.03.014.hVmx@2021.026.06.00.01.481.1.xml"
}
/* 8 */
{
    "_id" : 507,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.21.740.irPv@2021.026.06.00.03.605.1.xml"
}
/* 9 */
{
    "_id" : 508,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.45.221.eGzo@2021.026.13.00.00.987.1.xml"
}
/* 10 */
{
    "_id" : 509,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.50.27.009.BeOM@2021.026.13.00.02.037.1.xml"
}
/* 11 */
{
    "_id" : 510,
    "filename" : "LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.117.OLQd@2021.026.17.00.00.614.1.xml"
}
/* 12 */
{
    "_id" : 511,
    "filename" : "LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.120.emxu@2021.027.05.00.02.792.1.xml"
}
/* 13 */
{
    "_id" : 512,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.49.03.016.wGWL@2021.027.06.00.01.725.1.xml"
}
/* 14 */
{
    "_id" : 513,
    "filename" : "LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.21.742.YzbD@2021.027.06.00.02.827.1.xml"
}
/* 15 */
{
    "_id" : 514,
    "filename" : "LTP03_noon_req_BatchRequest.PktTmRaw.SOL.0.2021.027.09.49.07.061.vXBE@2021.027.13.00.00.597.1.xml"
}
/* 16 */
{
    "_id" : 528,
    "filename" : "LTP03_noon_req_BatchRequest.PktTmRaw.SOL.0.2021.027.09.49.07.067.bqKi@2021.029.13.00.01.213.1.xml"
}
/* 17 */
{
    "_id" : 529,
    "filename" : "LTP03_morning_req_BatchRequest.PktTmRaw.SOL.0.2021.027.09.49.00.923.ToEs@2021.030.06.00.01.581.1.xml"
}

some of this files are missing on the GFTS server!

LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.111.JuWR@2021.025.17.00.00.570.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.49.03.014.hVmx@2021.026.06.00.01.481.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.21.740.irPv@2021.026.06.00.03.605.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.45.221.eGzo@2021.026.13.00.00.987.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.50.27.009.BeOM@2021.026.13.00.02.037.1.xml
LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.117.OLQd@2021.026.17.00.00.614.1.xml
LTP03_daily_BatchRequest.PktTmRaw.SOL.0.2021.005.12.30.26.120.emxu@2021.027.05.00.02.792.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.15.49.03.016.wGWL@2021.027.06.00.01.725.1.xml
LTP03_BatchRequest.PktTmRaw.SOL.0.2021.013.16.01.21.742.YzbD@2021.027.06.00.02.827.1.xml

but as the platform has injected them at some time in the past it looks like that this file has been revoked somehow.

this need investigation with ESA

[nicHoch]

@samaloney @drhlxiao any helpful inputs are welcome as i have no idea where to start. it is very hard to search in the system for something that is not around

[samaloney]

I'd say more likely renamed, removed or deleted by accident ....

[nicHoch]

after copy of all possible sources (pup23, home dirs ...) most of the files are back now

only missing files

BatchRequest.PktTmRaw.SOL.0.2019.177.14.48.14.261.wbRb@2019.177.14.48.15.239STIX_SVT1a_2019-06-26T144814.261.xml
BatchRequest.PktTmRaw.SOL.0.2019.177.14.35.25.165.VzOE@2019.177.14.35.25.788STIX_SVT1a_2019-06-26T143525.165.xml
BatchRequest.PktTmRaw.SOL.0.2019.168.13.33.34.683.PUei@2019.168.13.33.35.542STIX_SVT1a_2019-06-17T133334.683.xml

@drhlxiao is it possible that this are files from the ground unit?

[nicHoch]

found and restored