NaT values for time and forecast_reference_time when loading datasets in AzureML

[hannahbrown7]

When loading merged radar and MOGREPS-G data from azureml datasets, there appears to be an issue with how datetime values are handled (during the load process?), with a significant number of rows in columns presenting datetime information being set to NaT. This sometimes only affects the time or forecast_reference_time column, both or neither, there isn't an obvious pattern of when it occurs.

In total 1211952 / 2776212 rows contain both time and forecast reference time data.

This issue does not occur when loading the same data in on local machine/SPICE.

[hannahbrown7]

For each event, the number of rows which are unaffected (e.g. have datetime values for both time and forecast ref time), then number of rows where time, forecast reference time or both are set to NaT.

prd_merged_202002_storm_dennis Both time values unaffected: 189636 Time value is NaT: 78588 Forecast reference time value is NaT: 79032 Both time values are NaT: 0

prd_merged_202002_storm_ciara Both time values unaffected: 189636 Time value is NaT: 78588 Forecast reference time value is NaT: 79032 Both time values are NaT: 0

prd_merged_202010_nswws_amber_oct Both time values unaffected: 79032 Time value is NaT: 162924 Forecast reference time value is NaT: 0 Both time values are NaT: 78588

prd_merged_202012_nswws_amber_dec Both time values unaffected: 81252 Time value is NaT: 52824 Forecast reference time value is NaT: 26196 Both time values are NaT: 0

prd_merged_202102_nswws_amber_feb Both time values unaffected: 52836 Time value is NaT: 0 Forecast reference time value is NaT: 107952 Both time values are NaT: 26196

prd_merged_202110_nswws_amber_oct Both time values unaffected: 81696 Time value is NaT: 52392 Forecast reference time value is NaT: 52896 Both time values are NaT: 0

prd_merged_202112_storm_barra Both time values unaffected: 134088 Time value is NaT: 78588 Forecast reference time value is NaT: 79320 Both time values are NaT: 0

prd_merged_2022_storm_eunice_franklin Both time values unaffected: 189636 Time value is NaT: 79032 Forecast reference time value is NaT: 78588 Both time values are NaT: 0

prd_merged_202008_storm_francis Both time values unaffected: 108468 Time value is NaT: 52392 Forecast reference time value is NaT: 52836 Both time values are NaT: 0

prd_merged_202008_storm_ellen Both time values unaffected: 105672 Time value is NaT: 0 Forecast reference time value is NaT: 189708 Both time values are NaT: 78588