Short-term Forecast Data issue

[danangmassandy]

There is a weird data of short-term forecast from zarr as below: csv:

json:

{
    "metadata": {
        "start_date": "2024-10-15T00:00:00+00:00",
        "end_date": "2024-10-15T23:59:59+00:00",
        "dataset": [
            {
                "provider": "Tomorrow.io",
                "attributes": {
                    "total_rainfall": {
                        "units": "mm",
                        "longname": "Total Rainfall"
                    },
                    "max_temperature": {
                        "units": "°C",
                        "longname": "Max Temperature"
                    },
                    "min_temperature": {
                        "units": "°C",
                        "longname": "Min Temperature"
                    },
                    "precipitation_probability": {
                        "units": "%",
                        "longname": "Precipitation Probability"
                    }
                }
            }
        ]
    },
    "results": [
        {
            "geometry": {
                "type": "Point",
                "coordinates": [
                    34.4635,
                    2.7018
                ]
            },
            "data": [
                {
                    "datetime": "2024-10-15T00:00:00Z",
                    "values": {
                        "total_rainfall": 1.3992289326878734e+306,
                        "max_temperature": null,
                        "min_temperature": null,
                        "precipitation_probability": null
                    }
                }
            ]
        }
    ]
}

[danangmassandy]

We should also check whether empty data is correctly stored in zarr, not as zero. The empty data can come from lat/lon pair that does not belong to any Grid.

2024-10-14,3.5625,33.4093,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.4456,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.4820,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.5184,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.5547,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.5911,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.6274,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.6638,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.7001,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.7365,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.7728,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.8092,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.8455,0.0000,0.0000,0.0000
2024-10-14,3.5625,33.8819,0.0000,0.0000,0.0000

[danangmassandy]

Update on 22 Oct 2024, the data seems normal for given point:

{
    "metadata": {
        "start_date": "2024-10-22T00:00:00+00:00",
        "end_date": "2024-11-04T23:59:59+00:00",
        "dataset": [
            {
                "provider": "Tomorrow.io",
                "attributes": {
                    "total_rainfall": {
                        "units": "mm",
                        "longname": "Total Rainfall"
                    },
                    "max_temperature": {
                        "units": "°C",
                        "longname": "Max Temperature"
                    }
                }
            }
        ]
    },
    "results": [
        {
            "geometry": {
                "type": "Point",
                "coordinates": [
                    34.4635,
                    2.7018
                ]
            },
            "data": [
                {
                    "datetime": "2024-10-22T00:00:00Z",
                    "values": {
                        "total_rainfall": 23.59,
                        "max_temperature": 21.25
                    }
                },
                {
                    "datetime": "2024-10-23T00:00:00Z",
                    "values": {
                        "total_rainfall": 27.165,
                        "max_temperature": 22.13
                    }
                },
                {
                    "datetime": "2024-10-24T00:00:00Z",
                    "values": {
                        "total_rainfall": 27.497,
                        "max_temperature": 26.94
                    }
                },
                {
                    "datetime": "2024-10-25T00:00:00Z",
                    "values": {
                        "total_rainfall": 4.78,
                        "max_temperature": 0.0
                    }
                },
                {
                    "datetime": "2024-10-26T00:00:00Z",
                    "values": {
                        "total_rainfall": 9.346097902069077e-307,
                        "max_temperature": 20.19
                    }
                },
                {
                    "datetime": "2024-10-27T00:00:00Z",
                    "values": {
                        "total_rainfall": 8.375,
                        "max_temperature": 20.5
                    }
                },
                {
                    "datetime": "2024-10-28T00:00:00Z",
                    "values": {
                        "total_rainfall": 4.16,
                        "max_temperature": 27.0
                    }
                },
                {
                    "datetime": "2024-10-29T00:00:00Z",
                    "values": {
                        "total_rainfall": 0.0,
                        "max_temperature": 26.0
                    }
                },
                {
                    "datetime": "2024-10-30T00:00:00Z",
                    "values": {
                        "total_rainfall": 10.662,
                        "max_temperature": 0.0
                    }
                },
                {
                    "datetime": "2024-10-31T00:00:00Z",
                    "values": {
                        "total_rainfall": 0.0,
                        "max_temperature": 24.0
                    }
                },
                {
                    "datetime": "2024-11-01T00:00:00Z",
                    "values": {
                        "total_rainfall": 17.934,
                        "max_temperature": 26.0
                    }
                },
                {
                    "datetime": "2024-11-02T00:00:00Z",
                    "values": {
                        "total_rainfall": 5.862,
                        "max_temperature": 30.0
                    }
                },
                {
                    "datetime": "2024-11-03T00:00:00Z",
                    "values": {
                        "total_rainfall": 5.634,
                        "max_temperature": 0.0
                    }
                },
                {
                    "datetime": "2024-11-04T00:00:00Z",
                    "values": {
                        "total_rainfall": 0.0,
                        "max_temperature": 26.5
                    }
                }
            ]
        }
    ]
}

[danangmassandy]

Reopening this ticket, I see below data from the API:

Suggestion: During ingestion, we could try to filter out this values.