The Thames Water history API occasionally returns an empty response when it really should throw an error. This is a problem because an empty response is how we previously checked to see if all records had been fetched from the API. The consequence was if this occurred, the history of discharges would erroneously stop at a random point in time. I've now fixed this by instead iterating until the API responses have a datetime that is older than a certain threshold date (1/5/2023). If an empty data table is returned, an exception is raised. Consequently, it is much harder for PooPy to inadvertently generate misleading historical records.
This patch has introduced an unfortunate edgecase however. If there number of records returned for a history is an exact integer multiple of the API request limit (as of 29/2/24 this is 1000) then an exception will be raised in error. It is not clear how this can sensibly be fixed as really the API shouldn't be throwing erroneous empty responses!
The Thames Water history API occasionally returns an empty response when it really should throw an error. This is a problem because an empty response is how we previously checked to see if all records had been fetched from the API. The consequence was if this occurred, the history of discharges would erroneously stop at a random point in time. I've now fixed this by instead iterating until the API responses have a datetime that is older than a certain threshold date (1/5/2023). If an empty data table is returned, an exception is raised. Consequently, it is much harder for PooPy to inadvertently generate misleading historical records.
This patch has introduced an unfortunate edgecase however. If there number of records returned for a history is an exact integer multiple of the API request limit (as of 29/2/24 this is 1000) then an exception will be raised in error. It is not clear how this can sensibly be fixed as really the API shouldn't be throwing erroneous empty responses!