SolarArbiter / solarforecastarbiter-api

HTTP API and database schema for the Solar Forecast Arbiter
https://api.solarforecastarbiter.org
MIT License
10 stars 6 forks source link

Replace any NaNs in json when storing to mysql #213

Closed sentry-io[bot] closed 4 years ago

sentry-io[bot] commented 4 years ago

for example when a NaN shows up in a metric value, json.dumps() sets the value as NaN and mysql can't interpret it. Instead, probably need some sentinel value to replace nans on inserts and replace with nan on read

Sentry Issue: API-2T

InternalError: (3140, 'Invalid JSON text: "Invalid value." at position 364899 in value for column \'.new_raw_report\'.')
(15 additional frame(s) were not displayed)
...
  File "pymysql/connections.py", line 732, in _read_query_result
    result.read()
  File "pymysql/connections.py", line 1075, in read
    first_packet = self.connection._read_packet()
  File "pymysql/connections.py", line 684, in _read_packet
    packet.check_error()
  File "pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
alorenzo175 commented 4 years ago

example of report that fails on prod

{
  "report_parameters": {
    "categories": [
      "date"
    ],
    "end": "2019-11-21T00:00:00+00:00",
    "filters": [
      {
        "quality_flags": [
          "USER FLAGGED",
          "NIGHTTIME"
        ]
      }
    ],
    "metrics": [
      "r"
    ],
    "name": "nanfail",
    "object_pairs": [
      {
        "forecast": "71628e0a-9a07-11e9-b9ec-0a580a8003d4",
        "observation": "c8668c10-7e49-11e9-8534-0a580a8003e9"
      }
    ],
    "start": "2019-11-19T00:00:00+00:00"
  }
}