search

newrelic / newrelic-airflow-plugin

Send airflow metrics to New Relic!
Apache License 2.0
25 stars 19 forks source link

Periodic Errors When Sending Events To NR #26

Open nkazarian-spokeo opened 3 years ago

nkazarian-spokeo commented 3 years ago

We get this error from the underlying newrelic-telemetry-sdk package indicating some kind of non-serializable JSON object:

Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: [2021-07-06 15:07:35,538] {harvester.py:83} ERROR - New Relic send_batch failed with an exception.
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: Traceback (most recent call last):
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/harvester.py", line 75, in _send
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: response = self.client.send_batch(*flush_result)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 235, in send_batch
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: payload = self._create_payload(items, common)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/var/lib/airflow/venv/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 201, in _create_payload
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: payload = json.dumps([payload], separators=(",", ":"))
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/__init__.py", line 238, in dumps
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: **kw).encode(obj)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 199, in encode
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: chunks = self.iterencode(o, _one_shot=True)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 257, in iterencode
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: return _iterencode(o, 0)
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: File "/usr/lib64/python3.7/json/encoder.py", line 179, in default
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: raise TypeError(f'Object of type {o.__class__.__name__} '
Jul 06 15:07:35 ip-172-24-80-88.us-west-2.compute.internal airflow[10015]: TypeError: Object of type Decimal is not JSON serializable

Our environment is an EC2 running Amazon Linux AMI, we are not running in a container either. Anything helps, thank you.