search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
248 stars 75 forks source link

Exporter fails with MySQL: get_dag_duration_info #99

Closed dmeibusch closed 3 years ago

dmeibusch commented 3 years ago

Airflow 2.0.1 with MySQL db airflow exporter 1.5.0

Python version: 3.8.6
Airflow version: 2.0.1
Node: redacted
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/airflow_exporter/prometheus_exporter.py", line 305, in list
    return Response(generate_latest(), mimetype='text')
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/prometheus_client/exposition.py", line 177, in generate_latest
    for metric in registry.collect():
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/prometheus_client/registry.py", line 83, in collect
    for metric in collector.collect():
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/airflow_exporter/prometheus_exporter.py", line 259, in collect
    for dag_duration in get_dag_duration_info():
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/airflow_exporter/prometheus_exporter.py", line 147, in get_dag_duration_info
    sql_res = Session.query( # pylint: disable=no-member
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3373, in all
    return list(self)
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 82, in instances
    rows = [
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 83, in <listcomp>
    keyed_tuple([proc(row) for proc in process])
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 83, in <listcomp>
    keyed_tuple([proc(row) for proc in process])
  File "/data/home/osint/airflow/.venv/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 1968, in process
    return value - epoch
TypeError: unsupported operand type(s) for -: 'decimal.Decimal' and 'datetime.datetime'
elephantum commented 3 years ago

Interesting.

Can you help me with creation of test case that fails?

dmeibusch commented 3 years ago

This was out of the box Airflow 2.0.1 with MySQL DB and example dags provided. What would help for a test case?

dmeibusch commented 3 years ago

Ok, digging through the code, I can see that it is probably falling through to default in line 139 in the durations dict to possibly get this exception on a value - epoch expression. I'm using the mysql+mysqlconnector driver for the connection - I'm not sure with that driver what Session.bind.driver returns.

dmeibusch commented 3 years ago

Yup, I think that's the problem: 

>>> db.session.bind
Engine(mysql+mysqlconnector://root:***@127.0.0.1:3307/airflow?charset=utf8)
>>> db.session.bind.driver
'mysqlconnector'
dmeibusch commented 3 years ago

Adding mysqlconnector to the dict with the same value as mysqldb has fixed the issue for me.

elephantum commented 3 years ago

@dmeibusch I see, we have pending PR that fixes this issue: https://github.com/epoch8/airflow-exporter/pull/94

For some reason I was sure, that it is already merged. Will do

elephantum commented 3 years ago

Should be fixed in v1.5.1