search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
248 stars 75 forks source link

Exporter fails with Sqlite DB used in development environment #36

Closed ryan-carlson closed 5 years ago

ryan-carlson commented 5 years ago

When using the Sqlite DB, the following error occurs when attempting to open the metrics page.

Traceback (most recent call last): File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functionsrule.endpoint File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, *kwargs) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, args, **kwargs) File "/Users/ryan.carlson/airflow/plugins/prometheus_exporter/prometheus_exporter.py", line 164, in index return Response(generate_latest(), mimetype='text/plain') File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/prometheus_client/exposition.py", line 88, in generate_latest for metric in registry.collect(): File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/prometheus_client/core.py", line 147, in collect for metric in collector.collect(): File "/Users/ryan.carlson/airflow/plugins/prometheus_exporter/prometheus_exporter.py", line 150, in collect for dag in get_dag_duration_info(): File "/Users/ryan.carlson/airflow/plugins/prometheus_exporter/prometheus_exporter.py", line 108, in get_dag_duration_info DagRun.state == State.RUNNING File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2703, in all return list(self) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances util.raise_from_cause(err) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 78, in instances for row in fetch] File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 78, in for row in fetch] File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 77, in rows = [keyed_tuple([proc(row) for proc in process]) File "/Users/ryan.carlson/.virtualenvs/spothero-dataflows/lib/python3.6/site-packages/sqlalchemy/sql/sqltypes.py", line 1709, in process value = impl_processor(value) ValueError: Couldn't parse datetime string '0' - value is not a string.

ryan-carlson commented 5 years ago

I have code fix that I can share which enable support Sqlite. Is it possible for me to submit this as a PR?

elephantum commented 5 years ago

@ryan-carlson of course! We are glad to receive contributions.

ryan-carlson commented 5 years ago

Sounds great, please see https://github.com/epoch8/airflow-exporter/pull/37

elephantum commented 5 years ago

I believe this can be closed, as #37 is merged

ryan-carlson commented 5 years ago

Confirmed, I have tested with the latest master