search

epoch8 / airflow-exporter

Airflow plugin to export dag and task based metrics to Prometheus.
Other
247 stars 75 forks source link

Not getting metrics after pip install in container #65

Closed continuum-akshay-jadhav closed 5 years ago

continuum-akshay-jadhav commented 5 years ago

Hi Team,

I have ran PIP install in my airflow container and its telling Building wheel for prometheus-client (setup.py) ... done Created wheel for prometheus-client: filename=prometheus_client-0.7.1-cp36-none-any.whl size=41406 sha256=b9f03f1d9e47ac656a9b8b790ffe6fcb76de5570f8cb070d972378424c8eefcd Stored in directory: /root/.cache/pip/wheels/1c/54/34/fd47cd9b308826cc4292b54449c1899a30251ef3b506bc91ea Successfully built prometheus-client Installing collected packages: prometheus-client, airflow-exporter Successfully installed airflow-exporter-1.1.0 prometheus-client-0.7.1

but still can not view metrics at http:///admin/metrics/

instead i am getting Airflow 404 = lots of circles

Can some one please help us.

Also let me know if it supports Airflow 1.10.3

Or the Airflow version should be greater that 1.10.3

Thanks.

elephantum commented 5 years ago

This is very strange, did you restart your airflow webserver after pip install?

elephantum commented 5 years ago

Airflow version 1.10.3 should work.

continuum-akshay-jadhav commented 5 years ago

after the restart,Airflow container is getting restarted automatically and continuosly

kbbqiu commented 5 years ago

I had the same error Airflow 404 = lots of circles. Ended up downgrading to 1.0, it's fine.

Maybe some regression in 1.1.0?

continuum-akshay-jadhav commented 5 years ago

Hi @kbbqiu ,

So i should dowgrade it from 1.10.3 to 1.0.0

kbbqiu commented 5 years ago

Try: pip install -Iv airflow-exporter==1.0

I’m on airflow version 1.10.4

continuum-akshay-jadhav commented 5 years ago

Hi @kbbqiu ,

I tried above command inside my Airflow container ver 1.10.3 but still the container is getting restarted and getting Airflow 404 = lots of circles.

Can you please suggest somthing or should i upgrade my airflow version or this plugin supports Airflow 1.10.3+

elephantum commented 5 years ago

@continuum-akshay-jadhav / @kbbqiu Can you provide exact steps to reproduce?

continuum-akshay-jadhav commented 5 years ago

Hi @elephantum ,

We have Docker compose Airflow container running with version 1.10.3.

Steps: -I started the Container. -Loged into container as root and ran the command pip install airflow-exporter. -Then i logged out of my container. -At http:///admin/metrics/ their was Airflow 404 = lots of circles.

Also i tried to restart container,but after i restart it always kept restarting.

elephantum commented 5 years ago

@continuum-akshay-jadhav are you sure that after restart container was not recreated?

I would recommend adding pip install step into Dockerfile.

continuum-akshay-jadhav commented 5 years ago

Hi @elephantum ,

I have tried doing the same in docker file.. But still the container is getting retsarted without restart

elephantum commented 5 years ago

@continuum-akshay-jadhav can you make a gist with airflow logs when it's restarting?

continuum-akshay-jadhav commented 5 years ago

Hi @elephantum , As the container is getting restarted continuolsy i am not able to log in to see the logs

elephantum commented 5 years ago

@continuum-akshay-jadhav ok, can you provide a Dockerfile that can be used to replicate this issue?

continuum-akshay-jadhav commented 5 years ago

Hi @elephantum ,

PFB Docker file:

FROM puckel/docker-airflow:1.10.3 LABEL maintainer="Akshay Jadhav"

ENV DEBIAN_FRONTEND noninteractive ENV TERM linux

ARG AIRFLOW_VERSION=1.10.3 ARG AIRFLOW_USER_HOME=/usr/local/airflow ARG AIRFLOW_DEPS="kubernetes,statsd" ARG PYTHON_DEPS="psycopg2-binary snowflake-connector-python sqlparse python-jenkins Flask==1.0.4" ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}

ENV LANGUAGE en_US.UTF-8 ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8 ENV LC_CTYPE en_US.UTF-8 ENV LC_MESSAGES en_US.UTF-8

USER root

RUN set -ex \ && apt-get update -yqq \ && apt-get upgrade -yqq \
&& apt-get install -yqq --no-install-recommends unixodbc-dev \ && pip install apache-airflow[${AIRFLOW_DEPS}]==${AIRFLOW_VERSION} \ && if [ -n "${PYTHON_DEPS}" ]; then pip install ${PYTHON_DEPS}; fi \ && apt-get purge --auto-remove -yqq $buildDeps \ && apt-get autoremove -yqq --purge \ && pip install airflow-exporter \ && apt-get clean

COPY plugins ${AIRFLOW_USER_HOME}/plugins

COPY kube_context ${AIRFLOW_USER_HOME}/kube_context

RUN chown -R airflow: ${AIRFLOW_USER_HOME}

EXPOSE 8080 5555 8793

USER airflow WORKDIR ${AIRFLOW_USER_HOME} ENTRYPOINT ["/entrypoint.sh"] CMD ["webserver"]

elephantum commented 5 years ago

@continuum-akshay-jadhav I see that you are targeting kubernetes, probably that's the reason you can't see the logs.

Did you try to run this image locally to see docker output?

We use the same image (based on puckel/docker-airflow) for testing and we don't observe any issues.

continuum-akshay-jadhav commented 5 years ago

@elephantum , Some part of docker file was build by some one... Can you please secify wht do you mean by ...I see that you are targeting kubernetes

elephantum commented 5 years ago

@continuum-akshay-jadhav I've been able to recreate some issue with Airflow 1.10.3 (based on puckel docker image), it seems that it still connects to Airflow being incompatible with latest version of Flask.

Upgrading to Airflow 1.10.4 fixes it. Can you try upgrading to 1.10.4?

continuum-akshay-jadhav commented 5 years ago

Hi @elephantum ,

Thank you so much for your help.

Now I can see the metrics when i upgraded Airflow to 1.10.4

Before you close this issue just one question: Does airflow_dag_run_duration give duration of DAG that is running at a particular time?

elephantum commented 5 years ago

Does airflow_dag_run_duration give duration of DAG that is running at a particular time?

Yes, it is a duration of the longest dagrun currently running.