WIP: add configuration management and authentication features

[greenpau]

Resolves: epoch8/airflow-exporter#54, epoch8/airflow-exporter#55, epoch8/airflow-exporter#56

More info:

This commit adds the support for the configuration of the exporter via prometheus_exporter section of Airflow configuration.

The keys with expose_ prefix enable or disable the exposure of certain class of metrics. For example, Airflow variables-related metrics are not exposed by default. However, with expose_variables set to True, the exporter will start exposing the metrics. Similarly, by default, the exporter exposes scheduler-related metrics. However, with expose_scheduler set to False, the exporter will not export them.

Additionally, when expose_config is being enabled, the exporter will expose a subset of core and prometheus_exporter configuration settings.

[prometheus_exporter]
auth_enabled = True
auth_token = 4e1dba1c-2b66-4275-b8ae-292ee9665fa1
expose_variables = True
expose_config = False
expose_scheduler = False

It is possible to disable the exporter:

[prometheus_exporter]
disabled = True

When authentication is enabled, the metrics are accessible via:

curl -v -H 'Authorization: Bearer 4e1dba1c-2b66-4275-b8ae-292ee9665fa1' https://localhost:8443/admin/metrics/

Also, when the authentication is enabled, Prometheus scrape job might look like this:

  - job_name: 'airflow_exporters'
    metrics_path: /admin/metrics/
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token: '4e1dba1c-2b66-4275-b8ae-292ee9665fa1'
    scrape_interval: 5m
    static_configs:
    - targets:
      - '127.0.0.1:8443'
      labels:
        environment: 'dev'
    relabel_configs:
    - source_labels: [__address__]
      regex: "^(.*):(.*)$"
      target_label: instance
      replacement: ${1}
    - source_labels: [instance]
      regex: "^(127.0.0.1)$"
      target_label: instance
      replacement: "airflow"

[elephantum]

@greenpau nice job!

Is it ready for review?

[greenpau]

@elephantum , not yet. I will ping you when it is done.

[greenpau]

@elephantum , conceptually, you could try reviewing the exporter.

Also, do you think it is the code or the test itself causing the below?

[elephantum]

It is difficult to review this PR because you did so many changes at once + renamed files.

Let’s discuss if it is possible to split this PR into several that are incremental and change only small parts of code at a time.

[greenpau]

@elephantum, unfortunately, I don’t have much time to work on splitting it. By and large, it has these major changes:

• introduce exporter configuration
• introduce token based authentication (and a bypass if existing authenticated session)
• add scheduler_up metric

Additionally, it:

• it only runs the plugin for webserver process

[elephantum]

I see. In current state this PR can not be accepted: tests are broken, diff is 100%

We will use it as a reference to fix #54, #55, #56 separately.