Failed to find OpenMetadata - Managed Airflow APIs

[martinmajtan]

Hello, We have installed OpenMetadata instance shipped with Elastic and Airflow. We used official helm charts with their image "docker.getcollate.io/openmetadata/ingestion". But we're facing a problem with connection from OpenMetadata to Airflow API.

We set up a basic auth according docs (we have also tested a SSO KeyCloak which was working, but getting same issue ). I upgraded to newest helm chart version v1.1.4 and OpenMetadata version.

When I try add service - Pipeline or Database I got error: Authentication failed for user [admin] trying to access the Airflow APIs.

I read all available docs and update our configuration as official docs said:

values.yaml for dependencies chart:

airflow:
  enabled: {{ is_airflow_enabled }}
  airflow:
    image:
      repository: docker.getcollate.io/openmetadata/ingestion
      tag: {{ airflow_image_tag }}
      pullPolicy: "IfNotPresent"
    executor: "KubernetesExecutor"
    config:
      # This is required for OpenMetadata UI to fetch status of DAGs
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth"
      # OpenMetadata Airflow Apis Plugin DAGs Configuration
      AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/opt/airflow/dags"
      # OpenMetadata Airflow Secrets Manager Configuration
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION: ""
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID: ""
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY: ""
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'
      AIRFLOW__WEBSERVER__AUTHENTICATE: 'true'
      AIRFLOW__SCHEDULER__AUTHENTICATE: 'true'
      AIRFLOW__API__ENABLE_EXPERIMENTAL_API: 'true'
    users:
    - username: admin
      password: admin
      role: Admin
      email: spiderman@superhero.org
      firstName: Peter
      lastName: Parker

values.yaml for OpenMetadata chart:

    pipelineServiceClientConfig:
      enabled: true
      className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
      apiEndpoint: http://openmetadata-dependencies-web.{{ openmetadata_namespace }}.svc.cluster.local:8080
      metadataApiEndpoint: http://openmetadata.{{ openmetadata_namespace }}.svc.cluster.local:8585/api
      verifySsl: "no-ssl"
      hostIp: ""
      ingestionIpInfoEnabled: false
      # healthCheckInterval in seconds
      healthCheckInterval: 300
      # local path in Airflow Pod
      sslCertificatePath: "/no/path"
      auth:
        username: admin
        password:
          secretRef: airflow-secrets
          secretKey: openmetadata-airflow-password

I tried create different user but same error (created new secret, upgraded airlflow values.yaml file for new user and gave Admin role).

I don't understand why I'm getting that error in browser, because when I jump to OpenMetadata pod and test connection to Airflow API all seems fine:

bash-5.1# curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health                                                                                                     
{"status": "healthy", "version": "1.1.0.3"}

Also when I tried call Airflow API with user, I didn't get error due to 401 Unauthorized:

bash-5.1# curl -XPOST ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/enable --data-raw '{"dag_id": "123"}' -u "admin:admin" --header 'Content-Type: application/json'
{"error": "DAG 123 not found."}

Can someone help ?

[harshach]

@martinmajtan please use our slack support channel https://slack.open-metadata.org . Github issues are meant for product bugs/features. Not support questions.

[Johnny-ZTSD]

The problem has resolved in openmetadata v1.3.0? @harshach I had the same problem ...