search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.17k stars 981 forks source link

Kubernetes deployment #9735

Closed kananmammadli-vonage closed 1 year ago

kananmammadli-vonage commented 1 year ago

Is some content missing, wrong or not clear? Based on https://docs.open-metadata.org/deployment/kubernetes Following the mentioned documentation does not provide plug&play experience. Here are the points that need clarification:

  1. By default all services are not accessible from the other pods. mysql, elasticsearch, openmetadata-dependencies-web, openmetadata itself. Only MySQL has networkPolicy template file and hence can be enabled by setting mysql.networkPolicy.enabled: true in ./chart/deps/values.yaml. All the other require specifying networkPolicy explicitly.
  2. Default password for Airflow and MySQL are configurable, but not for OpenMetadata itself
  3. OpenMetadata uses single host name for both addressing ingestion airflow internally and generating url for ingestion DAGs. It makes the ingestion DAGs' links in OpenMetadata UI useless since they are not accessible out of cluster.

Describe the solution you'd like I see possible fix as follows:

  1. Add networkPolicy templates and add corresponding switchers in values.yaml
  2. Add parameters into values.yaml
  3. add guidance on possible alternatives
harshach commented 1 year ago

@akash-jain-10 lets follow-up on this one and close it

akash-jain-10 commented 1 year ago

Hello @kananmammadli-vonage . Thanks for bringing this up. Will take a look at adding network policy to OpenMetadata Helm Charts as part of Point 1. Could you please create an issue ticket here ? Thanks.

Will make sure to add a documentation for configuring/ updating default password via Bare Metal, Docker, K8s deployments with Basic Auth.

For Point 3, will take a look and prioritize getting this seperated. Current alternatives is to setup / map airflow webserver over dns name and use that with openmetadata configuration.

akash-jain-10 commented 1 year ago

Closing this issue and concluding things here -

  1. We now have Network Policies available for OpenMetadata Helm Chart. https://github.com/open-metadata/openmetadata-helm-charts/pull/126
  2. We have added documentation for usage of custom passwords with initial admins with Basic Auth SSO in docs here
  3. Airflow UI should be available over a DNS which can be used in configuring the Airflow Endpoint URL for OpenMetadata. This fixes the Issue for URL links which will not be broken anymore.