Kubernetes based deployment not processing ingestions

[peteroruba]

Describe the bug Datahub on Kubernetes does not execute ingestion tasks. Status is shown as "Pending...".

datahub-gms shown only one single log line: 12:35:44.420 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:47 - Successfully fed bulk request. Number of events: 4 Took time ms: -1

Ingestion Run Details / Logs: "No output found."

To Reproduce

• Deploy Datahub on Kubernetes
• Create a Postgres ingestion source
• Execute the ingestion via "Manual Execution"

Expected behavior The ingestion runs, terminates and produces Logs.

Desktop (please complete the following information):

• Helm Chart Version 0.2.132
• Kubernetes 1.22 (AWS EKS)

Additional context Values:

datahub-frontend:
  service:
    type: "ClusterIP"
datahub-gms:
  enabled: true
  service:
    type: ClusterIP
postgresqlSetupJob:
  enabled: true
mysqlSetupJob:
  enabled: false
elasticsearchSetupJob:
  enabled: true
datahubUpgrade:
  enabled: false
  noCodeDataMigration:
    sqlDbType: "POSTGRES"
acryl-datahub-actions:
  enabled: false
global:
  graph_service_impl: elasticsearch
  kafka:
    bootstrap:
      server: "datahub-prerequisites-kafka:9092"
    zookeeper:
      server: "datahub-prerequisites-zookeeper:2181"
    schemaregistry:
      url: "http://datahub-prerequisites-cp-schema-registry:8081"
  sql:
    datasource:
      host: "datahub-prerequisites-postgresql:5432"
      hostForpostgresqlClient: "datahub-prerequisites-postgresql"
      port: "5432"
      url:  "jdbc:postgresql://datahub-prerequisites-postgresql:5432/datahub"
      driver: "org.postgresql.Driver"
      username: "postgres"
      password:
        secretRef: "postgresql-secrets"
        secretKey: "postgres-password"
  neo4j:
    host: "datahub-prerequisites-neo4j-community:7474"
    uri: "bolt://datahub-prerequisites-neo4j-community"
  elasticsearch:
    insecure: "true"
  datahub_analytics_enabled: false

Ingestion Recipe:

source:
    type: postgres
    config:
        database: testing
        username: null
        password: null
        host_port: 'somelocalpostgresdb.local:5432'
sink:
    type: datahub-rest
    config:
        server: 'http://localhost:8080'

Datahub on Docker does not show this behavior. We successfully managed to run ingestion tasks there.

Help is much appreciated. Happy to provide additional logs and details upon request. Thanks!

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.