search

fybrik / openmetadata-connector

Fybrik OpenMetadata Connector
Apache License 2.0
2 stars 7 forks source link

OpenMetadata Create Table Fails if No Metadata Included #40

Closed simanadler closed 1 year ago

simanadler commented 1 year ago

If I submit a fybrikapplication that includes metadata as part of the write flow params the connector succeeds in creating the table in openmetadata. However, if I do not include metadata then it fails when it tries to create the table (fails, and repeatedly retries). Here is a log from one of the failures.

Example with metadata:

apiVersion: app.fybrik.io/v1beta1
kind: FybrikApplication
metadata:
  name: mon-test4
  namespace: test
  labels:
    app: mon-test4
spec:
  selector:
    clusterName: thegreendragon
    workloadSelector:
      matchLabels:
        app: mon-test4
  appInfo:
    intent: Fraud Detection
  data:
    - dataSetID: 'new-data'
      flow: write
      requirements:
        flowParams:
          isNewDataSet: true
          catalog: fybrik-notebook-sample
          metadata:
            tags:
              Purpose.finance: true
            columns:
              - name: nameOrig
                tags:
                  PII.Sensitive: true
              - name: oldbalanceOrg
                tags:
                  PersonalData.Personal: true
        interface:
          protocol: fybrik-arrow-flight

Example without metadata

apiVersion: app.fybrik.io/v1beta1
kind: FybrikApplication
metadata:
  name: mon-test4
  namespace: test
  labels:
    app: mon-test4
spec:
  selector:
    clusterName: thegreendragon
    workloadSelector:
      matchLabels:
        app: mon-test4
  appInfo:
    intent: Fraud Detection
  data:
    - dataSetID: 'new-data'
      flow: write
      requirements:
        flowParams:
          isNewDataSet: true
          catalog: fybrik-notebook-sample
        interface:
          protocol: fybrik-arrow-flight

Log file of connector is attached, showing failure when no metadata.

1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:450 > Ingestion Pipeline not found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_service.go:112 > Ingestion Pipeline not found. Creating. connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:476 > Succeeded in creating Ingestion Pipeline: pipeline-testcatalog.resultset7 for Database Service Id: 6123d8e5-0a47-44f3-a8d8-4676a0580874 connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:494 > Database testcatalog-s3.default does not exist. Creating connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:19 POST /createAsset CreateAsset 385.179522ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:19 POST /createAsset CreateAsset 312.418751ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:19 POST /createAsset CreateAsset 303.640826ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:20 POST /createAsset CreateAsset 294.711102ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:20 POST /createAsset CreateAsset 303.411332ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:20 POST /createAsset CreateAsset 302.356709ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:21 POST /createAsset CreateAsset 342.833443ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:22 POST /createAsset CreateAsset 301.841522ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:23 POST /createAsset CreateAsset 295.3721ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:24 POST /createAsset CreateAsset 299.876316ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:27 POST /createAsset CreateAsset 312.294872ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:32 POST /createAsset CreateAsset 301.114224ms
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:111 > Successfully obtained token from Vault connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:141 > Successfully read Vault secret connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/vault/vault_secret_client.go:158 > Successfully extracted S3 credentials from Vault secret connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:368 > Found an identical database service connector="OpenMetadata Connector"
1:48PM WRN ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:419 > Could not find asset: testcatalog-s3.default.monaft7-test9ba4ef16e9.monaft79ba4ef16e9 connector="OpenMetadata Connector"
1:48PM INF ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:454 > Ingestion Pipeline found: testcatalog-s3."pipeline-testcatalog.resultset7" connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:490 > Database already exists: testcatalog-s3.default connector="OpenMetadata Connector"
1:48PM TRC ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:514 > Database Schema already exists: monaft7-test9ba4ef16e9 connector="OpenMetadata Connector"
1:48PM ERR ../home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:538 > createTable failed: monaft79ba4ef16e9 connector="OpenMetadata Connector"
2022/12/05 13:48:43 POST /createAsset CreateAsset 307.665493ms
user@SimaMac house_price_estimates %
simanadler commented 1 year ago

@cdoron Perhaps this is a duplicate of issue https://github.com/fybrik/openmetadata-connector/issues/31. I did not try with columns and without metadata, and I wasn't aware of the issue with columns. :(