The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
While trying to deploy airbyte on a Ubuntu 20 server, I am facing a problem with ./run-ab-platform.sh.
airbyte-temporal is having a problem with creating a config client.
Relevant log output
ubuntu@data:~/services/airbyte$ ./run-ab-platform.sh
█████╗ ██╗██████╗ ██████╗ ██╗ ██╗████████╗███████╗
██╔══██╗██║██╔══██╗██╔══██╗╚██╗ ██╔╝╚══██╔══╝██╔════╝
███████║██║██████╔╝██████╔╝ ╚████╔╝ ██║ █████╗
██╔══██║██║██╔══██╗██╔══██╗ ╚██╔╝ ██║ ██╔══╝
██║ ██║██║██║ ██║██████╔╝ ██║ ██║ ███████╗
╚═╝ ╚═╝╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝ ╚══════╝
Move Data
Downloading docker-compose.yaml
Downloading docker-compose.debug.yaml
Downloading .env
Downloading .env.dev
found flags.yml locally!
Loading Shell Variables from .env...
Loading Shell Variables from .env.dev...
Starting Docker Compose
WARN[0000] The "LAUNCHDARKLY_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "NEW_SCHEDULER" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_PERSISTENCE" variable is not set. Defaulting to a blank string.
WARN[0000] The "JOB_ERROR_REPORTING_SENTRY_DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "WORKER_ENVIRONMENT" variable is not set. Defaulting to a blank string.
WARN[0000] The "GITHUB_STORE_BRANCH" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEPLOYMENT_MODE" variable is not set. Defaulting to a blank string.
WARN[0000] The "LAUNCHDARKLY_KEY" variable is not set. Defaulting to a blank string.
WARN[0000] The "LOG_CONNECTOR_MESSAGES" variable is not set. Defaulting to a blank string.
WARN[0000] The "SECRET_PERSISTENCE" variable is not set. Defaulting to a blank string.
WARN[0000] The "JOB_ERROR_REPORTING_SENTRY_DSN" variable is not set. Defaulting to a blank string.
WARN[0000] The "APPLY_FIELD_SELECTION" variable is not set. Defaulting to a blank string.
WARN[0000] The "FIELD_SELECTION_WORKSPACES" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEPLOYMENT_MODE" variable is not set. Defaulting to a blank string.
WARN[0000] The "REMOTE_CONNECTOR_CATALOG_URL" variable is not set. Defaulting to a blank string.
WARN[0000] The "TEMPORAL_HISTORY_RETENTION_IN_DAYS" variable is not set. Defaulting to a blank string.
WARN[0000] The "UPDATE_DEFINITIONS_CRON_ENABLED" variable is not set. Defaulting to a blank string.
WARN[0000] The "CDK_VERSION" variable is not set. Defaulting to a blank string.
WARN[0000] The "DEPLOYMENT_MODE" variable is not set. Defaulting to a blank string.
WARN[0000] The "PYTHON_VERSION" variable is not set. Defaulting to a blank string.
[+] Running 10/0
✔ Container airbyte-db Created 0.0s
✔ Container airbyte-temporal Created 0.0s
✔ Container init Created 0.0s
✔ Container airbyte-bootloader Created 0.0s
✔ Container airbyte-server Created 0.0s
✔ Container airbyte-connector-builder-server Created 0.0s
✔ Container airbyte-cron Created 0.0s
✔ Container airbyte-worker Created 0.0s
✔ Container airbyte-webapp Created 0.0s
✔ Container airbyte-proxy Created 0.0s
Attaching to airbyte-bootloader, airbyte-connector-builder-server, airbyte-cron, airbyte-db, airbyte-proxy, airbyte-server, airbyte-temporal, airbyte-webapp, airbyte-worker, init
init | MOUNT: /local_parent
init | ROOT_PARENT: /tmp
init | ROOT: /tmp/airbyte_local
init | MOUNT_ROOT: /local_parent//airbyte_local
airbyte-temporal | Start init
airbyte-temporal | Done init
airbyte-temporal | Waiting for PostgreSQL to startup.
airbyte-db |
airbyte-db | PostgreSQL Database directory appears to contain a database; Skipping initialization
airbyte-db |
airbyte-db |
airbyte-db | 2023-07-20 12:16:43.403 UTC [1] LOG: starting PostgreSQL 13.11 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
airbyte-db | 2023-07-20 12:16:43.403 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
airbyte-db | 2023-07-20 12:16:43.403 UTC [1] LOG: listening on IPv6 address "::", port 5432
airbyte-db | 2023-07-20 12:16:43.404 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
airbyte-db | 2023-07-20 12:16:43.406 UTC [22] LOG: database system was shut down at 2023-07-20 12:16:09 UTC
airbyte-db | 2023-07-20 12:16:43.410 UTC [1] LOG: database system is ready to accept connections
init exited with code 0
airbyte-temporal | PostgreSQL started.
airbyte-temporal | touch: CONTAINER_ALREADY_STARTED_PLACEHOLDER: Permission denied
airbyte-db | 2023-07-20 12:16:44.384 UTC [30] ERROR: database "temporal" already exists
airbyte-db | 2023-07-20 12:16:44.384 UTC [30] STATEMENT: CREATE DATABASE temporal
airbyte-temporal | 2023-07-20T12:16:44.417Z INFO Starting schema setup {"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:58"}
airbyte-temporal | 2023-07-20T12:16:44.417Z DEBUG Setting up version tables {"logging-call-at": "setuptask.go:68"}
airbyte-temporal | 2023-07-20T12:16:44.421Z DEBUG Current database schema version 1.9 is greater than initial schema version 0.0. Skip version upgrade {"logging-call-at": "setuptask.go:117"}
airbyte-temporal | 2023-07-20T12:16:44.421Z INFO Schema setup complete {"logging-call-at": "setuptask.go:132"}
airbyte-db | 2023-07-20 12:16:44.465 UTC [32] ERROR: database "temporal_visibility" already exists
airbyte-db | 2023-07-20 12:16:44.465 UTC [32] STATEMENT: CREATE DATABASE temporal_visibility
airbyte-temporal | 2023-07-20T12:16:44.494Z INFO Starting schema setup {"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:58"}
airbyte-temporal | 2023-07-20T12:16:44.494Z DEBUG Setting up version tables {"logging-call-at": "setuptask.go:68"}
airbyte-temporal | 2023-07-20T12:16:44.498Z DEBUG Current database schema version 1.1 is greater than initial schema version 0.0. Skip version upgrade {"logging-call-at": "setuptask.go:117"}
airbyte-temporal | 2023-07-20T12:16:44.498Z INFO Schema setup complete {"logging-call-at": "setuptask.go:132"}
airbyte-temporal | Starting to update the temporal DB
airbyte-temporal | 2023-07-20T12:16:44.524Z INFO UpdateSchemeTask started {"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v96/temporal/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:98"}
airbyte-temporal | 2023-07-20T12:16:44.528Z DEBUG Schema Dirs: [] {"logging-call-at": "updatetask.go:187"}
airbyte-temporal | 2023-07-20T12:16:44.528Z DEBUG found zero updates from current version 1.9 {"logging-call-at": "updatetask.go:128"}
airbyte-temporal | 2023-07-20T12:16:44.529Z INFO UpdateSchemeTask done {"logging-call-at": "updatetask.go:121"}
airbyte-temporal | Update the temporal DB is done
airbyte-temporal | Starting to update the temporal visibility DB
airbyte-temporal | 2023-07-20T12:16:44.546Z INFO UpdateSchemeTask started {"config": {"DBName":"","TargetVersion":"","SchemaDir":"/etc/temporal/schema/postgresql/v96/visibility/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:98"}
airbyte-temporal | 2023-07-20T12:16:44.549Z DEBUG Schema Dirs: [] {"logging-call-at": "updatetask.go:187"}
airbyte-temporal | 2023-07-20T12:16:44.549Z DEBUG found zero updates from current version 1.1 {"logging-call-at": "updatetask.go:128"}
airbyte-temporal | 2023-07-20T12:16:44.549Z INFO UpdateSchemeTask done {"logging-call-at": "updatetask.go:121"}
airbyte-temporal | Update the temporal visibility DB is done
airbyte-temporal | starting temporal server
airbyte-temporal | Temporal CLI address: 192.168.16.3:7233.
airbyte-temporal | 2023/07/20 12:16:44 Loading config; env=docker,zone=,configDir=config
airbyte-temporal | 2023/07/20 12:16:44 Loading config files=[config/docker.yaml]
airbyte-temporal | Waiting for Temporal server to start...
airbyte-temporal | {"level":"info","ts":"2023-07-20T12:16:44.576Z","msg":"Build info.","git-time":"2023-03-29T22:11:52.000Z","git-revision":"edbafd63e19dc0d585ec8cd0ebd793d3c6c6f6d6","git-modified":true,"go-arch":"amd64","go-os":"linux","go-version":"go1.20.2","cgo-enabled":false,"server-version":"1.20.1","debug-mode":false,"logging-call-at":"main.go:143"}
airbyte-temporal | Unable to create dynamic config client. Error: unable to validate dynamic config: dynamic config: config/dynamicconfig/development.yaml: stat config/dynamicconfig/development.yaml: no such file or directory
marcosmarxm
commented
1 year ago
What method are you using to run Airbyte?
Docker
Platform Version or Helm Chart Version
v0.50.8
What step the error happened?
On deploy
Revelant information
While trying to deploy airbyte on a Ubuntu 20 server, I am facing a problem with ./run-ab-platform.sh. airbyte-temporal is having a problem with creating a config client.
Relevant log output