Closed hongbo-miao closed 2 months ago
Got some help from Rob Holland from Temporal team on Slack and thanks!
It may be because of these lines in Airbyte https://github.com/airbytehq/airbyte-platform/blob/686cdb20a42865cea557871020ecbd44ca8ef8e1/airbyte-temporal/scripts/update-and-start-temporal.sh#L50-L55
If you see latest Temporal code https://github.com/temporalio/docker-builds/blob/40955e0f772939045dc7830c20f704149d9e81c7/docker/auto-setup.sh#L208-L293
They allow to pass these parameters
POSTGRES_TLS_ENABLED
POSTGRES_TLS_DISABLE_HOST_VERIFICATION
POSTGRES_TLS_CERT_FILE
POSTGRES_TLS_KEY_FILE
POSTGRES_TLS_CA_FILE
POSTGRES_TLS_SERVER_NAME
when calling temporal-sql-tool setup-schema
. But Airbyte code does not provide these parameters.
This explains when I pass CA pem file, the pod "airbyte-temporal" log shows connecting to RDS successfully in the beginning then failed at sql schema version compatibility check step.
Thanks for the detailed issue @hongbo-miao I added it to the platform team to implement in the future sprints.
Following up on this, after a few internal tests we've validated that adding the following values should resolve this issue:
extraEnv:
- name: POSTGRES_TLS_ENABLED
value: "true"
- name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
value: "true"
- name: SQL_TLS_ENABLED
value: "true"
- name: SQL_TLS_DISABLE_HOST_VERIFICATION
value: "true"
We'll also look to resolve this so this is automatic, and you no longer need to manually inject these in the future.
Following up on this, after a few internal tests we've validated that adding the following values should resolve this issue:
extraEnv: - name: POSTGRES_TLS_ENABLED value: "true" - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION value: "true" - name: SQL_TLS_ENABLED value: "true" - name: SQL_TLS_DISABLE_HOST_VERIFICATION value: "true"
We'll also look to resolve this so this is automatic, and you no longer need to manually inject these in the future.
Thanks @Hesperide unfortunately, this still not work for us. 🥲 I have all these 3 values set in "Experiment 3-3" above and still failed if you read closely. Has your RDS's force SSL enabled by rds.force_ssl=1
? If not enabled, it will work for sure.
I explained the reason why it may not work at https://github.com/airbytehq/airbyte/issues/39636#issuecomment-2182034537. Basically Airbyte's for temporal-sql-tool setup-schema code is out-of-date, it needs to update latest temporal-sql-tool setup-schema code which is why during provision, Termporal failed to connect to RDS.
Do you mind re-opening it? Thank you!
Helm Chart Version
0.199.0
What step the error happened?
On deploy
Relevant information
Originally posted at Stack Overflow, here is a copy:
I am trying to deploy Airbyte in Kubernetes (Amazon EKS) with external Postgres (Amazon RDS).
I am using
temporalio/auto-setup
image is v1.23.0.Group 1
Experiment 1-1 (Succeeded, but with
rds.force_ssl
disabled)When I disabled
rds.force_ssl
in Amazon RDS parameter group withrds.force_ssl: 0
, Airbyte can be deployed successfully. It is worth mentioning the pod "airbyte-temporal" can talk to RDS successfully.Here is my Airbyte Helm my-values.yaml:
Here is the pod "airbyte-temporal" successful log:
https://gist.github.com/hongbo-miao/eb5dcc71ad60aa38d285a5ed816128ed
Experiment 1-2 (Failed with
rds.force_ssl
enabled)I do want to enable
rds.force_ssl
. When I userds.force_ssl: 1
with same my-values.yaml from experiment 1, pod "airbyte-temporal" failed deploying with error:Experiment 1-3 (Partially failed with
rds.force_ssl
enabled when pass CA pem file)I downloaded Amazon RDS's global-bundle.pem from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.CertificatesAllRegions
And deployed a file
Based on auto-setup source code, I found
POSTGRES_TLS_ENABLED
,POSTGRES_TLS_DISABLE_HOST_VERIFICATION
,POSTGRES_TLS_CA_FILE
. (It also hasPOSTGRES_TLS_CERT_FILE
andPOSTGRES_TLS_KEY_FILE
inside)I updated Airbyte Helm my-values.yaml
temporal
section toI can confirm pod airbyte-temporal picked up file
amazon-rds-ca-global-bundle.pem
correctly. As if the path is wrong, it will throw an path related error saying cannot find.Now pod "airbyte-temporal" log is different, it seems it failed at a later "sql schema version compatibility check" step.
Group 2
Experiment 2-1
force_ssl: 0
(force SSL disabled) in RDS.Whole Airbyte deployed successfully. However, pod "airbyte-temporal" log shows some error in the beginning, I guess it falls back to use non-SSL way to connect
Experiment 2-2
force_ssl: 0
(force SSL disabled) in RDS.Whole Airbyte deployed successfully. pod "airbyte-temporal" has no error at all.
Also, I found
POSTGRES_TLS_CA_FILE
here is acutally optional, as long as the PEM file at/etc/ssl/certs/amazon-rds-ca-global-bundle.pem
, then it is good - no "failed to verify certificate: x509: certificate signed by unknown authority" error. And I found all other CA pem files are at this folder as well.Group 3
Group 3 experiments are done together with Rob Holland from Temporal team. Slack conversation is here.
Experiment 3-1
force_ssl: 1
(force SSL enabled) in RDS.POSTGRES_TLS_DISABLE_HOST_VERIFICATION
is"false"
.Pod "airbyte-temporal" log shows connecting to RDS successfully in the beginning then failed with
Same log to "Experiment 1-3".
Experiment 3-3
force_ssl: 1
(force SSL enabled) in RDS.Pod "airbyte-temporal" log shows connecting to RDS successfully in the beginning then failed at sql schema version compatibility check step. Same log to "Experiment 1-3".