🐛 Secret store: Deserialization of GSM secrets fails during connection check

[alafanechere]

Environment

• Airbyte version: ?
• Deployment: Kubernetes
• Severity: High
• Step where error happened: Setup new connection

Current Behavior

1. Use GSM
2. Try to set up a new BigQuery source pasting JSON credentials in the Airbyte UI
3. These credentials are stored in GSM as JSON objects
4. Airbyte tries to read them when checking connection with the source (and fails to parse them as it expects a string)

Expected Behavior

The secret stored in GSM after connector configuration should be readable in the connection check job and next jobs.

Logs

Slack discussion

[jrhizor]

We should document the format of the secret. This is not a bug; the value was entered in the wrong format.

[EDjur]

Hey @jrhizor - I'm happy if this is the case! But a bit curious: I pased the JSON credentials just like always (as in without using GSM).

So the only format that could be wrong is the base GSM secrets that I entered into the .secrets conf. But those seemed to work too as Airbyte had access to read/write from GSM.

But happy if it's just a case of documenting the format.

[jzhang-georgian]

Hi @jrhizor @EDjur could you share the right format for specifying the secret? Also how should I specify the credentials in the .env file for SECRET_STORE_GCP_CREDENTIALS?

Right now I'm pasting the whole content from the credential json file but it doesn't see to work: SECRET_STORE_GCP_CREDENTIALS={"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}

I got this error when setting up a postgres connector:

I posted this question in the Slack channel but haven't got any replies yet. I would really appreciate if you can share some insights. Thank you!

[jrhizor]

This is likely a problem with how the environment variable is escaped. Here's an example below comparing a non-escaped version with the escaped version of the partial key that you provided above.

~/code/airbyte master ······················································································································································································
❯ SECRET_STORE_GCP_CREDENTIALS={"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}
zsh: parse error near `}'
~/code/airbyte master !1 ···················································································································································································
❯ SECRET_STORE_GCP_CREDENTIALS={\"type\":\ \"service_account\",\"project_id\":\"my_project\",\"private_key_id\":\ \"acd90445....\",\"private_key\":\"...\",...}
~/code/airbyte master !1 ···················································································································································································
❯ echo $SECRET_STORE_GCP_CREDENTIALS
{"type": "service_account","project_id":"my_project","private_key_id": "acd90445....","private_key":"...",...}

[evantahler]

Is this still a problem or can we close this issue?

[willi-mueller]

I had a similar error with a docker-compose deployment.

The reason was that I did not inject the following environment variables into docker-compose. I had to add to docker-compose.yaml as environment variables to the services server and worker:

- SECRET_STORE_GCP_PROJECT_ID=${SECRET_STORE_GCP_PROJECT_ID}
- SECRET_STORE_GCP_CREDENTIALS=${SECRET_STORE_GCP_CREDENTIALS}

These two variables need to be defined in the .env or loaded from the Google Secret Manager via another script like this:

export SECRET_STORE_GCP_CREDENTIALS=$(gcloud secrets versions access 1 --secret airbyte_runner_service_account_key)

Then, the service account json works without being wrapped in e.g. single quotes. The newlines can be kept and the quotes should NOT be escaped. Thus, it has to look like this:

{
    "type": "service_account",
    "project_id": "my-project-id",
    "private_key_id": "123abc",
    "private_key": "-----BEGIN PRIVATE KEY-----\n123\n456\n=\n-----END PRIVATE KEY-----\n",
    "client_email": "airbyte-runner@my-project-id.iam.gserviceaccount.com",
    "client_id": "987",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/airbyte-runner%40my-project-id.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
}