GitLab source resource broken when using private access token as authentication method #74

Open kevin-astrafy opened 8 months ago

General description

Creating a GitLab source using the Terraform provider with a private access token succeed, but then the connection resource fails to apply because the Airbyte application (worker) crash when configuring the connection. As a result, the connection cannot be set.

Configuring the same with the Airbyte UI or API doesn't have the problem

Environment

Airbyte OSS version 0.50.3 on Kubernetes
Terraform Provider version 0.3.7
GitLab.com (SaaS)

How to reproduce

Assuming you have the Terraform provider version 0.3.7 or lower (latest at this moment) and you can access the Airbyte API server from your Terraform configuration:

Create a GitLab source using the airbyte_source_gitlab terraform resource and use credentials type private_token (use a valid GitLab access token)
Create a dev-null destination (optionally using the terraform provider)
Create a connection using the airbyte_connection terraform resource -> Fail

The last step will produce a http 502 during the apply step

First analysis

After a first assessment, I discovered two things

The Airbyte worker logs show an exception because of the invalid Gitlab source configuration
The GitLab source configuration in the UI is displayed as invalid. The Authorization method is not set

Conclusion

The invalid payload when loading the source is creating the crash during the connection setup. The auth_type key in the credentials json payload is not present when the source is creating using the provider. If the source is created using the UI or the API, this field is present and the connection can be configured without issue.

To fix this, the provider must create the source according to the Airbyte API definition, which includes both the auth_type ant the access_token field

Hey @kevin-astrafy -- we'll look into this. In theory, the auth_type should always be sent over the wire as access_token when the credentials.private_token is set. Can you confirm it's definitely not? We'll try and replicate.

Moving parts in code are:

(const tag) https://github.com/airbytehq/terraform-provider-airbyte/blob/main/internal/sdk/pkg/models/shared/sourcegitlab.go#L40 (instantiation) https://github.com/airbytehq/terraform-provider-airbyte/blob/main/internal/provider/source_gitlab_resource_sdk.go#L42 (reflection to push the const tag into the marshalled JSON request body) https://github.com/airbytehq/terraform-provider-airbyte/blob/main/internal/sdk/pkg/utils/json.go#L249C10-L249C33 / https://github.com/airbytehq/terraform-provider-airbyte/blob/main/internal/sdk/pkg/utils/json.go#L361

Can you also confirm what terraform specification you're using for (https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/source_gitlab) (please redact all personal data) .

thanks @ThomasRooney

During my tests, I wasn't able to look at the unmarshalled JSON payload ( I can try by enabling debug mode i suppose) because i only focused on analysing the logs and the source payload. But when i look at the source from the UI, i can see that the auth_type is no present 2024-02-05_16-43

If i compare with the same source created from the UI, the auth_type is present and the configuration also automatically shows the authentcation method to be private token instead of the 3 dots when auth_type is not present 2024-02-05_16-53

This is also what is mentioned in the logs, because the app tries to read the auth_type from the python dict but the key doesn't exist.

Here is our terraform configuration for the source. The access token is provided via a variable that we have setup in our Terraform configuration

Hey -- I tried this on airbyte cloud and it looks like we're sending the right data:

Spec:

resource "airbyte_source_gitlab" "gitlab_source" {
  configuration = {
    api_url = "gitlab.com"
    credentials = {
      private_token = {
        access_token = "your_access_token"
      }
    }
    start_date = "2023-01-01T00:00:00Z"
  }
  name = "gitlab-test"
  workspace_id = airbyte_workspace.my_workspace.workspace_id
}

Post Request Body

{"configuration":{"api_url":"gitlab.com","credentials":{"access_token":"your_access_token","auth_type":"access_token"},"sourceType":"gitlab","start_date":"2023-01-01T00:00:00Z"},"name":"gitlab-test","workspaceId":"*redacted*"}

With a successful apply:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

So I think this looks to be an issue with something outside the speakeasy terraform provider (and hence out of my scope, sorry :( )

Thanks @ThomasRooney for trying this. I will also try with airbyte cloud, which I did not.

On top of that, I will also dump the json to see what payload I get on my own instance.

What is weird is that I am also getting a successful terraform apply, but I would expect a 400 if the json payload was invalid.

I will also dig more to find the root cause and post my results here

airbytehq / terraform-provider-airbyte