Unable to connect Dataform to Google Cloud Source Repository

[Stefn93]

Hello, I followed the example usage to create Dataform resources with terraform provided in documentation:

resource "google_sourcerepo_repository" "git_repository" {
  provider = google-beta
  name = "my/repository"
}

resource "google_secret_manager_secret" "secret" {
  provider = google-beta
  secret_id = "secret"

  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "secret_version" {
  provider = google-beta
  secret = google_secret_manager_secret.secret.id

  secret_data = "secret-data"
}

resource "google_dataform_repository" "dataform_respository" {
  provider = google-beta
  name = "dataform_repository"

  git_remote_settings {
      url = google_sourcerepo_repository.git_repository.url
      default_branch = "main"
      authentication_token_secret_version = google_secret_manager_secret_version.secret_version.id
  }

  workspace_compilation_overrides {
    default_database = "database"
    schema_suffix = "_suffix"
    table_prefix = "prefix_"
  }
}

It seems that Google Cloud Source Repositories doesn't work with this implementation due to its lack of token generation. How can I connect my Google Cloud Source Repository to a newly created Dataform Repository? Is Github\Gitlab\Bitbucket mandatory in order to link git with dataform? Do you have any alternative solutions to solve this issue?

Regards,

[edwardmedia]

@Stefn93 how did you know Google Cloud Source Repositories doesn't work with this implementation due to its lack of token generation?

Can you share your debug log when you apply above config?

[Stefn93]

@edwardmedia Terraform applies the resources successfully. The problem is that they actually don't work on the cloud platform. On UI, the following message appears:

We are unable to connect to your Git provider with the configured secret. Check that the token is valid and has not expired. View docs on connecting a remote git repository.

As a token I actually provided the string "secret-data" as I couldn't find where to generate it on google cloud source repositories. While creating a workspace in dataform repository, on UI, I then get this error:

API request error: Remote repository 'https://source.developers.google.com/p/<project-id>/r/<source-repo-name>' could not be reached.

On the documentation provided before, they just talk about connecting third-party repositories being Github\Gitlab repositories to dataform.

[edwardmedia]

@Stefn93 this site might not be the best place that can help you solve the problem. We are concerned about the terraform. If you experience failures when applying terraform configurations, that is what can help. It sounds like this is not the case as you said Terraform applies the resources successfully.

There are many possible reasons that could cause your problems. Without Terraform, are you able to make it working?

[edwardmedia]

@Stefn93 is this still an issue?

[Stefn93]

yes, in my opinion the example you provided that shows a google cloud source repo connecting to dataform can't work. It doesn't work out of terraform either. Google cloud source repo can't generate access tokens, therefore is not compatible as a third-party repo provided to Dataform

[edwardmedia]

@Stefn93 It is still not clear to me what is the expected behavior. Are you able to make it working with other means? If yes, please share.

@zli82016 do you know about this?

[Stefn93]

The expected behavior is that the google cloud source repository created by terraform should be correctly linked to dataform as its versioning system. But this is not possible since dataform expects an access token provided through a secret. Google Cloud Source Repository can't generate access token, therefore, it's not usable in this scenario

[tmaiaroto]

I believe this is a bug within GCP Dataform itself. It cannot connect to private GitHub repos. I've set them up manually with PATs and it just fails to connect with the same error message.

Once GCP fixes Dataform, I suspect this will work.

[Stefn93]

I do agree about it being a bug of Dataform GCP, but this issue is related to Google Cloud Source Repositories and not GitHub, which I've not been testing, so I cannot relate.

[philipp-heinrich]

I believe this is a bug within GCP Dataform itself. It cannot connect to private GitHub repos. I've set them up manually with PATs and it just fails to connect with the same error message.

Once GCP fixes Dataform, I suspect this will work.

same issue here. not using tf at all. so its dataform related i fear.

[lopezvit]

This is related with this issue on Google's side, so, as noted before, it will be solved when Google solves this issue. Answering to @edwardmedia : In my opinion, what could and should be fix is the documentation: it is misleading and could potentially make many people loose a lot of time if they think that it is possible to use CSR and (at least) currently is not the case. The least that could be added is a warning; and maybe even better, in the meanwhile, to change the example for an external GIT provider that does, indeed, work.