Terraform plan gets throttled

[brnck]

Hi,

we are adopting this provider to be used with bitbucket (thank you for creating it, by the way). Our Bitbucket is about 480-500 repositories and ~60 projects in one workspace.

Implementing this in our CI/CD we noticed that it gets throttled and starts to return error 429 for bitbucket_repository when running terraform plan.

After checking the documentation we see that only 1000 requests per hour are the Bitbucket threshold after which rate limiting kicks in. (https://support.atlassian.com/bitbucket-cloud/docs/api-request-limits/)

Support is saying that they can't help us and we should use several different accounts to prevent rate limits which might not be the best solution.

So we started exploring several possibilities for how to tackle this one, including maybe contributing to the provider and using some "cache" solution that would do something like:

repositories, err := bitbucketClient.Repositories.ListForAccount(&bitbucket.RepositoriesOptions{
        Owner: "workspace_name"
})

and then use a slice of repositories to correlate the information when running terraform plan. The problem is that we noticed that the provider can be used to create repositories in multiple workspaces in one go.

Are there any known users that are using this provider for multiple workspaces in one terraform state? Maybe anyone has more suggestions on how we can prevent requests from being rate limited or at least the possibility of being rate limited reduced as much as possible?

[zahiar]

It seems like the rate limiting is per user account so I guess you could try having multiple accounts for your CI setup, and cycle through them at random or something like that 🤷

[brnck]

Yes, rate limiting is per-user account. We ended up forking the provider and implementing a cache mechanism that would fetch all the repositories in one API call.

[zahiar]

Oh that's pretty cool, fetching all the repos in one call 😃