search

integrations / terraform-provider-github

Terraform GitHub provider
https://www.terraform.io/docs/providers/github/
MIT License
907 stars 748 forks source link

[BUG]: github_rest_api parses body inconsistently #1953

Open raihle opened 1 year ago

raihle commented 1 year ago

Expected Behavior

I expect to get the whole response in the "body" of the terraform data source.

Actual Behavior

I sometimes get a partial result, sometimes null. I have tried calling the API manually and using mitmproxy to intercept the data that is returned from GitHub when running terraform, and it is always complete. With TF_LOG=debug it seems that the provider is also getting a full response (see below).

Terraform Version

Terraform v1.5.5 on darwin_arm64

Affected Resource(s)

Terraform Configuration Files

terraform {
  required_version = " >= 1.5"
  required_providers {
    github = {
      source  = "integrations/github"
      version = ">= 5.39.0"
    }
  }
}

variable "github_token" {
  type        = string
  description = "An API token for GitHub"
  sensitive   = true
}

variable "github_organization" {
  type        = string
  description = "The login name of an organization to look up"
}

data "github_rest_api" "organization" {
  endpoint = "orgs/${var.github_organization}"
}

output "organization" {
  value = data.github_rest_api.organization.body
}

Steps to Reproduce

terraform apply

Debug Output

2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: 2023/10/12 10:36:18 [DEBUG] GitHub API Response Details:
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: ---[ RESPONSE ]--------------------------------------
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: HTTP/2.0 200 OK
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Access-Control-Allow-Origin: *
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Cache-Control: private, max-age=60, s-maxage=60
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Content-Security-Policy: default-src 'none'
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Content-Type: application/json; charset=utf-8
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Date: Thu, 12 Oct 2023 07:36:18 GMT
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Etag: W/"4a467e665db2f1c6d98fd445e272b2f0baa3a03a52d535bdf70c437d193c1cee"
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Github-Authentication-Token-Expiration: 2023-11-04 13:18:34 UTC
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Last-Modified: Fri, 02 Dec 2022 22:54:17 GMT
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Server: GitHub.com
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Vary: Accept, Authorization, Cookie, X-GitHub-OTP
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: Vary: Accept-Encoding, Accept, X-Requested-With
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Accepted-Oauth-Scopes: admin:org, read:org, repo, user, write:org
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Content-Type-Options: nosniff
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Frame-Options: deny
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Api-Version-Selected: 2022-11-28
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Media-Type: github.v3; format=json, github.stone-crop-preview; format=json
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Github-Request-Id: 0892:843E:6EDAC69:70024BA:6527A1F2
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Oauth-Scopes: repo, user:email
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Limit: 5000
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Remaining: 4995
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Reset: 1697099744
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Resource: core
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Ratelimit-Used: 5
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: X-Xss-Protection: 0
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: 
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: {
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "login": "org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "id": 00000000,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "node_id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "url": "https://api.github.com/orgs/org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "repos_url": "https://api.github.com/orgs/org-login/repos",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "events_url": "https://api.github.com/orgs/org-login/events",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "hooks_url": "https://api.github.com/orgs/org-login/hooks",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "issues_url": "https://api.github.com/orgs/org-login/issues",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_url": "https://api.github.com/orgs/org-login/members{/member}",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_members_url": "https://api.github.com/orgs/org-login/public_members{/member}",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "avatar_url": "https://avatars.githubusercontent.com/u/00000000?v=4",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "description": "Organization Description",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "name": "Organization Name",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "company": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "blog": "url",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "location": "country",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "email": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "twitter_username": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "is_verified": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "has_organization_projects": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "has_repository_projects": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "public_gists": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "followers": 409,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "following": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "html_url": "https://github.com/org-login",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "created_at": "2021-02-08T10:08:12Z",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "updated_at": "2022-12-02T22:54:17Z",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "archived_at": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "type": "Organization",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "total_private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "owned_private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "private_gists": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "disk_usage": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "collaborators": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "billing_email": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "default_repository_permission": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "two_factor_requirement_enabled": null,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_allowed_repository_creation_type": "private",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_public_repositories": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_private_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_internal_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_pages": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_fork_private_repositories": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "web_commit_signoff_required": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_public_pages": false,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "members_can_create_private_pages": true,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  "plan": {
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "name": "enterprise",
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "space": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "private_repos": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "filled_seats": 0,
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:   "seats": 0
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0:  }
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: }
2023-10-12T10:36:18.391+0300 [DEBUG] provider.terraform-provider-github_v5.39.0: -----------------------------------------------------
module.first_scope.data.github_rest_api.organization: Read complete after 0s [id=0892:843E:6EDAC69:70024BA:6527A1F2]

Panic Output

Not a panic per se, but the output of the module above is inconsistent (but always incomplete).

Some examples:

organization = tomap({
  "description" = "Organization Description"
  "private_gists" = ""
  "type" = "Organization"
})
organization = null
organization = tomap({
  "node_id" = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  "repos_url" = "https://api.github.com/orgs/login/repos"
})
organization = tomap({
  "disk_usage" = ""
  "login" = "login"
  "node_id" = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  "private_gists" = ""
})

No changes were made between the runs above - I'm simply rerunning terraform apply

Code of Conduct

SharpEdgeMarshall commented 9 months ago

This is critical for us

riezebosch commented 9 months ago

For me as well, probably related: #1776. Tried out this fix #2110 using 6.0.0-rc2 but still seeing null in the state file.

github-actions[bot] commented 3 days ago

👋 Hey Friends, this issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please add the Status: Pinned label if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!