search

integrations / terraform-provider-github

Terraform GitHub provider
https://www.terraform.io/docs/providers/github/
MIT License
912 stars 752 forks source link

404 error when creating files using github_repository_file resource #897

Open kunalnanda opened 3 years ago

kunalnanda commented 3 years ago

Terraform Version

= v1.0.5

Affected Resource(s)

github_repository_file

Terraform Configuration Files

resource "github_repository" "repos" {
  for_each = local.repos

  name         = each.key
  description  = join("", [each.value.description, " - Managed by Terraform"])
  visibility   = each.value.visibility
  homepage_url = each.value.homepage_url
}

resource "github_repository_file" "gitignore" {
  for_each = local.repos

  content             = file("templates/.gitignore")
  file                = ".gitignore"
  repository          = github_repository.repos[each.key].name
  branch              = "main"
  overwrite_on_create = true
  commit_message      = "Managed by Terraform"
  commit_author       = "user"
  commit_email        = "user@email.com"
}

Expected Behavior

The repositories should be created, and a default .gitignore file should be created with contents from the given template file.

Actual Behavior

The repositories are created. The .gitignore file fails with the following error:

Error: unexpected status code: 404 Not Found

with github_repository_file.gitignore["terraform"], on repos.tf line 10, in resource "github_repository_file" "gitignore": 10: resource "github_repository_file" "gitignore" {

This error is repeated 27 times, one for each repository.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Important Factoids

I am running this against my personal Github account. My Github token has the following permissions:

serain commented 3 years ago

I'm facing this but only when trying to create files in nested directories.

For example this works: ./circle/config.yaml And this fails: ./github/workflows/build.yaml

I can only create 1 directory deep.

EDIT: Fixed my issue, in my case it's because to update ./github/worklows you need the workflows scope on your token. Had nothing to do with nested directory.

jcudit commented 3 years ago

Can we get more detailed debug logging posted to the issue? Adding TF_LOG=DEBUG will help us determine which request is causing the 404.

kunalnanda commented 3 years ago

@jcudit Here is the debug log:

2021-09-29T11:24:49.630+1000 [DEBUG] Adding temp file log sink: /var/folders/q_/bp_xl53j0ms3k5y0b7d_c8g40000gn/T/terraform-log994684574
2021-09-29T11:24:49.631+1000 [INFO]  Terraform version: 1.0.5
2021-09-29T11:24:49.631+1000 [INFO]  Go runtime version: go1.16.4
2021-09-29T11:24:49.631+1000 [INFO]  CLI args: []string{"/usr/local/Cellar/tfenv/2.2.2/versions/1.0.5/terraform", "apply", "--auto-approve"}
2021-09-29T11:24:49.631+1000 [DEBUG] Attempting to open CLI config file: /Users/blah/.terraformrc
2021-09-29T11:24:49.631+1000 [INFO]  Loading CLI configuration from /Users/blah/.terraformrc
2021-09-29T11:24:49.634+1000 [INFO]  Loading CLI configuration from /Users/blah/.terraform.d/credentials.tfrc.json
2021-09-29T11:24:49.634+1000 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-09-29T11:24:49.635+1000 [DEBUG] ignoring non-existing provider search directory /Users/blah/.terraform.d/plugins
2021-09-29T11:24:49.635+1000 [DEBUG] ignoring non-existing provider search directory /Users/blah/Library/Application Support/io.terraform/plugins
2021-09-29T11:24:49.635+1000 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-09-29T11:24:49.636+1000 [INFO]  CLI command args: []string{"apply", "--auto-approve"}
2021-09-29T11:24:49.652+1000 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2021-09-29T11:24:50.943+1000 [DEBUG] Retrieve version constraints for service tfe.v2.1 and product terraform
2021-09-29T11:24:53.116+1000 [DEBUG] checking for provisioner in "."
2021-09-29T11:24:53.116+1000 [DEBUG] checking for provisioner in "/usr/local/Cellar/tfenv/2.2.2/versions/1.0.5"
2021-09-29T11:24:53.117+1000 [INFO]  Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2021-09-29T11:24:53.401+1000 [INFO]  backend/remote: starting Apply operation
rtjfarrimond commented 3 years ago

I was also facing this issue, adding auto_init = true to the github_repository fixed it for me. I did have to manually delete the repo and reapply the terraform config to get it to work.

The issue seems to have been that the repository was empty and had no branches. Adding auto_init = true created an initial commit with empty README (docs here), after which terraform apply was able to successfully create the files in the main branch. I didn't see it directly, but I would imagine that the underlying API call to create the files contains the branch name in the path, which would explain the 404 response received when the branch did not exist.

I also used the github_branch data source to access the main branch that was created by setting auto_init = true and replaced any literal "main" with data.github_branch.main.branch.

FalconerTC commented 3 years ago

Seeing the same thing with the following TF

resource "github_repository_file" "test" {
  repository = "RentTheRunway/bell_labs"
  branch     = "master"
  file       = ".testfile"
  content    = "**/*.tfstate"
}

Produces the following debug logs

2021/11/02 17:25:45 [DEBUG] EvalApply: ProviderMeta config value set
2021/11/02 17:25:45 [DEBUG] module.repositories.github_repository_file.test: applying the planned Create change
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: 2021/11/02 17:25:45 [DEBUG] Github API Request Details:
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: ---[ REQUEST ]---------------------------------------
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: GET /repos//RentTheRunway/bell_labs/branches/master HTTP/1.1
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Host: api.github.com
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: User-Agent: go-github
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Accept: application/vnd.github.v3+json
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Accept-Encoding: gzip
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0:
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0:
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: -----------------------------------------------------
2021-11-02T17:25:45.445-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: 2021/11/02 17:25:45 [TRACE] Acquiring lock for GitHub API request ("RentTheRunway/bell_labs:master")
2021-11-02T17:25:45.755-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: 2021/11/02 17:25:45 [TRACE] Releasing lock for GitHub API request ("RentTheRunway/bell_labs:master")
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: 2021/11/02 17:25:45 [DEBUG] Github API Response Details:
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: ---[ RESPONSE ]--------------------------------------
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: HTTP/2.0 404 Not Found
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Access-Control-Allow-Origin: *
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Content-Security-Policy: default-src 'none'
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Content-Type: application/json; charset=utf-8
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Date: Tue, 02 Nov 2021 21:25:45 GMT
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Server: GitHub.com
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: Vary: Accept-Encoding, Accept, X-Requested-With
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Content-Type-Options: nosniff
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Frame-Options: deny
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Github-Media-Type: github.v3; format=json
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Github-Request-Id: E962:0E29:24C546:471F11:6181ACD9
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Ratelimit-Limit: 60
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Ratelimit-Remaining: 49
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Ratelimit-Reset: 1635890283
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Ratelimit-Resource: core
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Ratelimit-Used: 11
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: X-Xss-Protection: 0
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0:
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: {
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0:  "message": "Not Found",
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0:  "documentation_url": "https://docs.github.com/rest"
UG] plugin.terraform-provider-github_v4.17.0: }
2021-11-02T17:25:45.756-0400 [DEBUG] plugin.terraform-provider-github_v4.17.0: -----------------------------------------------------
2021/11/02 17:25:45 [DEBUG] module.repositories.github_repository_file.test: apply errored, but we're indicating that via the Error pointer rather than returning it: unexpected status code: 404 Not Found
2021/11/02 17:25:45 [ERROR] eval: *terraform.EvalApplyPost, err: unexpected status code: 404 Not Found
2021/11/02 17:25:45 [ERROR] eval: *terraform.EvalSequence, err: unexpected status code: 404 Not Found
lerljaku commented 3 years ago

it has something to do with the set PAT. If i dont set it, url is malformed like that GET /repos//RentTheRunway/bell_labs/branches/master HTTP/1.1

if i set it it fills the username /repos/lerljaku/RentTheRunway/bell_labs/branches/master HTTP/1.1 but it doesnt work if the repo owner is organization instead of user

EDIT:

when passing filling owner to github provider, it works then

provider "github" {
  token = var.gh_pat
  owner = var.gh_org
}
lerrigatto commented 2 years ago

I actually had this error because in my state I have the resource pointing to a branch that doesn't exist anymore, even if the new one I specify was there. I solved deleting the resource from the state and then re-apply.

jamesggraf-m1 commented 2 years ago

I ran into this error and discovered the cause in my case was that there are two Github providers: hashicorp/github and integrations/github. The integrations/github provider works fine, the hashicorp/github provider throws 404 errors. So if your module doesn't explicitly require integrations/github, it'll use the hashicorp provider and blow up.

ahasna commented 2 years ago

Lost 3 hours of my life to this and the solution was to provide owner either explicitly in the provider or through GITHUB_OWNER 😭

Documentation is misleading mentioning that owner is optional.

rotilho commented 2 years ago

This problem happens when there's a mismatch between both implementations (integrations and hashicorp).

How to reproduce:

Expected behaviour:

Current behaviour:

rmn-lux commented 2 years ago

I have the same problem. My repo was empty, so i recreate it with non empty readme file and the problem has gone.

douglascayers commented 2 years ago

In my case,

So what I ended up doing was log in to my postgres database and manually update the state JSON. Since I had successfully been able to use github_repository_file resource with other repositories, I copied a good value and tweaked it for the repository that wasn't working. Then, I deleted my local terraform state (rm -f .terraform/terraform.tfstate) and reinitialized from my backend (terraform init --backend-config=${CONN_STR}).

Here's the JSON snippet I added to my state:

{
  "version": 4,
  "terraform_version": "1.1.9",
  "resources": [
    {
      "mode": "managed",
      "type": "github_repository_file",
      "name": "readme",
      "provider": "provider[\"registry.terraform.io/integrations/github\"]",
      "instances": [
        {
          "index_key": "my-awesome-repo",
          "schema_version": 0,
          "attributes": {
            "branch": "main",
            "commit_author": "my-git-username",
            "commit_email": "me@example.com",
            "commit_message": "Created by Terraform",
            "commit_sha": "af1d88db0dc7a616eca89d88ef25235de7cfe41e",
            "content": "my-awesome-repo",
            "file": "README.md",
            "id": "my-awesome-repo/README.md",
            "overwrite_on_create": false,
            "repository": "my-awesome-repo",
            "sha": "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed"
          },
          "sensitive_attributes": [],
          "private": "cyQstM==",
          "dependencies": [
            "github_repository.my-awesome-repo"
          ]
        }
      ]
    }
  ]
}

Note, commit_sha should be the commit that represents the file content, usually the most recent commit to the file in GitHub.

Note, sha is the SHA1 hash of the file contents. To get the known value in GitHub, use the following curl command against the API:

GITHUB_TOKEN="ghp_your_token"
OWNER="your-repo-owner"
REPO="your-repo-name"
COMMIT="commit-hash-of-your-file"

curl -X GET -H "Authorization: token ${GITHUB_TOKEN}" "https://api.github.com/repos/${OWNER}/${REPO}/git/blobs/${COMMIT}"

The JSON response includes the sha value.

shubham030 commented 2 years ago

I have the same issue, have checked all the above fixes/work arounds and nothing seems to work. I am trying to create 20+ files and some of them get created but the remaining end up with the same error unexpected status code: 404 Not Found

But if I apply the changes again then the remaining files get created. I have also checked the API rate limit and it looks good.

I am using the following config cdktf 0.13.1 terraform v1.3.2 github provider -> registry.terraform.io/integrations/github 4.31.0

kfcampbell commented 2 years ago

@shubham030 can you share a small snippet of HCL that reproduces the issue I can try?

johnjelinek commented 1 year ago

For me, I get the error when trying to import the file into state. The file was previously created by terraform, then deleted from state. Now, I get 404 when trying to re-import.

EDIT: turns out I needed to prepend the repo-name/ for the import id (eg: path/to/file:branch).

pneigel-ca commented 1 year ago

I have tried all provided workarounds above and the only one that has gotten me to a better place is manually creating the file outside of terraform and allowing it to overwrite it on next apply.

Terraform version: 1.4.5


terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "5.3.0"
    }
  }
}
pneigel-ca commented 1 year ago

This morning I created a new Personal Access Token and verified the problem is no longer present after granting it full permissions. There seems to be some permissions issue related to creating files in a repo for the first time, versus updating them in place.

I also verified my projects have branch protections in place - so perhaps it's less related to workflow permissions and more related to admin permissions to bypass these.

danielrive commented 1 year ago

Hello, i am facing the same problem, i change the permissions of the token to full permissions but didn't work

HariSekhon commented 1 year ago

I had this issue and realized the cryptic 404 on one repo despite having applied this same resource in a module to all my other github repos for a year was because that one repo only had the master branch whereas others had the main branch.

The 404 hint actually meant that the branch specified in the github_repository_file resource was main which didn't exist in that repo.

I solved it by replacing an explicit branch name of main with data.github_repository.default_branch which dynamically determines the default branch name.

You can see the real code example here as I deploy my CODEOWNERS file like this:

https://github.com/HariSekhon/Terraform/blob/master/github_repo/codeowners.tf#L27

and the data source chained from the repo creation to generate proper implicit depends_on:

https://github.com/HariSekhon/Terraform/blob/master/github_repo/repo.tf#L67

mejohnt commented 1 year ago

For me the issue was related to token permissions. I didn't narrow down the exact permissions, but trying just repo create and delete was not enough. I gave it full repo control too.

The boiler plate code works with it just fine.

I ended up with code that referred to multiple examples in this thread:


terraform {
  backend "local" {}

  required_providers {
    github = {
      source  = "integrations/github" #integrations/terraform-provider-github
      version = "~> 5.0"
    }
  }
}

provider "github" {
 token = "${var.gh_access_token}"
 owner = "${var.gh_owner}"
}
unoscar commented 9 months ago

Still facing this issue

yiskaneto commented 7 months ago

In my case something weird happened, I have a fine-grained PAT with content write permission for the target repo which has worked so far without issues, however when I tried to use both the github_repository_file data source and resource the Error: unexpected status code: 404 Not Found was returned, after trying must of the things mentioned above without success I decided to remove all the permissions from the PAT and then started added them again, the weird thing is that the permission combination that worked was properties and content write permissions, this is weird so I then remove the properties permission and guess what? it worked, so it work with the original permission I had at the beginning but I had to disable the same permissions, after this the issues didn't show up again.

dd84ai commented 1 week ago

I managed to resolve my case of problem receiving [] 404 error when creating web hoook

provider "github" {
  token = data.external.terraform_token.result["token"]
  owner = "my_organization_name" # THIS WAS MISSING
}

I granted all possible permissions (Administration, Webhooks, Content) But in the end it was resolved only when i specified owner in a provider

I reached success with fine grained token i made sure was create for organization