search

47degrees / github4s

A GitHub API wrapper written in Scala
http://47degrees.github.io/github4s
Apache License 2.0
228 stars 75 forks source link

"Invalid tree info" - Fileless directories #259

Open blast-hardcheese opened 5 years ago

blast-hardcheese commented 5 years ago

Following along with sbt-microsite: Publish using GitHub4s, I've created a token with the repo scope, and have Enable SSO'd (which is required by my environment).

I was then able to use the Github API to perform some simple tests:

Despite this, when attempting to use publishMicrosite, I get:

[error] Error committing filesGitHub returned an error: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}

        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:53)
        at sbtorgpolicies.github.syntax$EitherTOps.execE(syntax.scala:43)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:172)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:148)
        at microsites.MicrositeAutoImportSettings.$anonfun$micrositeTasksSettings$13(MicrositeKeys.scala:305)
        at scala.Function1.$anonfun$compose$1(Function1.scala:44)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: github4s.GithubResponses$UnsuccessfulHttpRequest: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}
        at github4s.HttpRequestBuilderExtensionJVM.toEntity(HttpRequestBuilderExtensionJVM.scala:84)
        at github4s.HttpRequestBuilderExtensionJVM.toEntity$(HttpRequestBuilderExtensionJVM.scala:74)
        at github4s.jvm.Implicits$.toEntity(Implicits.scala:21)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.$anonfun$runMap$3(HttpRequestBuilderExtensionJVM.scala:67)
        at scala.util.Try$.apply(Try.scala:209)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:57)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:56)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.runMap(HttpRequestBuilderExtensionJVM.scala:61)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.run(HttpRequestBuilderExtensionJVM.scala:34)
        at github4s.HttpClient.post(HttpClient.scala:159)
        at github4s.api.GitData.createTree(GitData.scala:251)
        at github4s.free.interpreters.Interpreters$$anon$9.$anonfun$apply$7(Interpreters.scala:295)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at github4s.Github$GithubIOSyntaxEither.exec(Github.scala:62)
        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:51)
        ... 19 more

I found https://github.com/47deg/sbt-org-policies/issues/388#issuecomment-437045396, which was largely unhelpful other than to suggest that perhaps despite supplying the token, it doesn't work as advertised (possibly in my SSO environment).

Any assistance would be appreciated!

bilki commented 5 years ago

Hi @blast-hardcheese! Is your microsite repo initialized? Also double check that your sbt values micrositeGithubOwner and micrositeGithubRepo are correct.

blast-hardcheese commented 5 years ago

@bilki I presume so. It would be easier to just link where I am at this point.

I was successfully able to publish to a non-SSO-protected repo with the API token, fwiw.

bilki commented 5 years ago

Hey, @blast-hardcheese would you mind (if possible) trying to publish the microsite to another clean empty branch, just to be sure it's not the content of gh-pages that is provoking this bug?

blast-hardcheese commented 5 years ago

@bilki OK, so after digging quite a bit, I discovered that the root problem here is I'm getting NewTreeRequest(Some(...), Nil), as I have a tree containing only trees.

I've narrowed down what I suspect is the culprit: def getAllFiles: List[File] = Option(dirToCommit.listFiles()).toList.flatten.filter(_.isFile).

It seems as though we're doing top-down to create all the directories, but I suspect github really wants us to navigate depth first, creating the trees as the full hierarchy.

I'm wrapping up for the night, but this is where I got. If you've got some advice on how to proceed, (is flipping processing from top-down to bottom-up acceptable to you?) I'd appreciate it.

Thanks so far.

bilki commented 5 years ago

Hey @blast-hardcheese, could you check if your site folder contains any empty dir (in the machine from where you are trying to publish)? I get the exact same error Invalid tree info when I try to commitDir a folder that contains empty dirs. As soon as I remove them, it works like a charm.

blast-hardcheese commented 5 years ago

@bilki I have a directory that is empty, save for other directories, which contain files. The problem, as I've seen it, is that we create trees only containing blobs, then go back and create the next deepest tree with those blobs, with the parent being another tree.

My proposal would be to do a depth-first approach, committing the deepest tree, then collecting that tree's hash as another sibling in the next highest up tree to generate.

What do you think?

stanleydunne commented 5 years ago

I am seeing the same error "Invalid tree info" when trying to use the API to add individual files to my repo. I don't have empty folders. Has a solution been found? Edit : I found other issue in my own code that was corrupting the data I was passing in. Ignore my question.

bilki commented 5 years ago

Sorry about the delay @blast-hardcheese. That sounds like a possible solution, have you already tried it? I honestly don't know in what order github expects dir creations to happen. Anyway, you could open an issue at https://github.com/47deg/sbt-org-policies/issues, which is where the order on how commit dirs are updated is decided, and it's where your suggestion will apply.

calvellido commented 4 years ago

I've been using github4s through sbt-microsites too, and after messing a bit with this, I can confirm that the error will happen with a project structure like this. Source directory (src/main):

Then, processed through sbt-microsites, this is the actual content that will be committed with github4s:

I found that github4s will struggle to commit this structure (422), with the culprit being the fileless microsite directory. As a workaround I added a .gitkeep file near the data directory there and then github4s can proceed.

Somehow, a thing that indirectly solve this, is to use sbt-microsites in a different module, instead of the root one :man_shrugging: This solves it because using sbt-microsites that way, the microsite/data/menu.yml won't be generated (hence not committed), and the case is that it shouldn't (https://github.com/47deg/sbt-microsites/issues/335).

But, summarizing, and getting back to the bug, after some other tests, we can conclude that github4s will fail to commit a tree that include some kind of fileless directory like this (I don't think other considerations on the tree shape will matter really for this case):

Being subdir the only child of the static directory.

artemkorsakov commented 3 years ago

I have the same problem. While I had such a structure the publication was successful:

When I got such a structure that the publication began to fail with an error "Invalid tree info":

I got directory dir3 with no files in it - just another directory (dir4) The error "Invalid tree info" went away only after I added the file to the directory dir3: