search

hub4j / github-api

Java API for GitHub
https://github-api.kohsuke.org/
MIT License
1.13k stars 723 forks source link

Unable to clone submodules #915

Open saptakniyogi opened 4 years ago

saptakniyogi commented 4 years ago

Describe the bug When Github organization scans for repos, it fails if the repo contains submodules with following error: Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of [Lorg.kohsuke.github.GHContent; out of START_OBJECT token

To Reproduce Steps to reproduce the behavior:

  1. Scan Github Org with repos having sub-modules

Expected behavior Scan and clone the sub-modules

Desktop (please complete the following information):

Additional context

bitwiseman commented 4 years ago

@saptakniyogi Could you point to a public example of a repository with submodules? Also please provide a full stack trace.

It is unlikely that the issue is purely in this library.

saptakniyogi commented 4 years ago

The error happens when jenkinsfile is part of the submodule

bitwiseman commented 4 years ago

@saptakniyogi This is definitely not something that can be fixed in this library alone. Some work will need to be done in the related Jenkins plugins to make this work. Make sure there as an issue filed in the Jenkins JIRA database and crosslink with this issue.

Still need the full stack trace and a public example of a repository that reproduces this problem.

saptakniyogi commented 4 years ago

You can use the following public repository: https://github.com/saptakniyogi/Test.git. Submodule is part of the repo where a sample Jenkinsfile is present

saptakniyogi commented 4 years ago 
ERROR: [Thu Aug 13 11:44:59 CEST 2020] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@4f469272
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `[Lorg.kohsuke.github.GHContent;` out of START_OBJECT token
 at [Source: (String)"{"name":"name","path":"path","sha":"c46a5f7...26a4","size":0,"url":"base_url/repos/owners/repo/contents/sub?ref=refs/heads/master","html_url":"url//owner/subRepo/tree/c46a5f7...26a4","git_url":"base_url/repos/owners/subRepo/git/trees/c46a5f7...26a4","download_url":null,"type":"submodule","submodule_git_url":"git@githu"[truncated 380 chars]; line: 1, column: 1]
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1468)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1242)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1148)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.handleNonArray(ObjectArrayDeserializer.java:329)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:175)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
    at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2057)
    at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1496)
    at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
    at org.kohsuke.github.GitHubPageIterator.lambda$fetch$0(GitHubPageIterator.java:145)
    at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:462)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:413)
Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 200, message: '200 OK' for URL: base_url3/repos/owner/repo/contents/subRepo?ref=refs%2Fheads%2Fmaster
    at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:501)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:421)
    at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:144)
    at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:91)
    at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:106)
    at org.kohsuke.github.PagedIterator.nextPageArray(PagedIterator.java:134)
    at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:78)
    at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:106)
    at org.kohsuke.github.PagedIterable.toList(PagedIterable.java:118)
    at org.kohsuke.github.GHRepository.getDirectoryContent(GHRepository.java:2308)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.stat(GitHubSCMProbe.java:138)
    at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
    at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
    at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:249)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1000)
    at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
    at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
    at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
    at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1405)
    at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1420)
    at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:254)
    at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:204)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1012)
    at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:488)
    at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
    at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
    at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:969)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
[Thu Aug 13 11:44:59 CEST 2020] Finished organization scan. Scan took 0.63 sec
FATAL: Failed to recompute children of org-name
org.kohsuke.github.HttpException: Server returned HTTP response code: 200, message: '200 OK' for URL: base_url/repos/owner/repo/contents/subRepo?ref=refs%2Fheads%2Fmaster
    at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:501)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:421)
    at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:144)
    at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:91)
    at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:106)
    at org.kohsuke.github.PagedIterator.nextPageArray(PagedIterator.java:134)
    at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:78)
    at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:106)
    at org.kohsuke.github.PagedIterable.toList(PagedIterable.java:118)
    at org.kohsuke.github.GHRepository.getDirectoryContent(GHRepository.java:2308)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.stat(GitHubSCMProbe.java:138)
    at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
    at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
    at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:249)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1000)
    at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
    at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
    at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
    at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1405)
    at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1420)
    at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:254)
    at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:204)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1012)
    at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:488)
    at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
    at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
    at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:969)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `[Lorg.kohsuke.github.GHContent;` out of START_OBJECT token
 at [Source: (String)"{"name":"name","path":"path","sha":"c46a5f7...26a4","size":0,"url":"base_url/repos/owner/repo/contents/subRepo?ref=refs/heads/master","html_url":"url/owner/subRepo/tree/c46a5f7...26a4","git_url":"base_url/repos/owner/subRepo/git/trees/c46a5f7...26a4","download_url":null,"type":"submodule","submodule_git_url":"git@githu"[truncated 380 chars]; line: 1, column: 1]
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1468)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1242)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1148)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.handleNonArray(ObjectArrayDeserializer.java:329)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:175)
    at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
    at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2057)
    at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1496)
    at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
    at org.kohsuke.github.GitHubPageIterator.lambda$fetch$0(GitHubPageIterator.java:145)
    at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:462)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:413)
    ... 27 more
Finished: FAILURE
bitwiseman commented 4 years ago

This is in the same area as #875 with many of the same issues. Basically, what you're trying to do is poorly supported by the GitHub REST API, making it hard to implement a good fix for this issue.

Traversing symlink or submodule folders is expensive, requiring multiple GitHub API calls to determine success or failure. This library could offer a convenience method to make this easier for clients, but clients (such as the Jenkins plugin mentioned here) would need to opt in. We would not want to make it the default behavior.

PRs are welcome for this but it will require significant design and testing.