search

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.5k stars 493 forks source link

[Bug][Devlake][GitHub] GitHub app connection data source usability issues #7655

Open petercort opened 1 week ago

petercort commented 1 week ago

Search before asking

What happened

Creating a GitHub App connection isn't working as expected.

  1. No organizations or repositories are found 'naturally' after the connection is created
  2. Can't multiselect to add repositories when they are specified into the datasources search

Image Image Image Image Image

What do you expect to happen

I would expect that the repositories in the organization where the app has been installed would auto populate and I could select multiples to add as a data source.

How to reproduce

  1. Configure a GitHub App as a connection
  2. Add data sources - no

Anything else

I was able to reproduce this behavior with a GitHub app installed into an organization and my personal account.

I'm willing to take a crack at a PR if it's not a massive thing. I'll spend some time setting up a debugger and seeing if I can get more info. Might need someone more skilled than me (cough cough copilot) to fix this.

I also tested this with the v0.21.0@ac4f7fe and it's reproducible there.

Here's the error message formatted nicely: 

level=error msg="HTTP 400 error
    caused by: attached stack trace
     -- stack trace:
        | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*DsScopeApiHelper[...]).PutMultiple
        | /app/helpers/pluginhelper/api/ds_scope_api_helper.go:114  
        | github.com/apache/incubator-devlake/plugins/github/api.PutScopes  
        | /app/plugins/github/api/scope_api.go:42
        | github.com/apache/incubator-devlake/server/api.handlePluginCall.func1
        | /app/server/api/router.go:135
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/apache/incubator-devlake/server/api.OAuth2ProxyAuthentication.func1
        | /app/server/api/middlewares.go:95
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/apache/incubator-devlake/server/api.RestAuthentication.func1
        | /app/server/api/middlewares.go:117
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.LoggerWithConfig.func1
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620
        | github.com/gin-gonic/gin.(*Engine).ServeHTTP
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576
        | net/http.serverHandler.ServeHTTP
        | /usr/local/go/src/net/http/server.go:2936
        | net/http.(*conn).serve
        | /usr/local/go/src/net/http/server.go:1995
        | runtime.goexit
        | /usr/local/go/src/runtime/asm_arm64.s:1172
    Wraps: (2) invalid data row (400)
    Error types: (1) *withstack.withStack (2) *errutil.leafError"

Version

v1.0.0-beta11@15c4c09

Are you willing to submit PR?

Code of Conduct

dosubot[bot] commented 1 week ago

Hello, @petercort! I'm here to assist you with any bugs, questions, or contribute to the repository. Let me know how I can help you while we wait for a human maintainer.

To resolve the issue where creating a GitHub App connection does not auto-populate organizations or repositories and does not allow multiselect to add repositories, ensure the following:

  1. Correct Permissions: Verify that the GitHub App has the necessary permissions. The required permissions for collecting data from repositories are:

    • Repository
      • Actions
      • Administration
      • Checks
      • Commit statuses
      • Contents
      • Deployments
      • Issues
      • Metadata
      • Pull requests
    • Organization
      • Members

  2. Endpoint URL: Ensure the Endpoint URL is correctly set to https://api.github.com/ and ends with a /.

  3. Personal Access Token: If using a Personal Access Token (PAT), ensure it has the required permissions:

    • For public repositories:
      • repo:status
      • repo_deployment
      • read:user
      • read:org
    • For private repositories:
      • repo
      • read:user
      • read:org

  4. GraphQL API: If using fine-grained PATs, ensure the Use GraphQL APIs option is disabled as fine-grained PATs do not support GraphQL APIs.

  5. Proxy Settings: If behind a corporate firewall or VPN, ensure the Proxy URL is correctly configured.

  6. Rate Limits: Ensure the rate limit settings do not exceed the maximum allowed by GitHub.

  7. Add Data Scopes: After creating the connection, ensure you add the data scopes correctly. Choose the GitHub repositories you wish to collect either by finding them in the miller column or searching. Note that you can only add public repositories through the search box.

  8. Test and Save Connection: Click Test Connection, if the connection is successful, click Save Connection to add the connection.

If all the above settings are correct and the issue persists, refer to the GitHub Configuration Guide for detailed instructions [1][2][3][4][5].

Additionally, specific error messages or logs can indicate why the organizations or repositories are not being auto-populated. Functions like listGithubOrgRepos, listGithubAppInstalledRepos, and listGithubUserOrgs return errors if they encounter issues during the API calls. These errors are propagated back to the caller, which can then log or display them to indicate the issue [6].

To continue talking to Dosu, mention @dosu.

klesh commented 1 week ago

Can you share the request payload from your browser under the Network tab? Please remove any sensitive information before you post but keep the overall structure.

petercort commented 6 days ago

For the connection being created and scopes loading:

Once the connection is created, I've got 2 calls http://localhost:4000/api/plugins/github/connections/11/scopes?page=1&pageSize=10&blueprints=true Response:

{"count":0,"scopes":[]}

http://localhost:4000/api/plugins/github/connections/11/remote-scopes Response:

{
    "children": [
        {
            "type": "scope",
            "parentId": "petes-fre-shavocado-stand",
            "id": "UID",
            "name": "roadwork",
            "fullName": "petes-fre-shavocado-stand/roadwork",
            "data": null
        }
    ],
    "nextPageToken": ""
}

Multi selecting scope on a manual search

http://localhost:4000/api/plugins/github/connections/11/scopes

payload for adding multiple repos