search

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.55k stars 508 forks source link

[Bug][gitextractor] Error running pipeline beta7 bitbucket server #7949

Closed Capone1983 closed 6 days ago

Capone1983 commented 2 weeks ago

Search before asking

What happened

attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:79 | [...repeated from below...] Wraps: (2) Error running pipeline 43. Wraps: (3) attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.RunTasksStandalone | /app/server/services/task.go:217 | github.com/apache/incubator-devlake/server/services.(*pipelineRunner).runPipelineStandalone.func1 | /app/server/services/pipeline_runner.go:42 | github.com/apache/incubator-devlake/core/runner.runPipelineTasks | /app/core/runner/run_pipeline.go:90 | github.com/apache/incubator-devlake/core/runner.RunPipeline | /app/core/runner/run_pipeline.go:54 | github.com/apache/incubator-devlake/server/services.(*pipelineRunner).runPipelineStandalone | /app/server/services/pipeline_runner.go:38 | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:76 | github.com/apache/incubator-devlake/server/services.RunPipelineInQueue.func1 | /app/server/services/pipeline.go:347 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1598 Wraps: (4) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:189 | Wraps: (2) Error running task 5403. | Wraps: (3) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:329 | | [...repeated from below...] | Wraps: (4) subtask Clone Git Repo ended unexpectedly | Wraps: (5) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:238 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCommandIn | | /app/plugins/gitextractor/parser/clone_gitcli.go:227 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:159 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).execGitCloneCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:128 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(*GitcliCloner).CloneRepo | | /app/plugins/gitextractor/parser/clone_gitcli.go:92 | | github.com/apache/incubator-devlake/plugins/gitextractor/tasks.CloneGitRepo | | /app/plugins/gitextractor/tasks/repo_cloner.go:57 | | github.com/apache/incubator-devlake/core/runner.runSubtask | | /app/core/runner/run_task.go:405 | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:327 | | github.com/apache/incubator-devlake/core/runner.RunPluginTask | | /app/core/runner/run_task.go:163 | | github.com/apache/incubator-devlake/core/runner.RunTask | | /app/core/runner/run_task.go:137 | | github.com/apache/incubator-devlake/server/services.runTaskStandalone | | /app/server/services/task_runner.go:113 | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:187 | | runtime.goexit | | /usr/local/go/src/runtime/asm_amd64.s:1598 | Wraps: (6) git cmd [git -c http.sslVerify=false clone https://********:*********@tools.standard.co.za/bitbucket/scm/ddsa/account-analysis-ds.git /tmp/gitextractor3340126072 --depth=1 --bare] in failed: unknown option: -c http.sslVerify=false | | usage: git [--version] [--help] [-C <path>] [-c <name>=<value>] | | [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path] | | [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare] | | [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>] | | <command> [<args>] | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

What do you expect to happen

The repo to get cloned

How to reproduce

Use beta7 and try to run a pipeline that has a connection to a bitbucket server repos

Anything else

linked to defect #7842 but different error

Version

v1.0.1-beta7@3f84bae

Are you willing to submit PR?

Code of Conduct

dosubot[bot] commented 2 weeks ago

I found two similar closed issues that might be relevant to your problem:

  1. [Bug][Git Extractor] Git Extractor task failed in a GitHub blueprint: This issue was resolved by changing the Docker base image [1].
  2. [Bug][GitExtractor] cloneGitRepo ended unexpectedly: reference delta not found: This issue was resolved by updating the go-git module [2].

To resolve the error when running a pipeline with gitextractor on beta7 for a Bitbucket server repository, ensure that the gitextractor stage is correctly added to the pipeline plan. Here is the relevant part of the code:

// add gitex stage
if utils.StringsContains(scopeConfig.Entities, plugin.DOMAIN_TYPE_CODE) {
    cloneUrl, err := errors.Convert01(url.Parse(repo.CloneUrl))
    if err != nil {
        return nil, err
    }
    cloneUrl.User = url.UserPassword(connection.Username, connection.Password)
    stage = append(stage, &coreModels.PipelineTask{
        Plugin: "gitextractor",
        Options: map[string]interface{}{
            "url":    cloneUrl.String(),
            "name":   repo.BitbucketId,
            "repoId": didgen.NewDomainIdGenerator(&models.BitbucketServerRepo{}).Generate(connection.ID, repo.BitbucketId),
            "proxy":  connection.Proxy,
        },
    })
}

Ensure that scopeConfig.Entities includes plugin.DOMAIN_TYPE_CODE and that the cloneUrl is correctly formed with the necessary credentials. If you encounter a 403 error, it is usually caused by the permission of your tokens. Make sure you are using a supported authentication method and that your token has the necessary permissions.

Additionally, consider upgrading to v0.15.x as it includes fixes for some known issues with the gitextractor task [3][4][5][6][7].

For the specific permissions required for the token used in the gitextractor task for Bitbucket server repositories, ensure the token has the following permissions:

Regarding the http.sslVerify option, it is handled by setting the environment variable GIT_SSL_NO_VERIFY to true if the IN_SECURE_SKIP_VERIFY configuration is enabled. However, there are known issues with self-signed certificates. A better approach would be to add your root CA to the devlake container [9][4][10].

If the issue persists, please provide more details or consider upgrading to a newer version.

To continue talking to Dosu, mention @dosu.

klesh commented 2 weeks ago

7944 should fix the problem. Will release a new beta this week.

klesh commented 6 days ago

Closing due to inactivity.