Error using --new-from-rev in Google Build Cloud

valbuena commented 4 years ago

Thank you for creating the issue!

[x] Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
[x] Yes, I've searched similar issues on GitHub and didn't find any.
[x] Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

```console Already have image: golangci/golangci-lint:v1.26-alpine golangci-lint has version 1.26.0 built from 6bd10d0 on 2020-05-01T15:33:57Z ```

Config file

```console # This file was inspired by the golangci-lint one: # https://github.com/golangci/golangci-lint/blob/master/.golangci.yml linters-settings: govet: check-shadowing: true golint: min-confidence: 0 gocyclo: min-complexity: 10 maligned: suggest-new: true goconst: min-len: 2 min-occurrences: 2 misspell: locale: US lll: line-length: 160 gofmt: simplify: false gocritic: enabled-tags: - diagnostic - experimental - opinionated - performance - style disabled-checks: - wrapperFunc - dupImport # https://github.com/go-critic/go-critic/issues/845 - ifElseChain - octalLiteral - hugeParam funlen: lines: 100 statements: 50 linters: # please, do not use `enable-all`: it's deprecated and will be removed soon. # inverted configuration with `enable-all` and `disable` is not scalable during updates of golangci-lint disable-all: true enable: - bodyclose - deadcode - depguard - dogsled - dupl - errcheck - gochecknoglobals - gochecknoinits - goconst - gocritic - gocyclo - gofmt - goimports - golint - gomnd - goprintffuncname - gosec - gosimple - govet - ineffassign - interfacer - lll - misspell - nakedret - rowserrcheck - scopelint - staticcheck - structcheck - stylecheck - typecheck - unconvert - unparam - unused - varcheck - whitespace - gocognit - maligned - prealloc # don't enable: # - funlen # - godox issues: # Show only new issues: if there are unstaged changes or untracked files, # only those changes are analyzed, else only changes in HEAD~ are analyzed. # It's a super-useful option for integration of golangci-lint into existing # large codebase. It's not practical to fix all existing issues at the moment # of integration: much better don't allow issues in new code. # Default is false. new: false ```

Verbose output of running

``` Status: Downloaded newer image for golangci/golangci-lint:v1.26-alpine docker.io/golangci/golangci-lint:v1.26-alpine level=info msg="[config_reader] Used config file .golangci.yml" level=info msg="[lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]" level=info msg="[lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]" level=info msg="[loader] Go packages loading at mode 575 (types_sizes|compiled_files|deps|exports_file|files|imports|name) took 59.137298537s" level=info msg="[runner/enabledLinters] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]" level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 76.533117ms" level=info msg="[runner/goanalysis_metalinter/goanalysis] analyzers took 6m36.98194291s with top 10 stages: buildssa: 1m35.86414484s, dupl: 18.380915243s, goimports: 13.446273376s, gocritic: 12.763288952s, gosec: 11.04550755s, unconvert: 10.856014233s, golint: 8.033825161s, interfacer: 7.647963528s, gofmt: 7.062299754s, unparam: 7.055950855s" level=info msg="[runner/unused/goanalysis] analyzers took 19.777256292s with top 10 stages: buildssa: 17.437252531s, U1000: 2.340003761s" level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"HEAD~\" \"\": exit status 128" ```

In my local environment this works fine:

golangci-lint run --config .golangci.yml -v --new-from-rev=HEAD~

but in Google Build Cloud I got this error:

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"HEAD~\" \"\": exit status 128"

jirfag commented 4 years ago

Hi, it looks like no git is installed. We need to improve the error message.

valbuena commented 4 years ago

Thank you for your answer @jirfag I thought this so I am using this image: golangci-lint:v1.26-alpine that I thought it has git installed: https://github.com/golangci/golangci-lint/blob/master/build/Dockerfile.alpine.

jirfag commented 4 years ago

it should have a git installed, you are right. Is your repository public?

valbuena commented 4 years ago

No sorry, it is no public.

I tried several configurations:

Using golanci.yml:

  issues:
     new: false
     new-from-rev: HEAD~

Using command line in google build cloud:

#
# Go Linter
#
- name: "golangci/golangci-lint:v1.26-alpine"
  args: ["golangci-lint","run","--config",".golangci.yml", "-v", "--timeout=20m", "--new-from-rev=HEAD~"]]

Running with docker

docker run --rm -v .:/app -w /app golangci/golangci-lint:v1.26.0 golangci-lint run -v --timeout=20m --new-from-rev=HEAD~

But I got the same result, it works in my local environment but it doesn't work in google cloud build.

The error is in this line: https://github.com/golangci/revgrep/blob/276a5c0a103935ee65af49afc254a65335bf1fcf/revgrep.go#L365 So I suppose if git doesn't work fine I had got an error before, here: https://github.com/golangci/revgrep/blob/276a5c0a103935ee65af49afc254a65335bf1fcf/revgrep.go#L337
And I found this guy had the same problem with TravisCI: https://github.com/golangci/golangci-lint/issues/948#issuecomment-583840264.

Thank you in advance for any help :-)

valbuena commented 4 years ago

@jirfag I got a solucion at the end:

I am using this: https://github.com/golangci/golangci-lint-action

But it works fine for me with this configuration:

name: golangci-lint
on:
  pull_request:
jobs:
  golangci:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v1
        with:
          # Required:  the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
          version: v1.26

          # Optional: golangci-lint command line arguments.
          # args: --timeout=20m

          # Optional: show only new issues if it's a pull request. The default value is `false`.
          only-new-issues: true

Because If I use for example push option and args to compare with master or HEAD~: I got again the same error with all the issues in the application:

name: golangci-lint
on:
  push:
jobs:
  golangci:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v1
        with:
          # Required:  the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
          version: v1.26

          # Optional: golangci-lint command line arguments.
          args: --timeout=20m --new-from-rev=master

          # Optional: show only new issues if it's a pull request. The default value is `false`.
          # only-new-issues: true

Error:

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"master\" \"\": exit status 128"

krhubert commented 3 years ago

I'm having the same issue, unfortunately, the repo is not public.

 level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"a32ce..382b\" \"\": exit status 128"

.github/workflows/test.yml


name: test
on: push
env:
  go_version: 1.16.0
jobs:
  backend-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-go@v2
        with:
          go-version: ${{ env.go_version }}

      - uses: actions/checkout@v2

      - name: Run linters
        uses: golangci/golangci-lint-action@v2
        with:
          version: v1.31.0
          skip-go-installation: true
          working-directory: backend/src
          only-new-issues: true

.golangci.yml

output:
  format: github-actions

issues:
  new-from-rev: a32ce..382b

linters:
  disable-all: true
  enable:
    - deadcode
    - errcheck
    - gosimple
    - govet
    - ineffassign
    - staticcheck
    - structcheck
    - typecheck
    - unused
    - varcheck

ldez commented 3 years ago

Some changes has been done in v1.37.0 about that topic, could you try with this version?

krhubert commented 3 years ago

@ldez I've tried v1.37.0 and I've got the same result :/

yash97 commented 3 years ago

facing similar issue any update?

ldez commented 3 years ago

Just to keep the issue in the right way.

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"HEAD~\" \"\": exit status 128"

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"a32ce..382b\" \"\": exit status 128"

These previous error messages can mean:

the command git is missing.
the commits are missing, probably because the git tree is incomplete.

If the command git is missing, you have to install it.

If the commits are missing, you have to configure your CI to get those commits.

With GitHub Actions

You have to define `fetch-depth`, ex: ```yml steps: - name: Check out code uses: actions/checkout@v2 with: fetch-depth: 0 ``` https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches

With Travis

You have to define `git.depth`, ex: ```yml git: depth: false ``` https://docs.travis-ci.com/user/customizing-the-build#git-clone-depth

For the other CI or platform, you have to check the documentation of your CI or platform about git fetching configuration.

To solve this issue, the error message must be improved if possible to get more context.

ymh199478 commented 3 years ago

I modified the configuration of git shallow clone in gitlab to solve this problem.

@ldez is correct. I used to limit the depth of git cloning in order to speed up the clone speed, but this resulted in golangci-lint not being able to obtain enough information (the commits are missing).

Now I will allow the depth of git clone to be 15，However, there may be potential risks in doing so. If you change more than 15 commit, there will be problems.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

OlegSchwann commented 2 years ago

Since it's on the front page of Google, it's a good place to share my experience with https://golangci-lint.run/usage/faq/#how-to-integrate-golangci-lint-into-large-project-with-thousands-of-issues .

Sometimes the command

golangci-lint run --new-from-rev=master

crashes inside the Gitlab CI with the error

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"master\" \"\": exit status 128"

The reason is that GitLab only checkout commits without downloading branches. In order to use the master, you need to download it explicitly

git fetch origin master && git branch master remotes/origin/master

taylorlee commented 2 years ago

:point_up_2: Oleg's comment was very helpful - builds in TeamCity also crash with the same error. Even using teamcity.git.fetchAllHeads=true, I still needed the git branch master remotes/origin/master

wijayaerick commented 2 weeks ago

For Google Cloud Build specifically, you'd need to fetch unshallow. This is because cloudbuild by default performs shallow clone.

  - id: gitFetch
    waitFor: ['-']
    name: 'gcr.io/cloud-builders/git'
    entrypoint: git
    args: ['fetch', '--unshallow', '--quiet']

  - id: lint
    waitFor: ['gitFetch']
    name: 'golangci/golangci-lint:${_GOLANGCI_VERSION}'
    entrypoint: golangci-lint
    args: ['run', '--new-from-rev=master']

For GitLab runners, I think configuring the git strategy would work although I cannot verify it as I don't use GitLab https://docs.gitlab.com/ee/ci/pipelines/settings.html#choose-the-default-git-strategy.

golangci / golangci-lint

Error using --new-from-rev in Google Build Cloud #1088