search

jfrog / jfrog-cli

JFrog CLI is a client that provides a simple interface that automates access to the JFrog products.
https://www.jfrog.com/confluence/display/CLI/JFrog+CLI
Apache License 2.0
531 stars 232 forks source link

Go publish checksums are different for packages with /vendor/ files. #598

Open dnozay opened 4 years ago

dnozay commented 4 years ago

Could be related to checksum issues in #376 but here's another issue just in case it's different.

main.go:

package main

import (
    "github.com/fatih/color"
)

func main() {
    c := color.New(color.FgCyan).Add(color.Underline)
    c.Println("Prints cyan text with an underline.")
}

go.mod

module github.com/confluentinc/test

no issues with:

$ go build main.go

This populates go.sum with the "correct" or "expected" checksum. Then take a look at the cached zip:

$ unzip -l ~/go/pkg/mod/cache/download/github.com/fatih/color/@v/v1.9.0.zip
Archive:  v1.9.0.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      448  00-00-1980 00:00   github.com/fatih/color@v1.9.0/.github/workflows/go.yml
     1079  00-00-1980 00:00   github.com/fatih/color@v1.9.0/LICENSE.md
     4930  00-00-1980 00:00   github.com/fatih/color@v1.9.0/README.md
    18321  00-00-1980 00:00   github.com/fatih/color@v1.9.0/color.go
     8491  00-00-1980 00:00   github.com/fatih/color@v1.9.0/color_test.go
     4397  00-00-1980 00:00   github.com/fatih/color@v1.9.0/doc.go
      126  00-00-1980 00:00   github.com/fatih/color@v1.9.0/go.mod
      753  00-00-1980 00:00   github.com/fatih/color@v1.9.0/go.sum
      209  00-00-1980 00:00   github.com/fatih/color@v1.9.0/vendor/modules.txt
---------                     -------
    38754                     9 files

Then blow away the ~/go/pkg directory to force download.

$ jfrog --version
jfrog version 1.33.2

Retry to build using artifactory this time:

$ jfrog rt go build main.go
[Info] Using go: go version go1.12.9 darwin/amd64

go: downloading github.com/fatih/color v1.9.0
verifying github.com/fatih/color@v1.9.0: checksum mismatch
    downloaded: h1:nHHU1ZN+sfpuT5HY2MMc2eOyCPq2z+PI/Wz2WB+7O10=
    go.sum:     h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
exit status 1

Inspect the cache:

$ unzip -l ~/go/pkg/mod/cache/download/github.com/fatih/color/@v/v1.9.0.zip067280972.tmp
Archive:  v1.9.0.zip067280972.tmp
  Length      Date    Time    Name
---------  ---------- -----   ----
      448  02-18-2020 22:48   github.com/fatih/color@v1.9.0/.github/workflows/go.yml
     1079  02-18-2020 22:48   github.com/fatih/color@v1.9.0/LICENSE.md
     4930  02-18-2020 22:48   github.com/fatih/color@v1.9.0/README.md
    18321  02-18-2020 22:48   github.com/fatih/color@v1.9.0/color.go
     8491  02-18-2020 22:48   github.com/fatih/color@v1.9.0/color_test.go
     4397  02-18-2020 22:48   github.com/fatih/color@v1.9.0/doc.go
      126  02-18-2020 22:48   github.com/fatih/color@v1.9.0/go.mod
      753  02-18-2020 22:48   github.com/fatih/color@v1.9.0/go.sum
      204  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/.travis.yml
     1085  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/LICENSE
     1374  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/README.md
      625  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/colorable_appengine.go
      645  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/colorable_others.go
    23997  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/colorable_windows.go
       80  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/go.mod
...

I think it is still tripping on things with vendor files... https://github.com/jfrog/jfrog-cli/blob/master/artifactory/utils/golang/project/archive.go#L95

jackwhelpton commented 4 years ago

Synchronicity... as the raiser of that original issue, yep, I've just started seeing this too. In my case it's an indirect reference to https://github.com/golang/gddo that reveals the problem: I've cloned the repo, and compared it to what's in my module cache. The only difference is the vendor directory.

Or-Geva commented 4 years ago

@dnozay & @jackwhelpton , Thank you for your report! this issue is known and related to Artifactory remote repository, which fails to create standard zip files. However, If you will publish "github.com/fatih/color" using jfrog-cli to your local repo (including transitive dependencies) and then resolve, you will not get an error mismatch. Artifactory 7.3.0/6.18.0 coming out soon, should solve the problem.

dnozay commented 4 years ago

this issue is known and related to Artifactory remote repository, which fails to create standard zip files. However, If you will publish "github.com/fatih/color" using jfrog-cli to your local repo (including transitive dependencies) and then resolve, you will not get an error mismatch.

In my case, it's about using jfrog rt go as a drop-in replacement for go, which I can't do because we have a lot of repos with go.sum and it's just hitting mismatches after mismatches.

Artifactory 7.3.0/6.18.0 coming out soon, should solve the problem.

Thanks for the update.

Or-Geva commented 4 years ago

@dnozay, Do you publish github.com/fatih/color to artifactory or just use remote\virtual to resolve your dependencies? what is your artifactory version?

jackwhelpton commented 4 years ago

@dnozay: that's pretty much our set-up too. This is a build occurring as part of a CI pipeline, and is using remote\virtual to resolve the dependencies. Our build slaves have a go.yaml config file which specifies a resolver and a deployer, both of which currently point to the remote\virtual repo.

That does raise a question: once the fix you mention is released, is this the recommended configuration, or should we be resolving via the remote\virtual repo but publishing to the local one?

dnozay commented 4 years ago 
jfrog rt curl -XGET /api/system/version
{
  "version" : "6.15.1",
  "revision" : "61501900",
  "addons" : [ "aol", "ha", "build", "docker", "vagrant",  ... , "go", "helm", "rest", "conda", "license", "puppet", "ldap", "sso", "layouts", "properties", ... ],
  "license" : "XXXXXXXXXXXXXXX"
}%
version: 1
type: go
resolver:
  repo: go
  serverId: xxxxx
deployer:
  repo: go-internal
  serverId: xxxxx

go-internal is local, go is virtual

BarakHc commented 4 years ago

@dnozay Can you please write us down what are the remote repositories under your virtual go repository?

dnozay commented 4 years ago

@dnozay Can you please write us down what are the remote repositories under your virtual go repository?

go includes go-internal , our company / private one, a bunch of other public ones (golang.org,...)

BarakHc commented 4 years ago

@dnozay Got it. So as a workaround you should do the following:

  1. If one of the public ones is github.com, please remove it from the virtual repo until Artifactory 7.3.0/6.18.0 will be out.
  2. Clean the remote cache of the remote repo pointing to gocenter.io. (You can also remove only "github.com/fatih/color" if you need to keep the rest).
  3. Zap the virtual go repo caches.
eyalbe4 commented 4 years ago

@dnozay, The latest release of Artifactory includes this fix. You may need clean the remote repository's cache once. Any feedback that you can provide would greatly appreciated.

dnozay commented 4 years ago

The latest release of Artifactory includes this fix. You may need clean the remote repository's cache once.

Can you please provide specific version? thank you

eyalbe4 commented 4 years ago

Sure @dnozay, Both versions 6.18.0 and 7.3.0 of Artifactory include this fix. Please let me know if you have additional questions.

dnozay commented 4 years ago 
$ jfrog --version
jfrog version 1.35.0

Both versions 6.18.0 and 7.3.0 of Artifactory include this fix. Please let me know if you have additional questions.

we have 6.18.1 which I guess should include the fix...

If one of the public ones is github.com, please remove it from the virtual repo until Artifactory 7.3.0/6.18.0 will be out. Clean the remote cache of the remote repo pointing to gocenter.io. (You can also remove only "github.com/fatih/color" if you need to keep the rest). Zap the virtual go repo caches.

image

try again: 

$ JFROG_CLI_LOG_LEVEL=DEBUG jfrog rt  go build main.go

[Debug] Go config file was found in: /Users/XXXXXXX/.jfrog/projects/go.yaml
[Debug] Preparing to read the config file /Users/XXXXXXX/.jfrog/projects/go.yaml
[Debug] Found resolver in the config file /Users/XXXXXXX/.jfrog/projects/go.yaml
[Debug] Found deployer in the config file /Users/XXXXXXX/.jfrog/projects/go.yaml
[Debug] Sending usage info...
[Info] Using go: go version go1.12.7 darwin/amd64

[Debug] Initializing protocol regexp
[Debug] Initializing not found regexp
[Debug] Initializing not found go 1.13 regexp
[Debug] Initializing unrecognized import path regexp
[Debug] Initializing unknown revision regexp
[Debug] Initializing not found zip file
[Debug] Sending HTTP GET request to: https://XXXXXXX.jfrog.io/XXXXXXX/api/system/version
go: downloading github.com/fatih/color v1.9.0
[Debug] The Artifactory version is: 6.18.1
[Debug] Sending HTTP POST request to: https://XXXXXXX.jfrog.io/XXXXXXX/api/system/usage
[Debug] Artifactory response: 200 OK
[Debug] Usage info sent successfully.
verifying github.com/fatih/color@v1.9.0: checksum mismatch
    downloaded: h1:nHHU1ZN+sfpuT5HY2MMc2eOyCPq2z+PI/Wz2WB+7O10=
    go.sum:     h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
exit status 1

as mentioned previously the vendor files are in the zip.

Archive:  /Users/XXXXXXX/go/pkg/mod/cache/download/github.com/fatih/color/@v/v1.9.0.zip361465003.tmp
  Length      Date    Time    Name
---------  ---------- -----   ----
      448  02-18-2020 22:48   github.com/fatih/color@v1.9.0/.github/workflows/go.yml
     1079  02-18-2020 22:48   github.com/fatih/color@v1.9.0/LICENSE.md
     4930  02-18-2020 22:48   github.com/fatih/color@v1.9.0/README.md
    18321  02-18-2020 22:48   github.com/fatih/color@v1.9.0/color.go
     8491  02-18-2020 22:48   github.com/fatih/color@v1.9.0/color_test.go
     4397  02-18-2020 22:48   github.com/fatih/color@v1.9.0/doc.go
      126  02-18-2020 22:48   github.com/fatih/color@v1.9.0/go.mod
      753  02-18-2020 22:48   github.com/fatih/color@v1.9.0/go.sum
      204  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/.travis.yml
     1085  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/LICENSE
     1374  02-18-2020 22:48   github.com/fatih/color@v1.9.0/vendor/github.com/mattn/go-colorable/README.md
...

in comparison, the zip downloaded by go build

Archive:  /Users/XXXXXXX/go/pkg/mod/cache/download/github.com/fatih/color/@v/v1.9.0.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      448  00-00-1980 00:00   github.com/fatih/color@v1.9.0/.github/workflows/go.yml
     1079  00-00-1980 00:00   github.com/fatih/color@v1.9.0/LICENSE.md
     4930  00-00-1980 00:00   github.com/fatih/color@v1.9.0/README.md
    18321  00-00-1980 00:00   github.com/fatih/color@v1.9.0/color.go
     8491  00-00-1980 00:00   github.com/fatih/color@v1.9.0/color_test.go
     4397  00-00-1980 00:00   github.com/fatih/color@v1.9.0/doc.go
      126  00-00-1980 00:00   github.com/fatih/color@v1.9.0/go.mod
      753  00-00-1980 00:00   github.com/fatih/color@v1.9.0/go.sum
      209  00-00-1980 00:00   github.com/fatih/color@v1.9.0/vendor/modules.txt
---------                     -------
    38754                     9 files

I provided detailed instructions to repro, so can you please detail how you verified the fix? Maybe I'm missing something.

Or-Geva commented 4 years ago

@dnozay, I have tried to reproduce your steps on my Artifactory (7.2.0) and I didn't have any checksum mismatch. Could you please create new local, remote & virtual repos for Go and start over https://github.com/jfrog/jfrog-cli/issues/598#issue-567214615? In that case, we ensure we don't have any cache interference.

dnozay commented 4 years ago

@Or-Gabay Which go version are you using for your repro?

 ...
[Info] Using go: go version go1.12.7 darwin/amd64

[Debug] Initializing protocol regexp
[Debug] Initializing not found regexp
[Debug] Initializing not found go 1.13 regexp
...

Could you please create new local, remote & virtual repos for Go and start over #598 (comment)? In that case, we ensure we don't have any cache interference.

I will report back when I get a chance to try that.

Or-Geva commented 4 years ago

@dnozay , I'm using 1.14, However, I tried to reproduce https://github.com/jfrog/jfrog-cli/issues/598 with go1.12.7 darwin/amd64 and everything works great. Let me know what was the result regarding this.

Could you please create new local, remote & virtual repos for Go and start over #598 (comment)? In that case, we ensure we don't have any cache interference.