Failed while attempting to download JFrog CLI (1.42.3) - no on-premise

philstussak commented 3 years ago

Describe the bug Hey all, Our builds interacting with Artifactory (company Artifactory installation) fail since 2021-04-11 because of an issue with the Artifactory Maven Azure DevOps task.

Build job 1: Simple generic Maven build

2021-04-12T13:40:54.9840790Z ##[section]Starting: Build Maven project 2021-04-12T13:40:54.9854502Z ============================================================================== 2021-04-12T13:40:54.9854801Z Task : Artifactory Maven 2021-04-12T13:40:54.9855296Z Description : Build with Apache Maven, while resolving dependencies and deploying artifacts from and to Artifactory and while allowing to collect build-info. The collected build-info can be later published to Artifactory by the "Artifactory Publish Build Info" task. 2021-04-12T13:40:54.9855767Z Version : 2.11.2 2021-04-12T13:40:54.9855953Z Author : JFrog 2021-04-12T13:40:54.9856410Z Help : [More Information](https://www.jfrog.com/confluence/display/RTF/Maven+Repository) 2021-04-12T13:40:54.9856740Z ============================================================================== 2021-04-12T13:40:55.2091588Z Downloading: https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64 2021-04-12T13:40:56.6322107Z ##[error]Error occurred while executing task: Failed while attempting to download JFrog CLI from https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64. If this build agent cannot access the internet, you may use the 'Artifactory Tools Installer' task, to download JFrog CLI through an Artifactory repository, which proxies https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64. You may also manually download version 1.42.3 of JFrog CLI and place it on the agent in the following path: /opt/hostedtoolcache/_jfrog/current/jfrog Error: Unexpected HTTP response: 502 2021-04-12T13:40:56.6336099Z ##[section]Finishing: Build Maven project

Build job 2: Release pipeline downloading an Artifactory artefact

2021-04-12T14:21:42.8638226Z ##[section]Starting: Download artifact - SERVICENAME (Artifactory) 2021-04-12T14:21:42.8834771Z ============================================================================== 2021-04-12T14:21:42.8835635Z Task : Artifactory Generic Download 2021-04-12T14:21:42.8836198Z Description : Download artifacts from Artifactory using FileSpecs, while allowing to collect build-info. The collected build-info can be later published to Artifactory by the "Artifactory Publish Build Info" task. 2021-04-12T14:21:42.8836593Z Version : 3.11.2 2021-04-12T14:21:42.8836919Z Author : JFrog 2021-04-12T14:21:42.8837328Z Help : 2021-04-12T14:21:42.8837579Z ============================================================================== 2021-04-12T14:21:43.6509974Z Downloading: https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64 2021-04-12T14:21:43.6537990Z ##[error]Error occurred while executing task: Failed while attempting to download JFrog CLI from https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64. If this build agent cannot access the internet, you may use the 'Artifactory Tools Installer' task, to download JFrog CLI through an Artifactory repository, which proxies https://api.bintray.com/content/jfrog/jfrog-cli-go/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64. You may also manually download version 1.42.3 of JFrog CLI and place it on the agent in the following path: /opt/hostedtoolcache/_jfrog/current/jfrog Error: Unexpected HTTP response: 403 2021-04-12T14:21:43.6576596Z ##[section]Finishing: Download artifact - SERVICENAME(Artifactory)

Both builds are failing in the according task.

To Reproduce Build failure:

Create new AzureDevOps pipeline
Add task 'Artifactory Maven'
Select pom.file

Current azure pipeline yaml task configuration: `[...]

pool: vmImage: 'ubuntu-latest'

[...]

task: ArtifactoryMaven@2 displayName: 'Build Maven project' inputs: mavenPomFile: 'pom.xml' goals: 'install' artifactoryResolverService: ... targetResolveReleaseRepo: ... targetResolveSnapshotRepo: ... artifactoryDeployService: ... targetDeployReleaseRepo: ... targetDeploySnapshotRepo: ... collectBuildInfo: true buildName: '$(Build.DefinitionName)' buildNumber: '$(Build.BuildNumber)'

[...]`

Release failure:

Create new Azure DevOps Release pipeline
Select "Artifactory" as artefact source
Configure build available in Artifactory

Expected behavior We expect the build pipelines to be able to download the frog-cli-go when using the Artifactory Maven build step or when using the Artifactory artefact source in Release Pipelines.

Versions

Artifactory Azure DevOps extension version: 1.11.2
Agent operating system: ubuntu-latest
JFrog Artifactory version: 7.7.8

Additional context

Artifactory Service connection was verified and is working from Project Settings -> Service Connection
Running on standard Azure DevOps Cloud. Build agent has internet connection and connection to Artifactory server.

Seidel-Michael commented 3 years ago

Same issue here. I would assume this is because of the currently scheduled maintenance. Quite annoying. https://status.bintray.com/incidents/4mzph0y8l6hk

philstussak commented 3 years ago

Didn't see this scheduled maintenance, seems like this is probably the cause. I will close this issue after the end of the maintenance if things are working again. Thanks for the hint!

Seidel-Michael commented 3 years ago

This might be a permanent issue after May 1st.

eyalbe4 commented 3 years ago

I apologize for this! We'll soon release patch, that will permanently change the download URL to releases.jfrog.io.

chatelain-io commented 3 years ago

ArtifactoryToolsInstaller task as a workaround

adamko147 commented 3 years ago

ArtifactoryToolsInstaller seems not available in yaml based build pipelines, only on Release pipelines... or am I missing something? Thanks!

eyalbe4 commented 3 years ago

We've just released patch 1.11.4 with the fix. We're looking forward to your feedback.

philstussak commented 3 years ago

Thanks, plugin is available and I got the new version in DevOps. Unfortunately I still get an HTTP 403 when the task is trying to download the build-info-extractor:

2021-04-12T17:04:32.5161887Z ##[section]Starting: Build Maven project 2021-04-12T17:04:32.5171694Z ============================================================================== 2021-04-12T17:04:32.5172042Z Task : Artifactory Maven 2021-04-12T17:04:32.5172662Z Description : Build with Apache Maven, while resolving dependencies and deploying artifacts from and to Artifactory and while allowing to collect build-info. The collected build-info can be later published to Artifactory by the "Artifactory Publish Build Info" task. 2021-04-12T17:04:32.5173211Z Version : 2.11.4 2021-04-12T17:04:32.5173470Z Author : JFrog 2021-04-12T17:04:32.5173803Z Help : [More Information](https://www.jfrog.com/confluence/display/RTF/Maven+Repository) 2021-04-12T17:04:32.5174289Z ============================================================================== 2021-04-12T17:04:32.7949859Z Downloading: https://releases.jfrog.io/artifactory/jfrog-cli/v1/1.42.3/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64 2021-04-12T17:04:33.4712216Z Caching tool: jfrog 1.42.3 x64 2021-04-12T17:04:33.4955954Z Running jfrog-cli from /opt/hostedtoolcache/jfrog/1.42.3/x64/jfrog. 2021-04-12T17:04:33.5070313Z JFrog CLI version: 1.42.3 2021-04-12T17:04:33.5074628Z M2_HOME is not defined. Retrieving Maven home using mvn --version. 2021-04-12T17:04:33.9006156Z The Maven home location: /usr/share/apache-maven-3.6.3 2021-04-12T17:04:33.9095246Z [Info] Encrypting password... 2021-04-12T17:04:34.1211954Z [Info] Encrypting password... 2021-04-12T17:04:34.2975722Z [Info] maven build config successfully created. 2021-04-12T17:04:34.3070609Z [Info] Running Mvn... 2021-04-12T17:04:34.3072324Z [Info] Downloading bintray/jcenter/org/jfrog/buildinfo/build-info-extractor-maven3/2.20.0/build-info-extractor-maven3-2.20.0-uber.jar 2021-04-12T17:04:34.5041427Z [Error] 403 Forbidden 2021-04-12T17:04:34.5112365Z ##[error]Error: Command failed: /opt/hostedtoolcache/jfrog/1.42.3/x64/jfrog rt mvn install -f /home/vsts/work/1/s/pom.xml --build-name="x" --build-number="20210412.6" 2021-04-12T17:04:34.5348272Z ##[section]Finishing: Build Maven project

Seems like there are a few additional Bintray download URLs.

chatelain-io commented 3 years ago

ArtifactoryToolsInstaller seems not available in yaml based build pipelines, only on Release pipelines... or am I missing something? Thanks!

Yes it available:

`

task: ArtifactoryToolsInstaller@1 inputs: artifactoryService: '< artifactory service connection name >' cliInstallationRepo: '< artifactory jfrog cli repository >' `

Ref:

https://www.jfrog.com/confluence/display/JFROG/Artifactory+Azure+DevOps+Extension#ArtifactoryAzureDevOpsExtension-ArtifactoryToolsInstaller

You must have a remote repository to https://jfrog.bintray.com/jfrog-cli-go/ in your artifactory

eyalbe4 commented 3 years ago

@philstussak - we're working on fixing the build-info-extractor download as well.

eyalbe4 commented 3 years ago

Patch 1.11.5 is released and should resolve all issues. For those who are using the using the Artifactory Tools Installer task, we recommend modifying the URLs of the two remote Artifactory repositories as described here. Looking forward to everyone's feedback.

philstussak commented 3 years ago

Thank you! Build works fine now, however, during release when downloading the artefact, there is the following issue "panic: runtime error: invalid memory address or nil pointer dereference" See below:

021-04-13T06:51:22.1061670Z ##[section]Starting: Download artifact - _x (Artifactory) 2021-04-13T06:51:22.1307583Z ============================================================================== 2021-04-13T06:51:22.1308783Z Task : Artifactory Generic Download 2021-04-13T06:51:22.1309613Z Description : Download artifacts from Artifactory using FileSpecs, while allowing to collect build-info. The collected build-info can be later published to Artifactory by the "Artifactory Publish Build Info" task. 2021-04-13T06:51:22.1310134Z Version : 3.11.5 2021-04-13T06:51:22.1310659Z Author : JFrog 2021-04-13T06:51:22.1311393Z Help : 2021-04-13T06:51:22.1311695Z ============================================================================== 2021-04-13T06:51:23.0410126Z Downloading: https://releases.jfrog.io/artifactory/jfrog-cli/v1/1.46.1/jfrog-cli-linux-amd64/jfrog?bt_package=jfrog-cli-linux-amd64 2021-04-13T06:51:23.1463336Z Caching tool: jfrog 1.46.1 x64 2021-04-13T06:51:23.1787784Z Running jfrog-cli from /opt/hostedtoolcache/jfrog/1.46.1/x64/jfrog. 2021-04-13T06:51:23.1875264Z JFrog CLI version: 1.46.1 2021-04-13T06:51:23.1915853Z Artifact source download... 2021-04-13T06:51:23.3335442Z [Info] Searching items to download... 2021-04-13T06:51:23.3705702Z panic: runtime error: invalid memory address or nil pointer dereference 2021-04-13T06:51:23.3707082Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xa681cc] 2021-04-13T06:51:23.3707578Z 2021-04-13T06:51:23.3708069Z goroutine 8 [running]: 2021-04-13T06:51:23.3709406Z github.com/jfrog/jfrog-client-go/utils/io/content.(*ContentReader).NextRecord(0x0, 0xd9eca0, 0xc0002939d0, 0xc000580000, 0x103) 2021-04-13T06:51:23.3710972Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/io/content/contentreader.go:67 +0x5c 2021-04-13T06:51:23.3712432Z github.com/jfrog/jfrog-client-go/utils/io/content.splitReaderToSortedBufferSizeReadersByCalculatedKey(0x0, 0xc0001059a0, 0xc000145c01, 0xc000145cc0, 0xc000145d10, 0xc000145d60, 0x2, 0xc000145c20) 2021-04-13T06:51:23.3714254Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/io/content/contentreader.go:288 +0x1c2 2021-04-13T06:51:23.3715464Z github.com/jfrog/jfrog-client-go/utils/io/content.SortContentReaderByCalculatedKey(0x0, 0xc0001059a0, 0x1, 0x0, 0x0, 0x0) 2021-04-13T06:51:23.3718118Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/io/content/contentreader.go:271 +0x98 2021-04-13T06:51:23.3719561Z github.com/jfrog/jfrog-client-go/artifactory/services.(*DownloadService).produceTasks(0xc000287cc0, 0x0, 0xc000138c60, 0x10001, 0x1400, 0x3, 0x3, 0x10fca60, 0xc000101f50, 0xc000197710, ...) 2021-04-13T06:51:23.3720766Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/download.go:199 +0x163 2021-04-13T06:51:23.3722061Z github.com/jfrog/jfrog-client-go/artifactory/services.(*DownloadService).prepareTasks.func1(0x10fca60, 0xc000101f50, 0xc000072150, 0xc000287cc0, 0xc00000e508, 0xc000101f20, 0x1, 0x1, 0xc0002bdce0, 0x3, ...) 2021-04-13T06:51:23.3734159Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/download.go:160 +0x648 2021-04-13T06:51:23.3736089Z created by github.com/jfrog/jfrog-client-go/artifactory/services.(*DownloadService).prepareTasks 2021-04-13T06:51:23.3736938Z /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/download.go:113 +0xc5 2021-04-13T06:51:23.3797214Z ##[error]Error: Command failed: /opt/hostedtoolcache/jfrog/1.46.1/x64/jfrog rt dl "*" "/home/vsts/work/r1/a/x/" --build="x/20201112.2" --flat --fail-no-op --url="https://artifactory.boschdevcloud.com/artifactory" --user=*** --password=*** 2021-04-13T06:51:23.3871095Z ##[section]Finishing: Download artifact - _x (Artifactory)

lindhe commented 3 years ago

@eyalbe4 Hey this "fix" is breaking our CI. I guess you weren't aware, but every single commit is being made available to Azure DevOps pipelines and there's barely any control over it from our side. So when you push things to master, it's forced upon us without any notice really.

For those who are using the using the Artifactory Tools Installer task, we recommend modifying the URLs of the two remote Artifactory repositories as described here.

We followed instructions from the official documentation before and that told us to use https://jfrog.bintray.com/jfrog-cli-go/ as the URL for the remote repo, and since 1.46.1 does not exist there everything broke with your patch. But in the docs you link to it says to use https://releases.jfrog.io/artifactory/jfrog-cli/v1/ instead for the remote repo, and there we find 1.46.1. Has this changed recently? How were we supposed to know that?

This is the umpteenth time you guys screw up the Azure CLI tasks in a majorly breaking way for us. We are moving away from JFrog because of this.

eyalbe4 commented 3 years ago

@philstussak - we're looking into the panic you receieved. We'll send out an update here shortly.

eyalbe4 commented 3 years ago

@philstussak - we're starting to get better understanding of this issue. The command generated by the task looks like this:

jfrog rt dl "*" "/home/vsts/work/r1/a/x/" --build="gradle-cli/1" --flat --fail-no-op

We think that the panic is thrown in the case where this specific build does not exist. It looks like we missed this scenario while implementing the recent changes in JFrog CLI. In any case, we're working on a patch (for both JFrog CLI and Azure DevOps), and are aiming to release it soon. We'll post here anither update soon.

philstussak commented 3 years ago

Thanks for the update, I will check the missing build issue. The selected build is available in artifactory and can be selected from azure DevOps release pipeline task.

eyalbe4 commented 3 years ago

@philstussak, Are you sure you can see build x with build number 20201112.2 in Artifactory? This is the build provided in to your Artifactory Generic Download task according to the log you provided.

svetoslavs commented 3 years ago

We are experiencing a problem with upload as well (Artifactory Generic Upload task) in our pipelines. It is probably related.

[Error] refname "214f451aba1053be608f2096b71f90d7584a5789" is ambiguous
[Error] refname "214f451aba1053be608f2096b71f90d7584a5789" is ambiguous
{
  "status": "failure",
[Error] Upload finished with errors, Please review the logs.
  "totals": {
    "success": 0,
    "failure": 0
  }
}

eyalbe4 commented 3 years ago

@svetoslavs, I winder what refname "214f451aba1053be608f2096b71f90d7584a5789" is ambiguous means... To troubleshoot this, will you be able to copy the upload command from the logs, and run it from your terminal? If you can't, it's not a must, just may be easier to run multiple times if needed. In any case, if you sent the JFROG_CLI_ERROR_HANDLING environment variable with paniv as its value, the jfrog-cli will panic, and print the stacktrace. This may help us understand what this error means. Also, can you please share the complete JFrog CLI command from the task log?

philstussak commented 3 years ago

@eyalbe4 I created a new build just now to be sure it's available and not "outdated". The artefact name ("service name") contains a space which was encoded to %20 ("service%20name") when selecting the build in the release pipeline artefact source configuration. That was probably the reason that the build was not found. After manually adding a space, changing the build name back to "service name", the artefact can be downloaded.

All my issues are resolved now, thank you so much. I leave it up to you to close this as seen fit because of the open points of svetoslavs.

eyalbe4 commented 3 years ago

Thanks @philstussak for sharing this information. I'd like thank everyone for assisting and providing helpful information, which helped us resolve the issues. I also want to apologise for the inconvenience caused by this issue. Since all issues discussed in this issue are resolved, I'm closing it now.

svetoslavs commented 3 years ago

@eyalbe4 the id "214f451aba1053be608f2096b71f90d7584a5789" is the id of the last commit in the repo on which we are running the build.

This is what we got when setting "JFROG_CLI_ERROR_HANDLING" to "panic":

goroutine 23 [running]:
github.com/jfrog/jfrog-cli-core/utils/coreutils.PanicOnError(0x11046c0, 0xc00089c3c0, 0x28, 0x18bf900)
    /root/go/pkg/mod/github.com/jfrog/jfrog-cli-core@v1.5.0/utils/coreutils/utils.go:85 +0x60
github.com/jfrog/jfrog-client-go/utils.(*manager).readMessage(0xc0000b3958)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/git.go:294 +0x16c
github.com/jfrog/jfrog-client-go/utils.(*manager).ReadConfig(0xc0000b3958, 0x2, 0x2)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/git.go:46 +0x8f
github.com/jfrog/jfrog-client-go/utils.extractGitDetails(0xc000814080, 0x2, 0x0, 0xc000814101, 0x6, 0xde26c0, 0xc0000b3a28, 0x0, 0x0, 0xc000814100)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/vcsdetails.go:103 +0xef
github.com/jfrog/jfrog-client-go/utils.tryGetGitDetails(0xc000814080, 0x2, 0x2, 0x0, 0x0, 0xc000814080, 0x2, 0x0, 0x0, 0x0)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/vcsdetails.go:96 +0xca
github.com/jfrog/jfrog-client-go/utils.(*VcsCache).GetVcsDetails(0xc00006a120, 0xc000814080, 0xf, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/utils/vcsdetails.go:65 +0x1fd
github.com/jfrog/jfrog-client-go/artifactory/services.getVcsProps(0xc000886000, 0x44, 0xc00006a120, 0x60, 0x0, 0x0, 0xc00042ab40)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/upload.go:825 +0x96
github.com/jfrog/jfrog-client-go/artifactory/services.collectFilesForUpload(0xc00042ab40, 0x0, 0x0, 0xc0005f0af0, 0x63, 0x1010000, 0x3, 0x2800, 0x0, 0x0, ...)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/upload.go:240 +0x5a2
github.com/jfrog/jfrog-client-go/artifactory/services.(*UploadService).prepareUploadTasks.func1(0x111f4c0, 0xc00064a3c0, 0xc0005c9cc0, 0x1, 0x1, 0xc0000be208, 0xc0005c9d10, 0xc00064a390)
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/upload.go:119 +0x251
created by github.com/jfrog/jfrog-client-go/artifactory/services.(*UploadService).prepareUploadTasks
    /root/go/pkg/mod/github.com/jfrog/jfrog-client-go@v0.21.0/artifactory/services/upload.go:103 +0x9c

Tried with both V1 and V2 of the Generic Upload task, the result was the same. This is the command executed in the log for V1

##[error]Error: Command failed: C:\hostedtoolcache\windows\jfrog\1.46.1\x64\jfrog.exe rt u --url="<--my-url-here-->" --spec="D:\a\1\s\uploadSpec1618309752855.json" --user=*** --password=***

This is the command executed in the log for V2

##[error]Error: Command failed: C:\hostedtoolcache\windows\jfrog\1.46.1\x64\jfrog.exe rt u --url="<--my-url-here-->" --user=*** --password=*** --fail-no-op=true --dry-run=false --insecure-tls=false --threads=3 --retries=3 --symlinks=false

I assume the issue is related as it started appearing this morning, without any changes that we've done on the pipeline.

jfrog / jfrog-azure-devops-extension

Failed while attempting to download JFrog CLI (1.42.3) - no on-premise #235