search

jfrog / frogbot

🐸 Scans your Git repository with JFrog Xray for security vulnerabilities. 🤖
https://docs.jfrog-applications.jfrog.io/
Apache License 2.0
289 stars 61 forks source link

scan-multiple-repositories always clones source repo on Azure DevOps #629

Open anael-l opened 5 months ago

anael-l commented 5 months ago

Describe the bug

The scan-multiple-repositories command always clones the repo containing the frogbot-config.yml file instead of the repos defined in it My pipeline + frogbot config are in a repo called CockpIT_frogbot My frogbot config points to two repos CockpIT-front and CockpIT-back As shown in the log, the config is read, but for both scans, the repo CockpIT_frogbot is cloned instead

Current behavior

##[section]Starting: Download and Run Frogbot
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.178.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
[command]/usr/bin/bash --noprofile --norc /opt/agt/_work/_temp/e71a6319-464e-4155-973a-d17de31b9031.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2395  100  2395    0     0  16292      0 --:--:-- --:--:-- --:--:-- 16292
Downloading the latest version of Frogbot...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 27.9M  100 27.9M    0     0  76.0M      0 --:--:-- --:--:-- --:--:-- 76.0M
Frogbot downloaded successfully!
17:08:38 [Info] Frogbot version: 2.19.8
17:08:38 [Debug] Reading config from file system. Looking for .frogbot/frogbot-config.yml
17:08:38 [Debug] frogbot-config.yml found in /opt/agt/_work/1/s/.frogbot/frogbot-config.yml
17:08:38 [Debug] The content of frogbot-config.yml that will be used is:
- params:
    git:
      repoName: CockpIT-front
      branches:
        - master

- params:
    git:
      repoName: CockpIT-back
      branches:
        - master
17:08:38 [Debug] Locking config file to run config AddOrEdit command.
17:08:38 [Debug] Sending HTTP HEAD request to: 'https://github.com/jfrog/frogbot'
17:08:38 [Debug] Creating lock in: /tmp/jfrog.cli.temp.-1706371718-4135036626/locks/config
17:08:38 [Warn] couldn't extract payload from Access Token.
The provided access token is not a valid JWT, probably a reference token.
Some package managers only support basic authentication which requires also a username.
If you plan to work with one of those package managers, please provide a username.
17:08:38 [Debug] Releasing lock: /tmp/jfrog.cli.temp.-1706371718-4135036626/locks/config/jfrog-cli.conf.lck.398.1706371718273425349
17:08:38 [Debug] Config AddOrEdit command completed successfully. config file is released.
17:08:38 [Debug] Usage Report: Sending info...
17:08:38 [Info] Running Frogbot "scan-multiple-repositories" command
17:08:38 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:08:38 [Debug] Sending HTTP POST request to: https://usage-ecosystem.jfrog.io/api/usage/report
17:08:38 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/artifactory/api/system/version
17:08:38 [Debug] Artifactory response: 200 
17:08:38 [Debug] JFrog Artifactory version is: 7.55.10
17:08:38 [Debug] Sending HTTP POST request to: https://<artifactory-server-url>/artifactory/api/system/usage
17:08:38 [Debug] JFrog Xray version is: 3.71.6
17:09:08 [Debug] Can't check access to 'https://github.com/jfrog/frogbot', error while sending request:
Head "https://github.com/jfrog/frogbot": dial tcp 140.82.121.4:443: i/o timeout
17:09:08 [Debug] Setting timeout for go-git to 120 seconds ...
17:09:08 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1706371748-2178551145
17:09:08 [Debug] Running git clone https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot (master branch)...
17:09:08 [Debug] Project cloned from https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot to /tmp/jfrog.cli.temp.-1706371748-2178551145
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/entitlements/feature/contextual_analysis
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/HEAD' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/config' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/index' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/info' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.idx' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/heads' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/heads/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes/origin' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/remotes/origin/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/refs/tags' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-2178551145/.git/shallow' is excluded
17:09:08 [Info] Couldn't determine a package manager or build tool used by this project. Skipping the SCA scan...
17:09:08 [Info] Xray scan completed
17:09:08 [Info] Didn't find vulnerable dependencies with existing fix versions for CockpIT-front
17:09:08 [Debug] Setting timeout for go-git to 120 seconds ...
17:09:08 [Debug] Created temp working directory: /tmp/jfrog.cli.temp.-1706371748-1524852656
17:09:08 [Debug] Running git clone https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot (master branch)...
17:09:08 [Debug] Project cloned from https://<azure-devops-server-url>/Global/CockpIT/_git/CockpIT_frogbot to /tmp/jfrog.cli.temp.-1706371748-1524852656
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/system/version
17:09:08 [Debug] Sending HTTP GET request to: https://<artifactory-server-url>/xray/api/v1/entitlements/feature/contextual_analysis
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/HEAD' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/config' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/index' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/info' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.idx' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/objects/pack/pack-c71ee7558fb4c6f988daa2bd1351a05dff7c589f.pack' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/heads' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/heads/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes/origin' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/remotes/origin/master' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/refs/tags' is excluded
17:09:08 [Debug] The path '/tmp/jfrog.cli.temp.-1706371748-1524852656/.git/shallow' is excluded
17:09:08 [Info] Couldn't determine a package manager or build tool used by this project. Skipping the SCA scan...
17:09:08 [Info] Xray scan completed
17:09:08 [Info] Didn't find vulnerable dependencies with existing fix versions for CockpIT-back
17:09:08 [Info] Frogbot "scan-multiple-repositories" command finished successfully
##[section]Finishing: Download and Run Frogbot

Reproduction steps

No response

Expected behavior

Each repo declared in the frogbot-config.yml file should be cloned and scan instead of the repo containing the pipeline and config.

JFrog Frogbot version

2.19.8

Package manager info

maven/npm but not relevant

Git provider

Azure DevOps

JFrog Frogbot configuration yaml file

frogbot.yml (pipeline)

pr: none
trigger: none

pool: Linux-Build

variables:
  JF_GIT_PROJECT: $(System.TeamProject)
  JF_GIT_REPO: $(Build.Repository.Name)
  JF_GIT_API_ENDPOINT: $(System.CollectionUri)
  JF_GIT_BASE_BRANCH: $(Build.SourceBranchName)
  JF_GIT_OWNER: $(System.TeamProject)
  JF_GIT_PROVIDER: 'azureRepos'

jobs:
  - job:
    displayName: "Frogbot Scan Repository and Fix"
    steps:
       - task: CmdLine@2
         displayName: 'Download and Run Frogbot'
         env:
            JF_URL: $(JF_URL)
            JF_ACCESS_TOKEN: $(JF_ACCESS_TOKEN)
            JF_GIT_TOKEN: $(System.AccessToken)
            JF_RELEASES_REPO: "frogbot-generic-external"
            JFROG_CLI_LOG_LEVEL: "DEBUG"
         inputs:
            script: |
               getFrogbotScriptPath=$(if [ -z "$JF_RELEASES_REPO" ]; then echo "https://releases.jfrog.io"; else echo "${JF_URL}/artifactory/${JF_RELEASES_REPO}"; fi)
               curl -fLg "$getFrogbotScriptPath/artifactory/frogbot/v2/[RELEASE]/getFrogbot.sh" --header "X-JFrog-Art-Api: $JF_ACCESS_TOKEN" | sh
               ./frogbot scan-multiple-repositories

frogbot-config.yml

- params:
    git:
      repoName: CockpIT-front
      branches:
        - master

- params:
    git:
      repoName: CockpIT-back
      branches:
        - master

Operating system type and version

RHEL 8

JFrog Xray version

3.71.6

eranturgeman commented 5 months ago

Hello @anael-l, thank you for using Frogbot! In your frogbot-config.yml, you've configured 'params' for both CockpIT-front and CockpIT-back. Before delving into the issue, please try using only one set of 'params' and specify the required working directories for scanning under params/projects/workingDirs. I want to eliminate the possibility that you initiated two separate scans. Please refer to our documentation and the frogbot-config.yml schema to configure it correctly.

anael-l commented 5 months ago

Hello @eranturgeman, My goal IS to launch two scans of two different git repositories. I've followed this doc: https://docs.jfrog-applications.jfrog.io/jfrog-applications/frogbot/setup-frogbot/frogbot-configuration#can-one-frogbot-config.yml-file-be-used-for-multiple-git-repositories To setup one central frogbot config, to scans multiple other repositories that are in the same organization. Isn't what the scan-multiple-repositories command is for ?

eranturgeman commented 5 months ago

@anael-l You are correct this is what it suppose to do. Thank you for the reporting the issue. Our team will look into it and we will keep you updated here