fortify / fcli

fcli is a command-line utility for interacting with various Fortify products
https://fortify.github.io/fcli/
Other
27 stars 16 forks source link

Error uploading file at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:79) #507

Closed quinnturner closed 1 month ago

quinnturner commented 5 months ago

I am trying to scan a C# project using Fortify from GitHub Actions.

name: Fortify on Demand SAST Scan

on:
  workflow_dispatch:
    inputs:
      branch:
        description: "Branch to scan"
        required: true
        default: "main"
  schedule:
    - cron: "30 1 * * 1"
  push:
    paths:
      - .github/workflows/fortify.yml
jobs:
  FoD-SAST-Scan:
    # Use the appropriate runner for building your source code.
    # Use Windows runner for projects that use msbuild. Additional changes to RUN commands will be required.
    runs-on: windows-latest

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.branch || 'main' }}

      - name: Setup MSBuild
        uses: microsoft/setup-msbuild@v1
        with:
          vs-version: 16

      - name: Run FoD SAST Scan
        uses: fortify/github-action@v1
        with:
          sast-scan: true
        env:
          FOD_URL: "https://emea.fortify.com"
          FOD_TENANT: ${{ secrets.FOD_TENANT }}
          FOD_CLIENT_ID: ${{ secrets.FOD_API_KEY }}
          FOD_CLIENT_SECRET: ${{ secrets.FOD_SECRET_KEY }}
          FOD_RELEASE: ${{ secrets.FOD_RELEASE_ID }}

Every time I run this, I get the following logs:

Time Elapsed 00:00:43.16
Packaging project...
Run fortify/github-action/internal/run@v1.1.0
  with:
    cmd: "${FCLI_CMD}" fod sast-scan start --rel "${FOD_RELEASE}" -f package.zip --store fod_scan ${EXTRA_FOD_SAST_SCAN_OPTS}
  env:
    FCLI_INSTALL_DIR: D:\a\_temp\fortify\fcli\2.1.0
    FCLI_BIN_DIR: D:\a\_temp\fortify\fcli\2.1.0\bin
    FCLI_CMD: D:\a\_temp\fortify\fcli\2.1.0\bin\fcli.exe
    _FOD_LOGIN_OPTS: ""
    SC_CLIENT_INSTALL_DIR: D:\a\_temp\fortify\sc-client\23.1.0
    SC_CLIENT_BIN_DIR: D:\a\_temp\fortify\sc-client\23.1.0\bin
    SC_CLIENT_CMD: D:\a\_temp\fortify\sc-client\23.1.0\bin\scancentral.bat
    FOD_URL: [https://emea.fortify.com](https://mcas-proxyweb.mcas.ms/certificate-checker?login=false&originalUrl=https%3A%2F%2Femea.fortify.com.mcas.ms%3FMcasTsid%3D11760&McasCSRF=8121137312ae6f1791bdfd39c226bae5244a1d91ae012444c5905a90b9da3ee7)
    FOD_TENANT: ***
    FOD_CLIENT_ID: ***
    FOD_CLIENT_SECRET: ***
    FOD_RELEASE: ***
D:\a\_temp\fortify\fcli\2.1.0\bin\fcli.exe fod sast-scan start --rel *** -f package.zip --store fod_scan

Upload package.zip: 0 of 20524567 bytes complete

Upload package.zip: 1048576 of 20524567 bytes complete

Upload package.zip: 2097152 of 20524567 bytes complete

Upload package.zip: 3145728 of 20524567 bytes complete

Upload package.zip: 4194304 of 20524567 bytes complete

Upload package.zip: 5242880 of 20524567 bytes complete

Upload package.zip: 6291456 of 20524567 bytes complete

Upload package.zip: 7340032 of 20524567 bytes complete

Upload package.zip: 8388608 of 20524567 bytes complete

Upload package.zip: 9437184 of 20524567 bytes complete

Upload package.zip: 10485760 of 20524567 bytes complete

Upload package.zip: 11534336 of 20524567 bytes complete

Upload package.zip: 12582912 of 20524567 bytes complete

Upload package.zip: 13631488 of 20524567 bytes complete

Upload package.zip: 14680064 of 20524567 bytes complete

Upload package.zip: 15728640 of 20524567 bytes complete

Upload package.zip: 16777216 of 20524567 bytes complete

Upload package.zip: 17825792 of 20524567 bytes complete

Upload package.zip: 18874368 of 20524567 bytes complete
java.lang.RuntimeException: Error uploading file
    at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:79)
    at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScan(FoDScanSastHelper.java:83)
    at com.fortify.cli.fod._common.scan.helper.sast.FoDScanSastHelper.startScanWithDefaults(FoDScanSastHelper.java:55)
    at com.fortify.cli.fod.sast_scan.cli.cmd.FoDSastScanStartCommand.startScan(FoDSastScanStartCommand.java:71)
    at com.fortify.cli.fod._common.scan.cli.cmd.AbstractFoDScanStartCommand.getJsonNode(AbstractFoDScanStartCommand.java:36)
    at com.fortify.cli.fod._common.output.cli.AbstractFoDJsonNodeOutputCommand.getJsonNode(AbstractFoDJsonNodeOutputCommand.java:23)
    at com.fortify.cli.common.output.cli.cmd.AbstractOutputCommand.run(AbstractOutputCommand.java:33)
    at picocli.CommandLine.executeUserObject(CommandLine.java:2103)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2538)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2530)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2492)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2350)
    at picocli.CommandLine$RunLast.execute(CommandLine.java:2494)
    at picocli.CommandLine.execute(CommandLine.java:2247)
    at com.fortify.cli.app.runner.DefaultFortifyCLIRunner.run(DefaultFortifyCLIRunner.java:49)
    at com.fortify.cli.app.FortifyCLI.execute(FortifyCLI.java:38)
    at com.fortify.cli.app.FortifyCLI.main(FortifyCLI.java:32)
Caused by: com.fortify.cli.common.rest.unirest.UnexpectedHttpResponseException: 
Request: POST [https://api.emea.fortify.com/api/v3/releases/***/static-scans/start-scan-with-defaults?isRemediationScan=false&scanTool=fcli&scanToolVersion=2.1.0&scanMethodType=Other&fragNo=-1&offset=19922944:](https://mcas-proxyweb.mcas.ms/certificate-checker?login=false&originalUrl=https%3A%2F%2Fapi.emea.fortify.com.mcas.ms%2Fapi%2Fv3%2Freleases%2F***%2Fstatic-scans%2Fstart-scan-with-defaults%3FisRemediationScan%3Dfalse%26scanTool%3Dfcli%26scanToolVersion%3D2.1.0%26scanMethodType%3DOther%26fragNo%3D-1%26offset%3D19922944%3A%26McasTsid%3D11760&McasCSRF=8121137312ae6f1791bdfd39c226bae5244a1d91ae012444c5905a90b9da3ee7) 
Response: 500 Internal Server Error
Response Body:
{"errors":[{"errorCode":1001,"mes***":"Unexpected error processing request"}]}
    at com.fortify.cli.common.rest.unirest.config.UnirestUnexpectedHttpResponseConfigurer$UnexpectedHttpResponseInterceptor.onResponse(UnirestUnexpectedHttpResponseConfigurer.java:36)
    at kong.unirest.CompoundInterceptor.lambda$onResponse$1(CompoundInterceptor.java:48)
    at java.base@17.0.9/java.util.ArrayList.forEach(ArrayList.java:1511)
    at kong.unirest.CompoundInterceptor.onResponse(CompoundInterceptor.java:48)
    at kong.unirest.apache.ApacheClient.request(ApacheClient.java:134)
    at kong.unirest.Client.request(Client.java:57)
    at kong.unirest.BaseRequest.request(BaseRequest.java:365)
    at kong.unirest.BaseRequest.asString(BaseRequest.java:218)
    at com.fortify.cli.fod._common.rest.helper.FoDFileTransferHelper.uploadChunked(FoDFileTransferHelper.java:72)
    ... 16 more
Error: Failed to run command:
"D:\a\_temp\fortify\fcli\2.1.0\bin\fcli.exe" fod sast-scan start --rel "***" -f package.zip --store fod_scan 
Error: Error: The process 'D:\a\_temp\fortify\fcli\2.1.0\bin\fcli.exe' failed with exit code 1

The failure always occurs at the same amount of data uploaded.

Hoping for some advice!

MikeTheSnowman commented 5 months ago

Hey @quinnturner. Until the smart people chime in, are you able to see if there are any errors/failures in the Application Scans page of the application release that you tried to upload to? If you see something like that, click on the ... button on the far right of the failed scan to see if there's any log or manifest that you can bring back over here for us to take a look at.

image
rsenden commented 5 months ago

Hi @quinnturner, as you can see in the output, the GitHub Action uses fcli to upload the package to FoD. To get a better idea about whether this issue may be on the fcli-side or the FoD side, can you try submitting the scan request (with the same package) using FoDUploader?

You can either archive and download the package.zip file to try manually, or you can use the fortify/github-action/setup action to install FoDUploader and run it from your pipeline. If you could share the package with us, that would be even better to allow us to try for ourselves.

MikeTheSnowman commented 5 months ago

Hey @quinnturner . Have you had any luck with making progress on your issue?

quinnturner commented 5 months ago

Hi all, I will be tackling this tomorrow. Will report back when I have more info!

quinnturner commented 3 months ago

I just wanted to let you know I got this working using the uploader. I think we were not using scan-central to build the application, or there was some issue with variable substitution.

I will close this ticket.

name: Fortify on Demand SAST Scan

on:
  workflow_dispatch:
    inputs:
      branch:
        description: "Branch to scan"
        required: true
        default: "main"
  schedule:
    #        ┌───────────── minute (0 - 59)
    #        │  ┌───────────── hour (0 - 23)
    #        │  │ ┌───────────── day of the month (1 - 31)
    #        │  │ │ ┌───────────── month (1 - 12 or JAN-DEC)
    #        │  │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
    #        │  │ │ │ │
    #        │  │ │ │ │
    #        │  │ │ │ │
    #        *  * * * *
    - cron: "30 1 * * 1"
  push:
    paths:
      - .github/workflows/fortify.yml
jobs:
  FoD-SAST-Scan:
    # Use the appropriate runner for building your source code.
    # Use Windows runner for projects that use msbuild. Additional changes to RUN commands will be required.
    runs-on: windows-latest

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.branch || 'main' }}

      - name: Setup MSBuild
        uses: microsoft/setup-msbuild@v1
        with:
          vs-version: 16

      # Java is required to run ScanCentral Client and may be required for your build.
      # The Java version to use depends on the Java version required to run your build (if any),
      # and the Java version supported by the ScanCentral Client version that you are running.
      - name: Setup Java
        uses: actions/setup-java@v1
        with:
          java-version: 1.8

      - name: Setup Fortify tools
        uses: fortify/github-action/setup@v1
        with:
          tool-definitions: https://github.com/fortify/tool-definitions/releases/download/v1/tool-definitions.yaml.zip
          export-path: true
          fcli: latest
          sc-client: 23.1.0
          fod-uploader: latest
          vuln-exporter: v2
          bugtracker-utility: skip
          debricked-cli: skip

      - name: Run scancentral
        run: |
          scancentral package --build-tool msbuild --build-file project.sln --output package.zip

      - name: Perform SAST Scan
        run: FoDUpload -z package.zip -ep 2 -aurl "https://api.emea.fortify.com" -purl "https://emea.fortify.com" -tc "${{ secrets.FOD_TENANT }}" -ac "${{ secrets.FOD_API_KEY }}" "${{ secrets.FOD_SECRET_KEY }}" -rid "${{ secrets.FOD_RELEASE_ID }}" -n "Triggered by GitHub Actions (${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})"
rsenden commented 3 months ago

Hi @quinnturner, good to see that you've found a work-around that works for you. However, we'd like to understand why the upload with fcli was failing before. Ultimately the goal is to integrate all functionality provided by other Fortify command-line utilities (like FoDUploader) into fcli, so obviously if fcli fails to upload some payloads, we'd like to understand why, and fix it.

Given your sample workflow above, can you try replacing the FoDUpload call with the corresponding fcli calls, i.e., something like the following (please double-check for typos), and let us know the results?

fcli fod session login --url https://emea.fortify.com -t "${{ secrets.FOD_TENANT }}" --client-id "${{ secrets.FOD_API_KEY }}" --client-secret "${{ secrets.FOD_SECRET_KEY }}"
fcli fod sast-scan start -f package.zip --rel "${{ secrets.FOD_RELEASE_ID }}" --notes "Triggered by GitHub Actions (${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})"
fcli fod session logout

Given that your earlier output showed the fcli upload failing halfway, I very much doubt that this is related to incorrect variable substitution. Our GitHub Action should have taken care of the scancentral invocation to package the source code, although by default it uses auto-detection (so it wouldn't have the explicit --build-tool and --build-file options), but I don't see how that could cause the upload to fail halfway.

So, either this is an fcli-specific issue or an issue with the FoD upload endpoint that fcli is using (which may be different than the endpoint used by FoDUploader), or maybe it was just some temporary FoD-side issue.

rsenden commented 1 month ago

Closing as we haven't been able to reproduce this issue, and user started using FoDUploader instead.