search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.49k stars 1.49k forks source link

Ruby scanning job hangs forever and doesn't complete on Ubuntu-latest #12349

Open jedrekdomanski opened 1 year ago

jedrekdomanski commented 1 year ago

Hello,

We have set up a CodeQL code scannig job in our Ruby project and it takes over 6 hours to run and never completes. I have tried using both the default queries as well as security-extended and security-and-quality but they hang forever and never complete. We run two jobs (for Ruby and Javascript) using a language matrix. This is our codeql-analysis.yml file. Currently the timeout-minutes is set to 25 but it is only so to limit the run time and cut the cost of the job because we pay for it but it never completes. It was set to 6 hours but it didn't complete either.

name: "CodeQL"

on:
  push:
    branches: [ "master" ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ "master" ]

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    timeout-minutes: 25
    permissions:
      actions: read
      contents: read
      security-events: write

    strategy:
      fail-fast: false
      matrix:
        language: [ 'ruby', 'javascript' ]
        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality

    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
    # If this step fails, then you should remove it and run the build manually (see below)
    - name: Autobuild
      uses: github/codeql-action/autobuild@v2

    # ℹ️ Command-line programs to run using the OS shell.
    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

    #   If the Autobuild fails above, remove it and uncomment the following three lines.
    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

    # - run: |
    #   echo "Run, Build Application using script"
    #   ./location_of_script_within_repo/buildscript.sh

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}"

Here is some logs, as you can see it just seats there and does not progress at all.

Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/MissingRegExpAnchor.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-020/OverlyLargeRange.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-022/PathInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/CommandInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/KernelOpen.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/NonConstantKernelOpen.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-078/UnsafeShellCommandConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/ReflectedXSS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/StoredXSS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-079/UnsafeHtmlConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-089/SqlInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-094/CodeInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-094/UnsafeCodeConstruction.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/BadTagFilter.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/IncompleteMultiCharacterSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-116/IncompleteSanitization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-117/LogInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/PolynomialReDoS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/ReDoS.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-1333/RegExpInjection.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-134/TaintedFormatString.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-209/StackTraceExposure.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-295/RequestWithoutValidation.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-312/CleartextLogging.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-312/CleartextStorage.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-327/BrokenCryptoAlgorithm.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-502/UnsafeDeserialization.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-506/HardcodedDataInterpretedAsCode.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-598/SensitiveGetQuery.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-601/UrlRedirect.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-611/Xxe.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-798/HardcodedCredentials.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-829/InsecureDownload.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-912/HttpToFileAccess.ql.
Starting evaluation of codeql/ruby-queries/queries/security/cwe-918/ServerSideRequestForgery.ql.
Starting evaluation of codeql/ruby-queries/queries/summary/LinesOfCode.ql.
[3/46 eval 2.4s] Evaluation done; writing results to codeql/ruby-queries/queries/summary/LinesOfCode.bqrs.
Starting evaluation of codeql/ruby-queries/queries/summary/LinesOfUserCode.ql.
Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.ql.
[4/46 eval 8ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/LinesOfUserCode.bqrs.
Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.ql.
[5/46 eval 3ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.bqrs.
[6/46 eval 5ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.bqrs.
[7/46 eval 1m16s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.bqrs.
[8/46 eval 1m30s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.bqrs.
[9/46 eval 1m33s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.bqrs.
[10/46 eval 1m36s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.bq
aibaars commented 1 year ago

@jedrekdomanski Thanks for reporting. It looks like your repository somehow runs into a performance issue with one of the queries. Did things work for you in the past or did you just setup CodeQL analysis for your repository.

If things used to work, could you try running a previous version of CodeQL as a workaround. This can be done by setting the tools: property of the github/codeql-action/init to the download URL of codeql-bundle-linux64.tar.gz of an earlier release https://github.com/github/codeql-action/releases .

Could you try :

   - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}" 
      env:
        CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

This should limit runs of codeql database analyze to about 10 minutes for a query. This way the log should inform us which query it was working on when the timeout was reached. Could you also re-run the workflow with the debug logging: https://github.blog/changelog/2022-05-24-github-actions-re-run-jobs-with-debug-logging/ ? That should collect more detailed logs and also save intermediate artifacts such as the CodeQL database.

If this is an open source repository, could you share the URL and any debug artifact so we can investigate.

If this is a closed source repository, please contact GitHub support to continue this conversation via an internal support ticket.

jedrekdomanski commented 1 year ago

Thank you for your quick reply. It was never successful before, we've only just started running the scannig jobs in the project. I'll try your suggestions. In the meantime, here's the full log of the job which I ran in debug mode. https://pipelines.actions.githubusercontent.com/serviceHosts/0887281a-bad0-42c6-a967-808339f1591c/_apis/pipelines/1/runs/4450/signedlogcontent/3?urlExpires=2023-03-01T12%3A57%3A59.3172412Z&urlSigningMethod=HMACV1&urlSignature=hq6k5n8uXCj3hMj8Dt0vXVrmPx%2FC3BJf%2FE3R0AU0bqw%3D

aibaars commented 1 year ago

Thanks for the quick reply, unfortunately the URL you posted had expired before I could download it.

jedrekdomanski commented 1 year ago

I've attached the logs of a failed job below. output.txt

jedrekdomanski commented 1 year ago

Here is another output file of a job that's just failed. output.txt

aibaars commented 1 year ago

Looking at the output of the "resolve files" command, it seems like your repository is quite large. Most likely CodeQL is running low on memory due to the size of the repository which causes it to slow down. You could try running the analysis on a larger runner or a self-hosted one: Using larger runners.
See also: Recommended hardware resources for running CodeQL

Another thing to try is to reduce the number of scanned files. The spec folder is probably test cases, and most likely do not need to be scanned for security vulnerabilities. You could also exclude the db migrations. See also: Specifying directories to scan

If reducing the files and increasing the RAM does not work then it would be helpful to do the following:

jedrekdomanski commented 1 year ago

@aibaars I've tried reducing the number of scanned files but this doesn't work. The documentation says to add this:

paths:
  - src
paths-ignore:
  - src/node_modules
  - '**/*.test.js'

And so says the example config file here https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#example-configuration-files

name: "My CodeQL config"

disable-default-queries: true

queries:
  - name: Use an in-repository QL pack (run queries in the my-queries directory)
    uses: ./my-queries
   ...
paths:
  - src 
paths-ignore: 
  - src/node_modules
  - '**/*.test.js'

So I added this to the root namespace in my config file:

name: "CodeQL"

on:
  push:
    branches: [ "master" ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ "master" ]

jobs:
  ...

paths:
  - app
  - lib
paths-ignore: 
  - spec
  - db

but I get an error

The workflow is not valid. .github/workflows/codeql-analysis.yml (Line: 74, Col: 1): Unexpected value 'paths' .github/workflows/codeql-analysis.yml (Line: 77, Col: 1): Unexpected value 'paths-ignore'
aibaars commented 1 year ago

I think you need to put the configuration in a separate file and refer to it using the config-file: property.

- name: Perform CodeQL Analysis
   uses: github/codeql-action/analyze@v2
   with:
     category: "/language:${{matrix.language}}" 
     config-file: ./.github/codeql/codeql-config.yml
   env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

See Working with custom configuration files

jedrekdomanski commented 1 year ago 
Unexpected input(s) 'config-file', valid inputs are ['check_name', 'output', 'upload', 'cleanup-level', 'ram', 'add-snippets', 'skip-queries', 'threads', 'checkout_path', 'ref', 'sha', 'category', 'upload-database', 'wait-for-processing', 'token', 'matrix', 'expect-error']

So I am surprised... that I do what the docs say and it doesn't work. But clearly it is a performance problem so the question is how do I reduce the number of scanned files? :)

aibaars commented 1 year ago

I'm sorry, the yaml snippet I included is wrong. The config-file property should be set on the github/codeql-action/init step.

- uses: github/codeql-action/init@v2
  with:
    config-file: ./.github/codeql/codeql-config.yml
jedrekdomanski commented 1 year ago

Is this a correct configuration?

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml

config file

# ./.github/codeql/codeql-config.yml

paths:
  - app
  - lib
paths-ignore:
  - spec
  - db
jedrekdomanski commented 1 year ago

Anyway, it doesn't seem to improve the runtime at all and env property

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

doesn't look like it has any effect. Also, I tried setting the previous version of CodeQl as you suggested in your first reply using tools property on the github/codeql-action/init step pointing to the URL of the previous version but it doesn't work.

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      tools: /github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
      with:
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml
The workflow is not valid. .github/workflows/codeql-analysis.yml (Line: 45, Col: 7): Unexpected value 'tools'
aibaars commented 1 year ago 
env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "analyze": ["--timeout", "600"] } }'

doesn't look like it has any effect.

My bad, I thought the codeql-action ran the codeql database analyze command but it looks like it is running the lower level database run-queries command instead: https://github.com/github/codeql-action/blob/a589d4087ea22a0a48fc153d1b461886e262e0f2/src/codeql.ts#L820

Could you try:

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "run-queries": ["--timeout", "600"] } }'
aibaars commented 1 year ago

Also, I tried setting the previous version of CodeQl as you suggested in your first reply using tools property on the github/codeql-action/init step pointing to the URL of the previous version but it doesn't work.

The tools property should be under the with: section (and include https://).

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        tools: https:/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml
jedrekdomanski commented 1 year ago

Unfortunately, it doesn't work

Did not find CodeQL tools version 0.0.0-20221024 in the toolcache.
  Downloading CodeQL tools from https:/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz. This may take a while.
  getaddrinfo EAI_AGAIN github
  Waiting [15](https://github.com/acima-credit/lease_management_system/actions/runs/4313596212/jobs/7525587011#step:3:16) seconds before trying again
  getaddrinfo EAI_AGAIN github
  Waiting 13 seconds before trying again
  Error: Error: getaddrinfo EAI_AGAIN github
  Error: Unable to download and extract CodeQL CLI
  Error: Unable to download and extract CodeQL CLI
      at setupCodeQL (/home/runner/work/_actions/github/codeql-action/v2/lib/codeql.js:134:15)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async initCodeQL (/home/runner/work/_actions/github/codeql-action/v2/lib/init.js:46:76)
      at async run (/home/runner/work/_actions/github/codeql-action/v2/lib/init-action.js:126:34)
      at async runWrapper (/home/runner/work/_actions/github/codeql-action/v2/lib/init-action.js:[20](https://github.com/acima-credit/lease_management_system/actions/runs/4313596212/jobs/7525587011#step:3:21)9:9)

My config looks like this:

- name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        tools: https:/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml
aibaars commented 1 year ago

Is this a correct configuration?

# Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml

config file

# ./.github/codeql/codeql-config.yml

paths:
  - app
  - lib
paths-ignore:
  - spec
  - db

I think that configuration looks good. Could you also set the following globally (near the top of the workflow):

env:
  RUST_LOG: info

That should print a line for each file that is scanned.

aibaars commented 1 year ago

My config looks like this:

- name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        tools: https:/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz
        languages: ${{ matrix.language }}
        config-file: ./.github/codeql/codeql-config.yml

Sorry, it should include the hostname too of course: https://github.com/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz .

jedrekdomanski commented 1 year ago

Thank you. Here's what I see now:

  1. I was able to run the previous version of CodeQL
  2. We were able to successfully limit the directories to scan only to app and lib ✅ . I can see what files are loaded it in the logs: 
    [2023-03-02 11:52:43] [build-stdout] [2023-03-02 11:52:43] [build-stdout]  INFO extracting: /home/runner/work/lease_management_system/lease_management_system/lib/...

    and

    [2023-03-02 11:52:43] [build-stdout] [2023-03-02 11:52:43] [build-stdout]  INFO extracting: /home/runner/work/lease_management_system/lease_management_system/app/..
  3. The job failed with timeout. Here's the logs 
    [6/36 eval 4ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.bqrs.
[7/36 eval 40.7s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.bqrs.
[8/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[9/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-918/ServerSideRequestForgery.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[10/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-829/InsecureDownload.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[11/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[12/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-611/Xxe.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[13/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-601/UrlRedirect.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[14/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-598/SensitiveGetQuery.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[15/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[16/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-327/BrokenCryptoAlgorithm.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[17/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-312/CleartextStorage.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[18/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-312/CleartextLogging.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[19/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-134/TaintedFormatString.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[20/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-1333/RegExpInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[21/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-1333/ReDoS.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[22/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-1333/PolynomialReDoS.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[23/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-116/IncompleteSanitization.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[24/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-116/IncompleteMultiCharacterSanitization.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[25/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-502/UnsafeDeserialization.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[26/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-116/BadTagFilter.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[28/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-089/SqlInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[27/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-094/CodeInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[29/36 timeout 16m19s] codeql/ruby-queries/queries/security/cwe-079/StoredXSS.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[30/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-079/ReflectedXSS.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[31/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-078/NonConstantKernelOpen.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[32/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-078/KernelOpen.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[33/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-022/PathInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[34/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-020/OverlyLargeRange.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[35/36 timeout 16m20s] codeql/ruby-queries/queries/security/cwe-078/CommandInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
[36/36 timeout 16m26s] codeql/ruby-queries/queries/security/cwe-020/IncompleteHostnameRegExp.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
Shutting down query evaluator.
29 of 36 queries timed out.
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33
  at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v2/lib/toolrunner-error-catcher.js:82:19)
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
  at async Object.databaseRunQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/codeql.js:495:13)
  at async runQueryGroup (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:238:9)
  at async runQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:178:43)
  at async run (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:169:24)
  at async runWrapper (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:238:9)
Error: Error running analysis for ruby: Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33

    Full logs. logs.txt

aibaars commented 1 year ago

Ok, so even when only scanning app and lib the analysis still fails? Have you tried with a runner with more RAM?

There are a couple more things to try to make CodeQL run with a single thread which may require less RAM. Add the following to the top of the workflow:

env:
  CODEQL_THREADS: 1

Add --tuple-counting, --evaluator-log=evaluator.log to collect statistics. 

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "run-queries": ["--timeout", "600", "--tuple-counting", "--evaluator-log=${{ runner.temp }}/evaluator.log"] } }'

and upload ${{ runner.temp }}/evaluator.log using the actions/upload-artifact action.

If you (re)run the workflow in debug mode it also uploads a debug artifact. This can be used for diagnosing problems. Note that it contain a copy of the scanned source code, so do not attach it to this public issue . You can attach parts of it of course, just be careful not to leak information you like to keep private.

jedrekdomanski commented 1 year ago

Yes, despite limiting the directories to scan to app and lib it still fails.

I don't know how to use a runner with more RAM. I didn't find any documentation on how to do that. We don't have our own runners. Is it possible to increase RAM?

I added the code you suggested but it still failed with the same error (timeout after ~ 16 minutes).

Error log:

[32/36 timeout 16m18s] codeql/ruby-queries/queries/security/cwe-078/CommandInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
  [33/36 timeout 16m18s] codeql/ruby-queries/queries/security/cwe-022/PathInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
  [34/36 timeout 16m17s] codeql/ruby-queries/queries/security/cwe-1333/RegExpInjection.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
  [35/36 timeout 16m18s] codeql/ruby-queries/queries/security/cwe-020/OverlyLargeRange.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
  [36/36 timeout 16m22s] codeql/ruby-queries/queries/security/cwe-020/IncompleteHostnameRegExp.ql Timeout (10m0s) in ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff.
  Shutting down query evaluator.
  29 of 36 queries timed out.
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33
  Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33
      at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v2/lib/toolrunner-error-catcher.js:82:19)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Object.databaseRunQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/codeql.js:495:13)
      at async runQueryGroup (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:238:9)
      at async runQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:178:43)
      at async run (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:169:24)
      at async runWrapper (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:238:9)
  Error: Error running analysis for ruby: Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33
  ##[debug]Sending status report: {"workflow_run_id":4314125041,"workflow_name":"CodeQL","job_name":"analyze","analysis_key":".github/workflows/codeql-analysis.yml:analyze","commit_oid":"2871b69de9b[3456](https://github.com/acima-credit/lease_management_system/actions/runs/4314125041/jobs/7527222951#step:5:3459)de826d80ef7c359d5c4483f1d","ref":"refs/pull/6387/merge","action_name":"finish","action_ref":"v2","action_oid":"unknown","started_at":"2023-03-02T13:11:33.454Z","action_started_at":"2023-03-02T13:12:11.896Z","status":"failure","testing_environment":"","runner_os":"Linux","action_version":"2.2.5","cause":"Error running analysis for ruby: Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33","exception":"CodeQLAnalysisError: Error running analysis for ruby: Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20221024/x64/codeql/codeql' failed with exit code 33\n    at runQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:215:19)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async run (/home/runner/wor...
  ##[debug]Node Action run completed with exit code 1
  ##[debug]CODEQL_ACTION_VERSION='2.2.5'
  ##[debug]CODEQL_ACTION_FEATURE_SARIF_COMBINE='true'
  ##[debug]CODEQL_ACTION_FEATURE_WILL_UPLOAD='true'
  ##[debug]CODEQL_ACTION_FEATURE_MULTI_LANGUAGE='false'
  ##[debug]CODEQL_ACTION_FEATURE_SANDWICH='false'
  ##[debug]Finishing: Perform CodeQL Analysis
jedrekdomanski commented 1 year ago

I added the step upload artifact but it doesn't work.

- name: Step 3 - Use the Upload Artifact GitHub Action
      uses: actions/upload-artifact
      with:
        name: my-artifacts
        path: ${{ runner.temp }}/evaluator.log
the `uses' attribute must be a path, a Docker image, or owner/repo@ref

I didn't know how to do that so I found this documentation https://github.com/actions/upload-artifact#upload-an-individual-file but it doesn't work

aibaars commented 1 year ago

I think you got the indentation wrong:

- name: Step 3 - Use the Upload Artifact GitHub Action
  uses: actions/upload-artifact
  with:
     name: my-artifacts
     path: ${{ runner.temp }}/evaluator.log
aibaars commented 1 year ago

Or perhaps the problem is that you forgot the @version tag: actions/upload-artifact@v3

aibaars commented 1 year ago

I don't know how to use a runner with more RAM. I didn't find any documentation on how to do that. We don't have our own runners. Is it possible to increase RAM?

See Using larger runners for information.

You can also try on a local machine (Linux, Windows, or OSX) :

jedrekdomanski commented 1 year ago

I added @v3 and it ran but the step did not run image

I ran codeql locally but it failed:

./codeql database create -lruby -s ../acima/lease_management_system /tmp/database-folder
Initializing database at /tmp/database-folder.
Finalizing database at /tmp/database-folder.
Successfully created database at /tmp/database-folder.

./codeql database analyze --tuple-counting --evaluator-log=/tmp/evaluator.log /tmp/database-folder
Missing required options [--format=<format>, --output=<output>]
Try codeql database analyze --help for usage help.

./codeql database analyze --tuple-counting --evaluator-log=/tmp/evaluator.log /tmp/database-folder --format=csv
Missing required option '--output=<output>'
Try codeql database analyze --help for usage help.

./codeql database analyze --tuple-counting --evaluator-log=/tmp/evaluator.log /tmp/database-folder --format=csv --output=/tmp/runner.log
Running queries.
A fatal error occurred: Query pack codeql/ruby-queries cannot be found. Check the spelling of the pack.
aibaars commented 1 year ago

Try adding --download codeql/ruby-queries or run codeql pack download codeql/ruby-queries first.

jedrekdomanski commented 1 year ago

It still fails

./codeql pack download codeql/ruby-queries
Package specifications to check for download: codeql/ruby-queries
Package install location: /Users/jedrek/.codeql/packages
Installed fresh codeql/ruby-queries@0.5.3

 ./codeql database analyze --tuple-counting --evaluator-log=/tmp/evaluator.log /tmp/database-folder --format=csv --output=/tmp/runner.log
Running queries.
A fatal error occurred: Failed to create JSON log output
(eventual cause: FileAlreadyExistsException "/tmp")
aibaars commented 1 year ago

That's indeed strange, I get the same error. Try with --evaluator-log=/tmp/log/evaluator.log . Apologies for the inconvenience. I have no idea why writing directly in /tmp does not work.

jedrekdomanski commented 1 year ago

It's taking over an hour to run and it's stuck and not progressing at all. image

I am running MacBook M1 Ventura 32GB of RAM

aibaars commented 1 year ago

Thanks 32GB should be enough normally. Could you cancel the command and attach a zip file with the evaluator log? You could still attempt different values of the --ram and --threads flag. For example --threads 1 and --ram 28g .

Could you tell me how many lines of ruby code your repository has? A 2 million line code base should take about 20 to 30 minutes. Sometimes the analysis behaves badly on unusual code patterns. Hopefully the evaluation log informs us of what's going on.

jedrekdomanski commented 1 year ago

Counting only files in /app and /lib

find app/ -name '*.rb' | xargs wc -l | grep total
118988 total

find lib/ -name '*.rb' | xargs wc -l | grep total
   16886 total

that's 135874 in total. So since 2 milion line code base should take up to 30 minutes why does 135874 lines of code not complete at all? Attached is the output file but it doesn't really tell much. evaluator.log.zip

aibaars commented 1 year ago

Thanks, we'll have a look at the log.

aibaars commented 1 year ago

FYI you can get a more readable log by running codeql generate log-summary evaluator.log --format=text , at the end there are some tables with statistics:

...
Most expensive predicates for unfinished query PathInjection.ql:
        time         | evals |   max @ iter | predicate
        -------------|-------|--------------|----------
        (incomplete) |   144 |   38s @ 138  | DataFlowImpl#084fa68b::MkStage#Stage3#::Stage#Stage4Param#::fwdFlowThrough#11#fffffffffff@f9e54ymi
               6m27s |    55 | 28.2s @ 12   | ApiGraphs#3116a2f3::API::Impl::trackUseNode#2#fff@f8873629
        (incomplete) |   144 |  7.4s @ 130  | DataFlowImpl#084fa68b::MkStage#Stage3#::Stage#Stage4Param#::fwdFlow#8#ffffffff@f9e54xmi
        (incomplete) |   144 |  5.8s @ 129  | DataFlowImpl#084fa68b::MkStage#Stage3#::Stage#Stage4Param#::fwdFlow0#8#ffffffff@f9e54wmi
        (incomplete) |   145 |  4.8s @ 134  | DataFlowImpl#084fa68b::MkStage#Stage3#::Stage#Stage4Param#::fwdFlowRetFromArg#9#fffffffff#reorder_2_3_4_8_0_1_5_6_7@f9e542wi
               37.9s |   330 |    3s @ 40   | DataFlowDispatch#36b84300::trackInstance#3#ffff#reorder_3_0_1_2@039b1mw4
        (incomplete) |   145 |  1.3s @ 131  | DataFlowImpl#084fa68b::MkStage#Stage3#::Stage#Stage4Param#::fwdFlowStore#9#fffffffff@f9e540wi
...
aibaars commented 1 year ago

I added @v3 and it ran but the step did not run image

By default steps do not run if a previous step has failed. However, you can change this by adding if: always() : https://docs.github.com/en/actions/learn-github-actions/expressions#always

- name: Step 3 - Use the Upload Artifact GitHub Action
  uses: actions/upload-artifact@v3
  if: always()
  with:
     name: my-artifacts
     path: ${{ runner.temp }}/evaluator.log
aibaars commented 1 year ago

If you have time, could you add if: always() to the actions/upload-artifact step in the workflow as above and add a 2 hour (7200 seconds) timeout:

env:
     CODEQL_ACTION_EXTRA_OPTIONS: '{ "database": { "run-queries": ["--timeout", "7200", "--tuple-counting", "--evaluator-log=${{ runner.temp }}/evaluator.log", "--threads", "1"] } }'

After these changes, trigger the workflow and cancel the run manually after a few minutes, to test that the log is indeed getting uploaded. If that works, then re-run the job in debug mode. It should stop after roughly 2 hours and upload a log file.

jedrekdomanski commented 1 year ago

It failed with Unknown option: '--thread'

 Unknown option: '--thread'
  Try codeql database run-queries --help for usage help.
  Error: The process '/opt/hostedtoolcache/CodeQL/2.12.2-20230207/x64/codeql/codeql' failed with exit code 2
  Error: The process '/opt/hostedtoolcache/CodeQL/2.12.2-20230207/x64/codeql/codeql' failed with exit code 2
      at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v2/lib/toolrunner-error-catcher.js:82:19)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async Object.databaseRunQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/codeql.js:495:13)
      at async runQueryGroup (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:238:9)
      at async runQueries (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:144:17)
      at async run (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:169:24)
      at async runWrapper (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:238:9)
  Error: Error running analysis for ruby: Error: The process '/opt/hostedtoolcache/CodeQL/2.12.2-20230207/x64/codeql/codeql' failed with exit code 2

I see the command that was used to run codeql was:

 /opt/hostedtoolcache/CodeQL/2.12.2-20230207/x64/codeql/codeql database run-queries --ram=5919 --threads=2 /home/runner/work/_temp/codeql_databases/ruby --min-disk-free=1024 -v --timeout 7200 --tuple-counting --evaluator-log=/home/runner/work/_temp/evaluator.log --thread 1 --expect-discarded-cache

and one of the options used to run it was --ram=5919 --threads=2. So honestly I am not sure why you want us to limit the resources even further by setting --threads=1 instead of 2.

aibaars commented 1 year ago

It failed with Unknown option: '--thread'

That should indeed have been --threads.

aibaars commented 1 year ago

and one of the options used to run it was --ram=5919 --threads=2. So honestly I am not sure why you want us to limit the resources even further by setting --threads=1 instead of 2.

Memory is shared by the different threads, so if memory is getting low the 2 threads may be competing for resources and make things even worse. On the other hand 2 is already quite low, so it's probably fine.

jedrekdomanski commented 1 year ago

We progressed a bit further and reached as far as 29/39 query but it failed after 2 hours.

Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.ql.
  [4/39 eval 15ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/LinesOfUserCode.bqrs.
  Starting evaluation of codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.ql.
  [5/39 eval 3ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfFilesExtractedWithErrors.bqrs.
  [6/39 eval 4ms] Evaluation done; writing results to codeql/ruby-queries/queries/summary/NumberOfSuccessfullyExtractedFiles.bqrs.
  [7/39 eval 8.5s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-300/InsecureDependencyResolution.bqrs.
  [8/39 eval 36s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-020/IncompleteUrlSubstringSanitization.bqrs.
  [9/39 eval 41s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-352/CSRFProtectionDisabled.bqrs.
  [10/39 eval 42s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-732/WeakCookieConfiguration.bqrs.
  [11/39 eval 40m27s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-327/BrokenCryptoAlgorithm.bqrs.
  [12/39 eval 40m28s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-611/Xxe.bqrs.
  [13/39 eval 40m34s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-078/KernelOpen.bqrs.
  [14/39 eval 40m34s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-078/NonConstantKernelOpen.bqrs.
  [15/39 eval 40m35s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-134/TaintedFormatString.bqrs.
  [16/39 eval 40m33s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-598/SensitiveGetQuery.bqrs.
  [17/39 eval 40m35s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-078/UnsafeShellCommandConstruction.bqrs.
  [18/39 eval 40m35s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-079/UnsafeHtmlConstruction.bqrs.
  [19/39 eval 40m37s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-079/ReflectedXSS.bqrs.
  [20/39 eval 40m35s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-312/CleartextLogging.bqrs.
  [21/39 eval 40m37s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-601/UrlRedirect.bqrs.
  [22/39 eval 40m40s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-089/SqlInjection.bqrs.
  [23/39 eval 40m40s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-209/StackTraceExposure.bqrs.
  [24/39 eval 40m40s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-1333/RegExpInjection.bqrs.
  [25/39 eval 40m45s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-020/IncompleteHostnameRegExp.bqrs.
  [26/39 eval 40m38s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-502/UnsafeDeserialization.bqrs.
  [27/39 eval 40m42s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-020/OverlyLargeRange.bqrs.
  [28/39 eval 40m42s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-078/CommandInjection.bqrs.
  [29/39 eval 40m40s] Evaluation done; writing results to codeql/ruby-queries/queries/security/cwe-312/CleartextStorage.bqrs.

When I reduce the number of files to scan only to lib folder it runs within 2 minutes ✅ But that is only 16,000 lines of code. But it's weird that it worked before on entire repository before. Around October 27th it was the last time it worked for us. I even tried going back in history of our repo and running the CodeQL job on the code from before October 15th and it didn't work either and we didn't add much code since then, I mean we did but not millions lines of code. We are talking about 160,000 lines of code only in app and lib folders in total and because you said 2 millions lines of code runs around 20-30 minutes it is very weird for me that 160,000 lines of code cannot run within 2 hours. It looks like something's changed under the hood. Can you check how many resources (RAM, CPU, threads, ect.) were allocated to the job around October? Right now we have --ram=5919 --threads=2 so if the resources of the runner have not changed it would mean that CodeQL has a performance issue.

aibaars commented 1 year ago

Thanks for the update. Could you attach the evaluator log of the 2 hour run?

We progressed a bit further and reached as far as 29/39 query but it failed after 2 hours.

That's nearly done. Could you retry with a timeout of 5 hours? It would be really great to get an evaluation log of a completed run and it looks like that may be possible.

Around October 27th it was the last time it worked for us.

That is really good to know. The RAM, CPU allocations should be the same, the spec of the Actions VM hasn't changed. Let's try with the CodeQL version of September: https://github.com/github/codeql-action/releases/tag/codeql-bundle-20220923 by setting tools: https://github.com/github/codeql-action/releases/download/codeql-bundle-20220923/codeql-bundle-linux64.tar.gz . If that works we can repeat with some of the other versions on https://github.com/github/codeql-action/releases until we find which version introduced the performance regression on your code.

We do performance tests on over 2000 repositories for each release, but perhaps your code base has some code patterns that confuse the analyzer for some reason. Do you know if any of the files in the app directory contain "weird" code? For example, I've seen CodeQL perform badly once on a file that contained an entire phone book in a very large array literal.

jedrekdomanski commented 1 year ago

Do you know if any of the files in the app directory contain "weird" code? For example, I've seen CodeQL perform badly once on a file that contained an entire phone book in a very large array literal.

No, we don't have such code in our repo :) I'll try to run it for 5 hours again and send you the logs.

jedrekdomanski commented 1 year ago

The 2h run log was too large (57M) so you'll have to unzip the 3 files and combine them into one. Sorry. xac.zip xab.zip xaa.zip

jedrekdomanski commented 1 year ago

@aibaars Do you have any update on this?

aibaars commented 1 year ago

A colleague of mine just informed me

Yeah I looked at that yesterday, but it just very hard to deal with without a reproduction example. It will take some time. I was thinking we could try putting a lot of different Rails apps in the same database until we have a fake Rails app with a similar number of endpoint

The logs suggest that there are quite a lot of end-points in your application. Does that sound right? As you can imagine, it is quite hard to create an example database to reproduce the same problems as you are experiencing on the real one.

jedrekdomanski commented 1 year ago

We have 54 API endpoints in our app. How much is "a lot" for you and how much can CodeQL handle? WIll increasing the runner solve our problem?

aibaars commented 1 year ago

We have 54 API endpoints in our app. How much is "a lot" for you and how much can CodeQL handle? WIll increasing the runner solve our problem?

That is not really a lot. I would consider hundreds or thousands of end-points "a lot".

A large runner should work better, and may be able to complete the analysis. However, I suspect there is something in your code base that somehow "confuses" CodeQL, so I don't expect great performance even with a large runner. Still worth a try though.

Have you had a chance to try with an old version of CodeQL, for example the September version?

tools: https://github.com/github/codeql-action/releases/download/codeql-bundle-20220923/codeql-bundle-linux64.tar.gz

jedrekdomanski commented 1 year ago

I've just tried it and it completed within less than 3 minutes on our latest code base on both app and lib folder :) I suspect something was changed along the way in CodeQL or new ruby queries were added. Reading the changelog it doesn't tell me much so could you please investigate the changes and see what might be causing the timeout? Here's the changelog I saw https://github.com/github/codeql/blob/codeql-cli/v2.12.3/ruby/ql/src/CHANGELOG.md

We are at version 0.4.0 now.

aibaars commented 1 year ago

Three minutes, that is more like it! Could you try to find the release in which the performance regression was introduced?

These are the releases since September:

The easiest is probably to try them one by one in order until the first one that is slow. You can also try a "matrix" job to try them all at the same time: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

jedrekdomanski commented 1 year ago

This is the last version that works (October 10th) https://github.com/github/codeql-action/releases/download/codeql-bundle-20221010/codeql-bundle-linux64.tar.gz The latter version (October 24th) https://github.com/github/codeql-action/releases/download/codeql-bundle-20221024/codeql-bundle-linux64.tar.gz fails

aibaars commented 1 year ago

@jedrekdomanski Sorry that this is taking so long.

Unfortunately, we have not been able to reproduce the issue you are experiencing. The relevant change in the October 24th version is likely the improvements to the call graph (matching method calls with method definitions). However, the call graph computation itself is not slow in the log file, so possibly there is a problem that was unreachable before, but became reachable due to the changes in the call graph.

The most effective way to continue the investigation would be to have a copy of the CodeQL database. Would it be possible for you to share that with GitHub engineers? Note, that a CodeQL database contains a copy of the analyzed source code, so: