Something in the latest versions of foundry is causing my github pipeline to fail (it was working before)

[JasoonS]

Rerunning old tests also breaks on tests that used to pass, so potentially something new in the latest foundry is causing an issue.

It would be super nice to be able to specify a fixed version of foundry rather than nightly to prevent these kinds of issues in the future.

Here is my pipeline:

name: Run Contract FORGE Tests

on:
  pull_request:
    paths:
      - "contracts/**/*.sol"
      - "contracts/test/**"

env:
  FOUNDRY_PROFILE: ci # This foundry profile is set to test the contracts slighly more vigorously.

defaults:
  run:
    working-directory: ./contracts

jobs:
  run-forge-contract-tests:
    name: Run contract FORGE tests
    runs-on: ubuntu-latest

    steps:
      # Checks out a copy of your repository on the ubuntu-latest machine
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Cache contract node_modules
        id: cache_contracts_node_modules
        uses: actions/cache@v2
        with:
          path: ~/work/monorepo/monorepo/contracts/node_modules
          key: ${{ runner.os }}-modules-${{ hashFiles('~/work/monorepo/monorepo/contracts/yarn.lock') }}

      - name: install node packages
        if: steps.cache_contracts_node_modules.outputs.cache-hit != 'true'
        id: install_node_packages_contracts
        run: |
          yarn

      - name: Install Foundry
        uses: onbjerg/foundry-toolchain@v1
        with:
          version: nightly # Note be careful, this might break at some point! We should fix a version rather - wasn't clear what the version was though from docs at time

      - name: Cache the foundry/forge cache
        id: cache_foundry_compile
        uses: actions/cache@v2
        with:
          path: |
            ~/work/monorepo/monorepo/contracts/cache
            ~/work/monorepo/monorepo/contracts/out
          key: ${{ runner.os }}-foundry-cache-v0.8.15-${{ hashFiles('~/work/monorepo/monorepo/contracts/contracts/**') }}
          restore-keys: |
            ${{ runner.os }}-foundry-cache-v0.8.15

      - name: Install forge dependencies
        id: run_install_dependencies
        run: |
          forge install

      - name: Pre-build contracts for following steps
        id: build_contracts
        run: |
          forge build #--extra-output abi --extra-output userdoc --extra-output devdoc --extra-output evm.methodIdentifiers

      - name: Run forge tests
        id: run_forge_tests
        run: |
          forge snapshot --check # -vvvv means that all stack and setup traces are displayed

      ## No benefit to running slither in the pipeline currently... 😢
      # - name: Run Slither
      #   id: slither
      #   uses: crytic/slither-action@v0.1.0
      #   continue-on-error: true
      #   with:
      #     ignore-compile: true
      #     node-version: 16
      #     sarif: results.sarif

      # # TODO:  re-add this once github has enabled this feature for our account.
      # - name: Upload SARIF file
      #   id: upload-sarif-file
      #   uses: github/codeql-action/upload-sarif@v2
      #   with:
      #     sarif_file: ${{ steps.slither.outputs.sarif }}

I'm unable to replicate this failure locally, and multiple people from my team also don't have the issue locally, so it seems like it is something about how the pipeline takes the output from build.

[onbjerg]

What's your version locally? When was this pipeline run the last time?

It is not possible to pin to something that is not nightly yet (and we only keep the last 3 nightlies), as we are still not in a stable state. We're moving fast and things will break, so we currently require people to update. We're moving towards a stable release, which you can follow here: https://github.com/foundry-rs/foundry/milestone/1

If your Forge version does not say 0.2.0, then you are out of date: run foundryup again. If running foundryup does not get you to 0.2.0, then your foundryup is out of date and you need to reinstall that as well.

Going to close this issue since there is no way for us to support pinning to a non-nightly version until we cut a stable release :slightly_frowning_face: If you still have issues, reach out on our support Telegram or report your bugs to the main Foundry repo

[JasoonS]

Thanks locally I have the latest version, I have run foundryup.

The issue isn't that I can't pin a stable version - that was just a side comment.

The issue is that forge build doesn't work in the pipeline (with same version of forge locally and on remote) but it works locally.

Additionally, going back and re-running a pipeline that passed 1 week ago now fails.

Maybe this is a problem with upstream forge?

[onbjerg]

I would urge you to open an issue in the Foundry repo with some reproduction steps so we can resolve the issue :smile:

[JasoonS]

Sure, will give that a try when I get a chance!

[JasoonS]

I worked out what was breaking the pipeline. SMT stuff was the culprit. Somethiing related to that must have changed in forge.

Removing the following from my foundry.toml fixed it:

## NOTE: the below configuration isn't woring properly yet. Needs some work.
#        https://docs.soliditylang.org/en/latest/smtchecker.html
[ci.model_checker]
contracts = { 'contracts/market/template/MarketTieredLeverage.sol' = [
  'MarketTieredLeverage',
] }
engine = 'chc' # or 'all'/'bmc'
timeout = 10000
targets = [
  'assert',
  'divByZero',
  'underflow',
  'overflow',
  'divByZero',
  'constantCondition',
  'popEmptyArray',
  'outOfBounds',
] # not included: 'balance'