How to use automation tools like Nox or Tox with this action

[edgarrmondragon]

I'm following the docs to set up CodSpeed for a public Python project.

The project uses Nox to manage environments, so I'd want it to also use Nox to run benchmarks. The project already uses pytest-benchmark for this purpose, but I seem unable to use this action with that approach.

Here's what the step looks like:

    - name: Run Nox
      uses: CodSpeedHQ/action@v1
      if: always() && (matrix.session == 'benches')
      with:
        token: ${{ secrets.CODSPEED_TOKEN }}
        run: nox -v --python=${{ matrix.python-version }}

Nox creates a Python virtual environment under .nox/benches but this actions seems to expect dependencies to be installed in the runner's own Python venv because I see the following:

Warning: pytest-codspeed is not installed in your environment. Installing it...

Locally, running with Nox works:

$ GITHUB_ACTIONS=1 nox -rs benches
nox > Running session benches
nox > Re-using existing virtual environment at .nox/benches.
nox > poetry build --format=wheel --no-ansi
nox > pip uninstall --yes file:///Users/imauser/Code/meltano/sdk/dist/singer_sdk-0.33.0b1-py3-none-any.whl
nox > python -m pip install --constraint=.nox/benches/tmp/requirements.txt 'singer-sdk[s3] @ file:///Users/imauser/Code/meltano/sdk/dist/singer_sdk-0.33.0b1-py3-none-any.whl'
nox > python -m pip install --constraint=.nox/benches/tmp/requirements.txt 'coverage[toml]' duckdb duckdb-engine pytest pytest-benchmark pytest-codspeed pytest-durations pytest-snapshot pyarrow requests-mock time-machine black cookiecutter PyYAML darglint flake8 flake8-annotations flake8-docstrings mypy
nox > pytest --codspeed
================================================================ test session starts ================================================================
platform darwin -- Python 3.10.13, pytest-7.4.2, pluggy-1.2.0 -- /Users/imauser/Code/meltano/sdk/.nox/benches/bin/python
codspeed: 2.2.0 (callgraph: not supported)
NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.
CodSpeed had to disable the following plugins: pytest-benchmark
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
sqlalchemy: 2.0.21
rootdir: /Users/imauser/Code/meltano/sdk
configfile: pyproject.toml
testpaths: tests
plugins: snapshot-0.9.0, time-machine-2.10.0, codspeed-2.2.0, durations-1.2.0, requests-mock-1.11.0, benchmark-4.0.0
collected 393 items / 392 deselected / 1 selected                                                                                                   

tests/core/test_mapper.py::test_bench_simple_map_transforms PASSED

I'm also unsure how I would use Poetry directly if I gave up on using Nox: poetry install doesn't seem to cut it because of the same issue of having a managed virtual environment, so maybe poetry export > requirements.txt followed by pip install -r requirements.txt?

Thanks in advance!

[art049]

Hello, I never ran it with nox before but I managed to make it work :)

Here is the noxfile I used

import nox

@nox.session
def codspeed(session):
    session.install('pytest')
    session.install('pytest-codspeed')
    session.run('pytest', '--codspeed')

Along with this workflow:

name: benchmarks

on:
  workflow_dispatch:
  push:
    branches:
      - "main"
  pull_request:

jobs:
  benchmarks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v3
        with:
          python-version: "3.12.0"

      - name: Install Nox
        run: pip install nox

      - name: Install dependencies
        run: nox --sessions codspeed --install-only

      - name: Run the action
        uses: CodSpeedHQ/action@v1
        with:
          run: nox --sessions codspeed --reuse-existing-virtualenvs --no-install
          token: ${{ secrets.CODSPEED_TOKEN }}

Splitting the install and the run part by reusing the venv is optional, but it will significantly reduce the time to run everything.

Nox creates a Python virtual environment under .nox/benches but this actions seems to expect dependencies to be installed in the runner's own Python venv because I see the following:

Yes, this is, unfortunately a legacy operation we need to remove in the next major release. #83

[art049]

I added those details in https://docs.codspeed.io/benchmarks/python#usage-with-nox Don't hesitate to reopen if you think something is missing!

[edgarrmondragon]

@art049 I tried that and I see a message like this:

Prepare environment
  Warning: pytest-codspeed is not installed in your environment. Installing it...

Which may be the reason why CodSpeed is reporting a performance degradation when I tried it.

• Workflow: https://github.com/meltano/sdk/actions/runs/6871300337/job/18687879776?pr=2049
• CodSpeed report: https://github.com/meltano/sdk/pull/2049#issuecomment-1811617019