Coverage not generating

[abhishek72850]

I am trying to generate coverage using github actions, all tests passes but in codecov upload it throws error that no coverage file located.

My github action

name: test

on: [push, pull_request]

env:
  PACKAGE: PyRock

jobs:
  run-tests:
    strategy:
      fail-fast: false
      matrix:
        st-version: [4]
    runs-on: "ubuntu-latest"
    steps:
      - uses: actions/checkout@v3
      - uses: SublimeText/UnitTesting/actions/setup@v1
        with:
          sublime-text-version: ${{ matrix.st-version }}
          package-name: ${{ env.PACKAGE }}
      - uses: SublimeText/UnitTesting/actions/run-tests@v1
        with:
          coverage: true
          # codecov-upload: true
      - uses: codecov/codecov-action@v3
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Codecov Workflow output:

[2023-09-30T21:50:24.341Z] ['info'] => Project root located at: /home/runner/work/pyrock/pyrock
[2023-09-30T21:50:24.342Z] ['info'] ->  Token found by environment variables
[2023-09-30T21:50:24.351Z] ['info'] Searching for coverage files...
[2023-09-30T21:50:24.413Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-09-30T21:50:24.413Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-09-30T21:50:24.414Z] ['error'] There was an error running the uploader: No coverage files located, please try use `-f`, or change the project root with `-R`

Check the attachment for detailed log

logs_34.zip

[randy3k]

Please use the codecov-upload option, the GitHub action won’t work with UnitTesting because action uses newer format of the coverage files.

[abhishek72850]

i used that too, but it still gives the same error

[gerardroche]

I don't think coverage works with ST4 yet due to package dependency issues in Package Control for ST4. Package Control 4 might solve this (re issues like https://github.com/wbond/package_control/issues/1612).

I only run coverage for ST3:

name: Continuous Integration

on: [push, pull_request]

jobs:
  tests:
    strategy:
      fail-fast: false
      matrix:
        st-version: [3, 4]
        os: ["ubuntu-latest", "windows-latest"]
    runs-on: ${{ matrix.os }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup
        uses: SublimeText/UnitTesting/actions/setup@v1
        with:
          sublime-text-version: ${{ matrix.st-version }}
      - name: Tests
        uses: SublimeText/UnitTesting/actions/run-tests@v1
        with:
          coverage: ${{ matrix.os == 'ubuntu-latest' && matrix.st-version == 3 }}
          codecov-upload: ${{ matrix.os == 'ubuntu-latest' && matrix.st-version == 3 }}
      - name: Syntax Tests
        uses: SublimeText/UnitTesting/actions/run-syntax-tests@v1
      - name: Coding guidelines check
        if: ${{ matrix.os == 'ubuntu-latest' && matrix.st-version == 4 }}
        run: |
          pip3 install flake8 flake8-docstrings pydocstyle
          flake8 -v

[abhishek72850]

I can live with that, but the problem is with ST3 its giving different error 😅

Wait for tests output..................................
(subl:3138): dbind-WARNING **: 02:21:50.021: AT-SPI: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
Wait for tests output..................................
Timeout: Could not obtain tests output.
Maybe Sublime Text is not responding or the tests output is being written to the wrong file.
Error: Process completed with exit code 1.`

Test itself doesn't get completed, the only combination works is ST4 with ubuntu (but in ST4 coverage doesn't work)

[abhishek72850]

I can live with that, but the problem is with ST3 its giving different error 😅

Wait for tests output..................................
(subl:3138): dbind-WARNING **: 02:21:50.021: AT-SPI: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
Wait for tests output..................................
Timeout: Could not obtain tests output.
Maybe Sublime Text is not responding or the tests output is being written to the wrong file.
Error: Process completed with exit code 1.`

Test itself doesn't get completed, the only combination works is ST4 with ubuntu (but in ST4 coverage doesn't work)

Ok this^ error gone after i also included sudo apt install at-spi2-core before running test

but now its just getting timeout

Run PACKAGE_FROM_INPUTS=""
  PACKAGE_FROM_INPUTS=""
  PACKAGE="${PACKAGE_FROM_INPUTS:-$PACKAGE}"

  if [ "true" == "true" ]; then
    python3 "$GITHUB_ACTION_PATH/../../scripts/run_tests.py" "$PACKAGE" --coverage
  else
    python3 "$GITHUB_ACTION_PATH/../../scripts/run_tests.py" "$PACKAGE"
  fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    PACKAGE: PyRock
    SUBLIME_TEXT_VERSION: 3
    SUBLIME_TEXT_ARCH: x64
    SUBLIME_TEXT_PACKAGES: /home/runner/.config/sublime-text-3/Packages
    DISPLAY: :1
Wait for tests output..................................
Wait for tests output..................................
Wait for tests output..................................
Timeout: Could not obtain tests output.
Maybe Sublime Text is not responding or the tests output is being written to the wrong file.
Error: Process completed with exit code 1.

[gerardroche]

Your plugin is maybe not compatible with ST3.

[abhishek72850]

Your plugin is maybe not compatible with ST3.

Yeah you right, i didn't notice, ST3 uses python 3.3

[abhishek72850]

Anyway workaround to get codecov on ST4?

[abhishek72850]

There's also one more issue with ST4 and macOS

Installing UnitTesting from https://github.com/SublimeText/UnitTesting@1.5.9 to /Users/runner/Library/Application Support/Sublime Text/Packages/UnitTesting
Installing coverage from https://github.com/codexns/sublime-coverage@1.0.0 to /Users/runner/Library/Application Support/Sublime Text/Packages/coverage
Run bash $GITHUB_ACTION_PATH/../../scripts/install_package_control.sh
creating Package Control.sublime-settings
/Users/runner/work/_actions/SublimeText/UnitTesting/v1/actions/setup/../../scripts/install_package_control.sh: line 67:  1033 Terminated: 15          subl
/Users/runner/work/_actions/SublimeText/UnitTesting/v1/actions/setup/../../scripts/install_package_control.sh: line 67:  1578 Terminated: 15          subl
/Users/runner/work/_actions/SublimeText/UnitTesting/v1/actions/setup/../../scripts/install_package_control.sh: line 67:  1758 Terminated: 15          subl
....................................Timeout: Fail to install Package Control.
Error: Process completed with exit code 1.

This happens during setup only, win and ubuntu are working fine

[deathaxe]

Package Control 3 may fail to run on recent MacOS builds as it requires openssl 1.1, while MacOS moved on to openssl 3.x

CI is therefore broken on MacOS, atm.

[deathaxe]

To get coverage working for ST4 on CI it would be enough to pip install coverage into ST's $data/Libs/python38 path.

That's what I did locally.

[abhishek72850]

To get coverage working for ST4 on CI it would be enough to pip install coverage into ST's $data/Libs/python38 path.

That's what I did locally.

can u tell the steps how to access ST python pip?

[abhishek72850]

Package Control 3 may fail to run on recent MacOS builds as it requires openssl 1.1, while MacOS moved on to openssl 3.x

CI is therefore broken on MacOS, atm.

weird than how does it work in my local, i am on macOS?

[deathaxe]

For details see https://github.com/wbond/package_control/issues/1612, but it seems to be related with tools being installed. Some of them seem to also pull OpenSSL3. Some users even manually downgraded to OpenSSL1.1 on OS level, to keep PC running. I'd rather recommend using PC4.0 beta.

[abhishek72850]

Ok for now i have done following to generate coverage:

1. Installed Sublime text 4143 (because coverage doesn't work in latest 4152)
2. did pip install coverage to get coverage library and copied it to lib/python38
3. Than ran Unittesting with coverage on local
   1. i needed an xml report, so i tweaked the Unittesting plugin to generate it
   2. in Packages/UnitTesting/unittesting/coverage.py:78 added one more line, this will generate the coverage.xml in root of my plugin:
      • cov.xml_report(outfile="{}/coverage.xml".format(package_path), ignore_errors=ignore_errors)
4. Then i updated my test github action to take this coverage.xml and upload it on the codecov

Sample test action:

name: test

on: [push, pull_request]

env:
  PACKAGE: PyRock

jobs:
  run-tests:
    strategy:
      fail-fast: false
      matrix:
        st-version: [4]
        os: ["ubuntu-latest", "windows-latest"]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v3
      - uses: codecov/codecov-action@v3
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
        with:
          files: ./coverage.xml
      - uses: SublimeText/UnitTesting/actions/setup@v1
        with:
          sublime-text-version: ${{ matrix.st-version }}
          package-name: ${{ env.PACKAGE }}
      - uses: SublimeText/UnitTesting/actions/run-tests@v1
        with:
          coverage: false
          codecov-upload: false

[abhishek72850]

Now i am facing some weird issue, tests are failing on github action, but working fine locally 😞

^^^^ False issue, ignore

[deathaxe]

I don't think codecov/codecov-action installs required codecov library to ST's Libs path.

[abhishek72850]

I don't think codecov/codecov-action installs required codecov library to ST's Libs path.

No i mean on local, i copied the coverage and tweaked the unittesting plugin to generate the xml file, and then uploaded that xml using codecov/codecov-action, this way now i am able to Run test and coverage on ST4 (v4143) with PC3

This is my final test workflow:

name: test

# on: [push, pull_request]
on: [push]

env:
  PACKAGE: PyRock

jobs:
  run-tests:
    strategy:
      fail-fast: false
      matrix:
        st-version: [4]
        os: ["ubuntu-latest", "windows-latest"]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v3
      - uses: SublimeText/UnitTesting/actions/setup@v1
        with:
          sublime-text-version: ${{ matrix.st-version }}
          package-name: ${{ env.PACKAGE }}
      - uses: SublimeText/UnitTesting/actions/run-tests@v1
      - 
      - name: Upload code coverage
        if: github.ref == 'refs/heads/master' 
        uses: codecov/codecov-action@v3
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
        with:
          files: ./coverage.xml

[deathaxe]

Unsure I can follow, but your Github Action installs ST4 and UnitTesting package on the runner and runs it. Hence codecov library also needs to be installed to ST's Libs directory on the runner, for code coverage to work in CI environment.

[abhishek72850]

Unsure I can follow, but your Github Action installs ST4 and UnitTesting package on the runner and runs it. Hence codecov library also needs to be installed to ST's Libs directory on the runner, for code coverage to work in CI environment.

I am actually not generating any coverage in github action, i did a hack, where i generate the coverage.xml on local, then also upload it in my repo.

now this github action then runs the test separately, and then uploads that already generated coverage file from the repo to codecov.

[abhishek72850]

I think if we update the Unittesting github action to install ST 4143 then that coverage will start working, and i don't have to do all these spearate coverage generation

[abhishek72850]

btw can we pls add support for generating xml report also? i can raise the small pr 😅 if u allow?

[deathaxe]

It's not only Package Control 4 causing issues with coverage in ST4.

With PC4 installed, latest coverage libraries are installed from pypi. Currently v7.4.4.

Those use sqlite3 databases to store recordings, while former coverage 4.5 uses custom text files.

UnitTesting however directly tries to manipulate those files, which fails on ST4 due to format changes.

This is fixed by #233.

Note: Coverage fails to run on ST4's python 3.8 plugin_host on MacOS (see #234) and is therefore skipped until a solution can be found.