Generate coverage results and upload to codecov

[tobiasdiez]

Use coverage.py to generate coverage results for the sage source code based on the test execution. In contrast to sage's naïve coverage tool, this not only looks for the existence of doctests for a method but actually measures which lines are actually executed by the tests.

These coverage results are then uploaded to codecov, where one can inspect the results. See https://codecov.io/gh/sagemath/sagetrac-mirror/tree/ec045eb0168b937e35375e803979975df4aa6564/src/sage for the current coverage. Codecov also displays the coverage of changed files, so that one can easily check if changes are indeed covered by tests. An example would be https://codecov.io/gh/sagemath/sagetrac-mirror/commit/ec045eb0168b937e35375e803979975df4aa6564 but there are of course no changes in python files in this ticket, so it shows nothing interesting. For a more interesting example from a different project see eg https://app.codecov.io/gh/JabRef/jabref/commit/2494509f2d75426bd7369b358880e7ffed2a47b4. The idea would be to add the link to such a report as a new badge in the trac ticket (after this ticket here is merged).

Run: https://github.com/sagemath/sagetrac-mirror/actions/workflows/build.yml?query=branch%3Apublic/build/codecov

CC: @mkoeppe

Component: build

Author: Tobias Diez

Branch: ec045eb

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33355

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3c315dd to 7ff3baa

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

7ff3baa

Try with editable install

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -1 +1,3 @@
+Run: https://github.com/sagemath/sagetrac-mirror/actions/workflows/build.yml?query=branch%3Apublic/build/codecov

+This currently fails because the sage environment is not correctly activated before running `sage-runtests` (coverage needs a python script as argument, so one cannot use `sage -t`). What's the best way to accomplish this?

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4c7c162

Try sourcing sage-env

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 7ff3baa to 4c7c162

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6a93454

No source cmd?!

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4c7c162 to 6a93454

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

97970d1

Try sage-env-config

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6a93454 to 97970d1

[mkoeppe]

comment:6

Use sage -sh -c "...." to run a shell command .... in the Sage environment.

[mkoeppe]

comment:7

Or use sage -python -m coverage ...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

af9b442

Try with sage -python

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 97970d1 to af9b442

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from af9b442 to 1c53f6f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1c53f6f

Ignore coverage warnings

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3bf2fdc

Install git

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1c53f6f to 3bf2fdc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3bf2fdc to e4bca51

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e4bca51

Try without sudo

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e4bca51 to 5523e44

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5523e44

Have to manually combine coverage results

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8b25060

Force 2 threads

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5523e44 to 8b25060

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

cc8e027

Fix indent

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 8b25060 to cc8e027

[tobiasdiez]

comment:15

Replying to @mkoeppe:

Or use sage -python -m coverage ...

Thanks that worked well! (Why is this command actually necessary? I thought its the job of sage-conf to handle all environment variables)

I just noticed that elementary things like git and sudo are missing in the standard docker image. Would it make sense to add these tools?

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

ec045eb

Try to find coverage file

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cc8e027 to ec045eb

[mkoeppe]

comment:17

Replying to @tobiasdiez:

Replying to @mkoeppe:

Or use sage -python -m coverage ...

Thanks that worked well! (Why is this command actually necessary? I thought its the job of sage-conf to handle all environment variables)

This is a goal that has not been achieved yet. What sage-env does is still needed for some things in the Sage library. See for example #31296.

[mkoeppe]

comment:18

Replying to @tobiasdiez:

Replying to @mkoeppe:

Or use sage -python -m coverage ...

Thanks that worked well! (Why is this command actually necessary? I thought its the job of sage-conf to handle all environment variables)

I just noticed that elementary things like git and sudo are missing in the standard docker image. Would it make sense to add these tools?

I added git to -recommended, see #33296, #33277, but we're not building the image yet. We could replace -standard but -recommended. Probably best after #33062.

You don't need sudo, you are running as root.

[tobiasdiez]

comment:19

Replying to @mkoeppe:

You don't need sudo, you are running as root.

It's more for convenience and coherence - normally github actions don't run as sudo.

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -1,3 +1,6 @@
+Use coverage.py to generate coverage results for the sage source code based on the test execution. In contrast to sage's naïve coverage tool, this not only looks for the existence of doctests for a method but actually measures which lines are actually executed by the tests.
+
+These coverage results are then uploaded to codecov, where one can inspect the results. See https://codecov.io/gh/sagemath/sagetrac-mirror/tree/ec045eb0168b937e35375e803979975df4aa6564/src/sage for the current coverage. Codecov also displays the coverage of changed files, so that one can easily check if changes are indeed covered by tests. An example would be https://codecov.io/gh/sagemath/sagetrac-mirror/commit/ec045eb0168b937e35375e803979975df4aa6564 but there are of course no changes in python files in this ticket, so it shows nothing interesting. For a more interesting example from a different project see eg https://app.codecov.io/gh/JabRef/jabref/commit/2494509f2d75426bd7369b358880e7ffed2a47b4. The idea would be to add the link to such a report as a new badge in the trac ticket (after this ticket here is merged). 
+
 Run: https://github.com/sagemath/sagetrac-mirror/actions/workflows/build.yml?query=branch%3Apublic/build/codecov

-This currently fails because the sage environment is not correctly activated before running `sage-runtests` (coverage needs a python script as argument, so one cannot use `sage -t`). What's the best way to accomplish this?

[mkoeppe]

comment:21

I am getting "There was a problem getting the source code from your provider. Unable to show line by line coverage. "

https://app.codecov.io/gh/sagemath/sagetrac-mirror/blob/ec045eb0168b937e35375e803979975df4aa6564/src/sage/categories/chain_complexes.py

[tobiasdiez]

comment:22

The link is working for me. Maybe a temporary issues on their side?

[mkoeppe]

comment:23

Working now for me too

[mkoeppe]

comment:24

Looking great, thanks for this work!

[mkoeppe]

Reviewer: Matthias Koeppe

[tobiasdiez]

comment:25

Thanks for the review!

[vbraun]

Changed branch from public/build/codecov to ec045eb

[fchapoton]

comment:27

currently not working in the "Build and Test" badge, shouting :

Run ./venv/bin/python3 -m coverage combine src/.coverage/
  ./venv/bin/python3 -m coverage combine src/.coverage/
  ./venv/bin/python3 -m coverage xml
  find . -name *coverage*
  shell: sh -e {0}
Couldn't combine from non-existent path 'src/.coverage/'
Error: Process completed with exit code 1.

[fchapoton]

Changed commit from ec045eb to none

[tobiasdiez]

comment:28

Which ticket shows this behavior? It is still working correctly in the latest beta: https://github.com/sagemath/sagetrac-mirror/runs/5710760370?check_suite_focus=true

[fchapoton]

comment:29

see #33629 for example

[tobiasdiez]

comment:30

Okay, was actually an issue with pyright. Fixed in #33631.

[mkoeppe]

comment:31

Can we get the badge now?

[fchapoton]

comment:32

the badge "Build & Test" is still broken by problems about coverage

for example, see

https://github.com/sagemath/sagetrac-mirror/runs/6117339523?check_suite_focus=true

shouting

Run ./venv/bin/python3 -m coverage combine src/.coverage/
6
/__w/sagetrac-mirror/sagetrac-mirror/venv/bin/python3: No module named coverage
7
Error: Process completed with exit code 1.

[mkoeppe]

comment:33

I've opened #33747 for this