BerkeleyLearnVerify / Scenic

A compiler and scenario generator for the Scenic scenario description language.
https://scenic-lang.org/
Other
258 stars 93 forks source link

feat: adding code cov for PyTest including statement & branch coverage #217

Closed abanuelo closed 4 months ago

abanuelo commented 4 months ago

Adding pytest-cov, a plugin on top of pytest for code coverage along with coverage tool in pyproject.toml:

Eric-Vin commented 4 months ago

Looks great to me! Just have two questions:

abanuelo commented 4 months ago
  1. Eventually, yes! examples/ was originally excluded from the CI and this PR does not change that. I think we can consider re-adding it to both CI and coverage when they stabilize.
  2. No. Everything within the src/scenic/simulators/* has been excluded from coverage.
Eric-Vin commented 4 months ago

Whoops, sorry! I meant to say Scenic model files, not the interfaces. Those are .scenic files though so I'm not sure if we could even use the same infrastructure to measure coverage for those.

abanuelo commented 4 months ago

Correct, I don't think we could use the same code coverage frame work since technically it looks for code coverage around .py files. I can look into potentially extending the existing coverage library to find line/branch coverage for .scenic files but that would require some work.

The question here is, how beneficial is line coverage for overall scenic simulation files? My first impression is we just care about expected inputs and outputs of those simulations to be correct and less about line coverage.

dfremont commented 4 months ago

The Scenic world models that Eric is talking about are general libraries used across many scenarios, so we should test their functionality and line coverage would be useful. But let's leave that for later -- it would require writing a plugin for coverage.py (the underlying library being used here) to give it support for Scenic files.

codecov[bot] commented 4 months ago

Welcome to Codecov :tada:

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered :open_umbrella: