BerkeleyLearnVerify / Scenic

A compiler and scenario generator for the Scenic scenario description language.
https://scenic-lang.org/
Other
256 stars 91 forks source link

Fix cross-entropy tests and leakage issue #259

Closed dfremont closed 2 months ago

dfremont commented 2 months ago

Description

The tests for the interface with VerifAI's cross-entropy sampler keep failing occasionally since I chose the parameters poorly. This PR improves the tests in various ways and should ensure the false positive rate is extremely low. In the process I noticed an issue whereby the distribution of one sampling run would leak into the next (even after calling resetExternalSampler), so I've fixed that and added an assertion catching it.

Issue Link

n/a

Checklist

Additional Notes

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 96.15385% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 86.12%. Comparing base (9ba90a3) to head (2a2569f).

Files Patch % Lines
tests/syntax/test_verifai_samplers.py 96.07% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #259 +/- ## ========================================== - Coverage 86.16% 86.12% -0.05% ========================================== Files 144 144 Lines 25012 25039 +27 ========================================== + Hits 21552 21564 +12 - Misses 3460 3475 +15 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.