proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.63k stars 152 forks source link

Change verbosity of shrink disabled and persisted failure messages #453

Open grandizzy opened 1 month ago

grandizzy commented 1 month ago

I am working to implement an (indicatif based) progress bar for tests that use proptest and I need to suppress couple of messages that I really think should follow the info verbose rule (as they have to be explicitly configured, so not unexpected errors per se). Please let me know if you'd be open for a PR with changes below. Thank you!

         fork_output: &mut ForkOutput,
         is_from_persisted_seed: bool,
     ) -> Option<Reason> {
+        if self.config.max_shrink_iters() == 0 {
+            verbose_message!(
+                self,
+                INFO_LOG,
+                "Shrinking disabled by configuration"
+            );
+            return None
+        }
+
grandizzy commented 4 weeks ago

ping @matthew-russo any insights highly appreciated. thank you!

matthew-russo commented 3 weeks ago

Sorry for the delay, I had to step away for the past couple months to deal with some things at home. I'll be back working on things starting this weekend and will start triaging the issues that have come in.

I think the first suggestion is non-controversial and easy to accept.

I think the second suggestion isn't as straightforward since its only logging because there was a failure (hence its in the context of an error) and its anticipating that it may be running in a CI system, where users may only keep stdout. How are you going about suppressing it right now?

grandizzy commented 3 weeks ago

Thank you! There's no suppression rn so basically the progress bar gets flaky when proptest prints messages. I will make the PR for first case, what if for 2nd one we add an env setting that if turned on (default off) would skip printing such messages?