search

proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.63k stars 152 forks source link

Provide access to state for (debug) printing transitions #456

Closed thomaseizinger closed 3 weeks ago

thomaseizinger commented 1 month ago

We are making heavy uses of sample::Index to deterministically access state as part of our transitions. This works well functionally but makes debugging difficult because it obfuscates, what the transition actually does.

Applying transition 15/18: SendQueryToDnsResource { r_idx: Index(3156979778377414915), r_type: A, query_id: 10855, dns_server_idx: Index(4470972928535618001) }

The above is a transition that simulates sending a DNS query to a system but the domain that is being queried and which server it uses are part of the state and thus determined as part of applying the transition.

It would be useful if the transitions would have to implement a specialised Debug trait that gives access to the current state and allowed resolving these indices to the actual value it is being applied.

matthew-russo commented 3 weeks ago

Thanks for the feedback. I'll be doing some triaging this weekend after being away for a bit and will take a look at what this may entail. Do you have a workaround you're currently using or is this an active thorn in your side?

thomaseizinger commented 3 weeks ago

Thanks for the feedback. I'll be doing some triaging this weekend after being away for a bit and will take a look at what this may entail. Do you have a workaround you're currently using or is this an active thorn in your side?

Not really using a workaround at the moment. We usually turn up logs of the production code which often contains the same, useful values (like domains & IPs in this case).

This isn't very critical but more a QoL thing as you tagged it as!

thomaseizinger commented 3 weeks ago

Perhaps I can hijack this thread to ask a question on whether we are doing things correctly. Looking at some of the code that proptest-state-machine executes, it looks like I should be able to directly return transitions that are based off the current state:

https://github.com/proptest-rs/proptest/blob/24412f5f10a365c8f12a71e3b3a053677ced91b0/proptest-state-machine/src/strategy.rs#L191-L194

There are two problems with this:

  1. ReferenceStateMachine::transitions doesn't allow returning borrowed data from Self::State, requiring lots of cloning
  2. It doesn't appear to play very well with shrinking. In particular (if I remember correctly) I've found that a strategy based on the initial state, once shrinking is applied to the state doesn't get re-initialised with the shrunk state but keeps its old state. Hence, if I clone the current state into the transition to return actual values instead of indices, then things don't work anymore because the values in the transition don't make any sense in the context of the new state.

If a transition could reference the current state and as in the above example, just select a DNS server or a domain, then this issue would also be resolved.

thomaseizinger commented 3 weeks ago

Do you have a workaround you're currently using or is this an active thorn in your side?

I noticed that one way of working around this could be to override test_sequential on StateMachineTest.

thomaseizinger commented 3 weeks ago

We've moved away from indices in most places, making this problem go away.