Storing symbolic values outside of the test case lifetime

[tybug]

values = []
@settings(backend="crosshair", deadline=None, database=None)
@given(st.integers())
def f(x):
    values.append(x)

f()
# crosshair.util.CrosshairInternal: Numeric operation on symbolic while not tracing
print("values", values) 

We do indeed call post_test_case_hook -> deep_realize here, but presumably that returns the realized value rather than realizing the value and any references in place. Is deep-realizing all references even possible to achieve?

Arguably this is an abuse of hypothesis, but we do use this technique in hypothesis tests, eg. cc @Zac-HD since this might involve a wider viability discussion.

[pschanely]

values = []
@settings(backend="crosshair", deadline=None, database=None)
@given(st.integers())
def f(x):
    values.append(x)

f()
# crosshair.util.CrosshairInternal: Numeric operation on symbolic while not tracing
print("values", values) 

We do indeed call post_test_case_hook -> deep_realize here, but presumably that returns the realized value rather than realizing the value and any references in place. Is deep-realizing all references even possible to achieve?

Correct - the realization code just returns the realized values; the argument (and any other references) will remain symbolic. I don't know of any way we'd update all references. What's worse is that interactions with symbolics produce new symbolics, and it's quite a task to even understand what are the set of references that need to be updated.

Arguably this is an abuse of hypothesis, but we do use this technique in hypothesis tests, eg. cc @Zac-HD since this might involve a wider viability discussion.

For my plugin, calling the post_test_case_hook inside the test is sufficient to realize the value. One option is to declare this to be an official part of the interface (and heh, probably rename it so it's no longer called post_test_case_hook) and manually update hypothesis tests to call it as needed. I don't know how many hypothesis tests would need to be updated though.

OR, of course, we can just mark some tests as not runnable with solver-based backends somehow.

[pschanely]

Oh, also, I tend to dismiss naturally-occurring cases like this because they'll usually run afoul of CrossHair's nondeterminism checks anyway.

[tybug]

IMO the least bad option here is for hypothesis to expose a realize method which defers to the current backend (and abandon all pretenses of this being anything but symbolic realization) and manually update tests hypothesis-side, but I'm curious what Zac thinks & will defer to him. What I'd like to avoid is the combinatorial explosion of having to wrap with every backend's realize method because you want your test to be generic under backends, which means this has to be a hypothesis api.

[Zac-HD]

A realize method seems pretty reasonable to me.

The other thing to check proposed solutions against is the json-serialization crash that you currently see from crosshair backend + observability mode. This doesn't need to work by the same mechanism, but it'd be a good sign that the solution is general if it does! (we can obviously work around the crash too, just haven't yet)

[tybug]

OK, I'll play around with adding such a hook in the next few days, and see if it's general enough to address similar pain points like the observability interaction.

[tybug]

Let's call this fixed with the addition of a realize hook in the latest hypothesis release.