[Medium] High bias in cases of nested constraints

[ShreyanJabade]

Hello, Specifying constraints on variables in members of list (eg. self.c1[0].a[0].value) leads to very high bias. Specifically, the highest value satisfying constraint is returned almost always. The same constraint given without list works normally as demonstrated in the output below. I request you to look into this aspect. Thank you

import vsc 
from vsc import *
import matplotlib.pyplot as plt

@vsc.randobj
class Parent:
    def __init__(self):
        self.id = 0
        self.c1 = vsc.rand_list_t(vsc.attr(Child1()))
        for i in range(10):    
            self.c1.append(vsc.attr(Child1()))

        self.c2 = vsc.rand_list_t(vsc.attr(Child2()))
        for i in range(10):
            self.c2.append(vsc.attr(Child2()))

        self.non_list_child = vsc.rand_attr(Child1())

        self.top_lvl_val = vsc.rand_uint8_t(0)

    @vsc.constraint
    def parent_c(self):
        self.top_lvl_val < 10       # Works fine

        # Presence of list causes high skewness
        self.c1[0].one_lvl_below_val < 10
        self.c1[0].a[0].value < 10      # Two levels below

        self.non_list_child.one_lvl_below_val < 10      # Works fine

@vsc.randobj
class Field():
    def __init__(self, name, def_value):
        self.name = name
        self.value = vsc.rand_uint8_t(def_value)

@vsc.randobj
class Child1:
    def __init__(self):
        self.a = vsc.rand_list_t(vsc.attr(Field('a', 10)))
        for i in range(5):    
            self.a.append(vsc.attr(Field('a', 10)))

        self.b = vsc.rand_list_t(vsc.attr(Field('b', 10)))
        for i in range(5):    
            self.b.append(vsc.attr(Field('b', 10)))

        self.one_lvl_below_val = vsc.rand_uint8_t(0)

@vsc.randobj
class Child2:
    def __init__(self):
        self.x = vsc.rand_list_t(vsc.attr(Field('x', 10)))
        for i in range(5):    
            self.x.append(vsc.attr(Field('x', 10)))

        self.y = vsc.rand_list_t(vsc.attr(Field('y', 10)))
        for i in range(5):    
            self.y.append(vsc.attr(Field('y', 10)))

    @vsc.constraint
    def test_c(self):
        self.x[0].value < self.x[1].value

inst=Parent()

top_lvl = []
one_below = []
two_below = []
non_list = []

for i in range(1000):
    inst.randomize()
    top_lvl.append(inst.top_lvl_val)
    one_below.append(inst.c1[0].one_lvl_below_val)
    two_below.append(inst.c1[0].a[0].value)
    non_list.append(inst.non_list_child.one_lvl_below_val)

plt.hist(top_lvl)
plt.title("Top level variable values")
plt.show()

plt.hist(one_below)
plt.title("Variables one level below (with list)")
plt.show()

plt.hist(two_below)
plt.title("Variables two levels below (with lists)")
plt.show()

plt.hist(non_list)
plt.title("Child not in list")
plt.show()

Output is:

[ShreyanJabade]

Related to #81, there are no more unwanted solutions due to overflow. I checked the distribution and it is similar to this issue. So have added this comment here.

import vsc
import matplotlib.pyplot as plt

@vsc.randobj
class Test:
    def __init__(self):
        self.a = vsc.rand_uint8_t()
        self.b = vsc.rand_uint8_t()

    @vsc.constraint
    def test_c(self):
        self.a*self.b == 60

inst = Test()

a_vals = []
for i in range(1000):
    inst.randomize()
    a_vals.append(inst.b)

plt.hist(a_vals)
plt.show()

Output is:

[mballance]

Hi @ShreyanJabade, I've corrected the original issue in #83 in the 0.4.5 release. The second issue is significantly different.

PyVSC uses a SMT solver to create valid solutions. SMT solvers are not designed to produce solutions with a random distribution, so PyVSC introduces randomness by randomly selecting a set of variables to constrain to a randomly-selected value. This approach has limitations when the solution space of a variable is very small and the domain is also non-trivial to establish. To the best of my understanding, that is what is happening in the second case.

With respect to the second case, do the constraints in use on your project heavily involve multiplication? Is the solution space also very sparse like that shown above?

[ShreyanJabade]

Hello, The nested constraints distribution is working perfectly now. Thank you for your consideration. The constraints in the project can involve multiplication to some extent, but not exclusively. Also, it is difficult to gauge the sparsity in general, since the constraint writer can have any set of constraints.