Support for better error trace log

[aneels3]

Hi @mballance I am creating this issue to have a better error trace for pyvsc errors. Sometimes it is difficult to understand the error and from where it is originating. I don't have a proper test case for it but sometimes program execution takes an undesirable long time to execute. For eg. 1) If some variable is not defined as pyvsc types, we won't get any warning/error for it for a constraint (Which never works as expected) and the program goes on an infinite loop. 2) Sometimes by looking at the current error log. It is difficult to point out from where it is originating (In a few cases).

Can you provide a proper error tracing for pyvsc?

[mballance]

Hi @aneels3, Thanks for raising this issue. As you note, this isn't a straight-forward bug that we can fix based on a testcase. It will require more investigation and design to reach a solution.

From your note, it sounds like slow performance (ie a 'hang') is the most user-visible aspect of an issue that requires debugging. Do you know whether the hang is within the PyVSC solver itself or within the RISCV-DV generator? Depending on where you tend to experience slow performance/hangs I'll focusing logging and debug aids differently.

You mention that referencing variables of non-pyvsc type tends to create problems. How frequently does your code do this intentionally? If the classes you randomize are not intended to contain fields of non-pyvsc type, we could add a check for these.

On your second point (identifying problems based on the error log), are you referring specifically to solve failures? If so, would more information tying a solve failure back to a source location help? Would better analysis to identify the conflicting variables help? Something else?

It will be important to get some testcases collected at some point. Please give some thought to what would be helpful here. They certainly don't need to be minimized. They could, for example, take the form of a set of changes to make to the RISCV-DV code that provokes a hang, difficult-to-understand solve failure, etc.

Look forward to working with you on this!

Best Regards, Matthew

[aneels3]

Okay, I think I got some examples where I think PyVSC needs some enhancement for better error log messages to debug and correct the issue faster.

Test 1:

import vsc

@vsc.randobj
class error_test:
    def __init__(self):
        self.a = vsc.rand_int_t()
        self.b = vsc.rand_bit_t(1)

    @vsc.constraint
    def error_constraint(self):
            self.a == 1
            self.b == 2  # This should be either 0 or 1 

er_inst = error_test()
for i in range(5):
    er_inst.randomize()
    print("a = ", er_inst.a, "b = ", er_inst.b)

Output:

Problem Constraints:
Constraint 1:
(b == 2);
(b == 2);

Traceback (most recent call last):
  File "error_test1.py", line 17, in <module>
    er_inst.randomize()
  File "/home/anil/.local/lib/python3.6/site-packages/vsc/rand_obj.py", line 135, in randomize
    raise e
  File "/home/anil/.local/lib/python3.6/site-packages/vsc/rand_obj.py", line 132, in randomize
    debug=debug)
  File "/home/anil/.local/lib/python3.6/site-packages/vsc/model/randomizer.py", line 703, in do_randomize
    r.randomize(ri, bounds_v.bound_m)
  File "/home/anil/.local/lib/python3.6/site-packages/vsc/model/randomizer.py", line 195, in randomize
    self.create_diagnostics(active_randsets))
vsc.model.solve_failure.SolveFailure: solve failure

The above test is pretty simple and it's quite obvious that the self.b can hold the values 0 and 1 only. So, Instead of the Problem Constraints: message, If it says something like, self.b is of 1-bit length but the value provided in the constraint is out of range along with the constraint_name from where it has generated the error and the corresponding line number. What do you think about this?

I think this will trigger the user to check for the declaration of the variable (Like data_type, bit length, etc) in the constructor function and resolve the issue.

When the constraint is complex, the user gets multiple problem constraint messages. I think this would help the user to debug efficiently in those cases.

I will post the other example in the next comment.

Regards, Anil

[mballance]

Hi @aneels3, I like the idea of performing some static checks on the constraints to catch cases like you show above. I've added the domain check suggested by the example above, and will build on that as other similar cases emerge.

I've also reworked the solve-failure diagnostics a bit, and I feel the results are much better. Release 0.5.1 contains the changes. The updated code analyzes the group of constraints that failed to identify:

• single constraints that fail
• pairs of constraints that fail
• triples of constraints that fail

Here's an example and the output:

        @vsc.randobj
        class my_c(object):

            def __init__(self):
                self.a = vsc.rand_uint8_t()
                self.b = vsc.rand_uint8_t()
                self.c = vsc.rand_uint8_t()
                self.d = vsc.rand_uint8_t()

            @vsc.constraint
            def abcd_c(self):
                self.c > self.b
                self.a < self.b
                self.c < self.a
                self.c < 10
                self.a > 0
                self.b <= 20

        it = my_c()

        try:        
            it.randomize()
            self.fail("Expected a solve failure")
        except vsc.SolveFailure as e:
            print("Exception: " + str(e.diagnostics))

Output:

Problem Set: 3 constraints
  /project/fun/pyvsc/pyvsc-lint/ve/unit/test_solve_failure.py:57:
    (c > b);
  /project/fun/pyvsc/pyvsc-lint/ve/unit/test_solve_failure.py:58:
    (a < b);
  /project/fun/pyvsc/pyvsc-lint/ve/unit/test_solve_failure.py:59:
    (c < a);

Note that the source file/line number is reported.

Look forward to your feedback.

Best Regards, Matthew

[aneels3]

Hi @mballance This looks really good for now. Thanks a lot!

I will provide you with some more test-case for further enhancement.

Regards, Anil

[mballance]

Closing this out for now. Please raise new issues as further enhancements are needed.