As I was taking a look at student's code for lab 2 in ECE 5745, I noticed an issue which we sometimes see. I am showing the test results below for one group. I am running the tests in two ways: first I run just the tests in lab2-sort, and then I run all of the tests. We would expect the test results to be the same regardless of how I actually run the tests:
Notice that all of the tests pass when I run them all together, but there is one test failure when I run just the tests in lab2-sort. This final test includes some random delays. We have seen this happen with our Verilog unit testing framework as well. Tests will pass when we run the entire suite but will fail when we run just one test case, or the other way around.
I always kind of knew what was wrong. Basically the random number generator is producing different random numbers based on how we run the tests. So for example, based on whether or not you run a test as part of a suite or in isolation, the random delays for a test source/sink will be different. Here is another example: first I run all of the tests for the TestRandomDelay model (which is used in the test source/sink) and then I run just one of the test cases. I am showing the output just for this test case in both contexts:
You can clearly see that there are different random delays based on whether we run a bunch of tests or one test. Note, that I did try and set the seed to a known value at the top of the TestRandomDelay model -- so this is not because we are seeding the random number generator from the time of day or anything. Again, we were seeing similar issues with our Verilog unit testing framework. This issue can be very frustrating because it is hard to reproduce the error in isolation, and it can also be confusing to new users.
The way to address this issue is that any object which needs random numbers must have its own random number generator. It cannot rely on a global shared random number generator. Setting the seed at the top of a model file will not help -- because that seeds the random number generator at import time. Here is a good article about random number seeding in SystemVerilog:
That paper mentions: "In class-based testbenches, objects will represent the components in the testbench and generate the random stimulus that passes into the design. In order to ensure random stability, each object also must have its own independent random number generator, and consequently, its own seed so previous random stimulus can be reproduced."
So I implemented this approach in PyMTL. The key insight is that Python gives you an easy way to create a private, non-shared random number generator. You can just instantiate a Random object:
>>> import random
>>> rgen = random.Random()
>>> rgen.randomint(0,10)
So I think anywhere in our PyMTL code where we want to generate a random number, we should always create a dedicated Random object for that purpose. So test delay models should have a member which is the random number generator. You can see example here:
Notice that I pass in the seed value as a constructor argument so that a parent can (and probably should) ensure that multiple instances of TestRandomDelay have different seeds. We should probably do the same thing even when we are generating random data in test scripts:
After making this change, the line traces for the TestRandomDelay_test.py test script look identical regardless of whether or not we run all of the test cases or a single test case in isolation.
As I was taking a look at student's code for lab 2 in ECE 5745, I noticed an issue which we sometimes see. I am showing the test results below for one group. I am running the tests in two ways: first I run just the tests in lab2-sort, and then I run all of the tests. We would expect the test results to be the same regardless of how I actually run the tests:
Notice that all of the tests pass when I run them all together, but there is one test failure when I run just the tests in lab2-sort. This final test includes some random delays. We have seen this happen with our Verilog unit testing framework as well. Tests will pass when we run the entire suite but will fail when we run just one test case, or the other way around.
I always kind of knew what was wrong. Basically the random number generator is producing different random numbers based on how we run the tests. So for example, based on whether or not you run a test as part of a suite or in isolation, the random delays for a test source/sink will be different. Here is another example: first I run all of the tests for the
TestRandomDelaymodel (which is used in the test source/sink) and then I run just one of the test cases. I am showing the output just for this test case in both contexts:You can clearly see that there are different random delays based on whether we run a bunch of tests or one test. Note, that I did try and set the seed to a known value at the top of the
TestRandomDelaymodel -- so this is not because we are seeding the random number generator from the time of day or anything. Again, we were seeing similar issues with our Verilog unit testing framework. This issue can be very frustrating because it is hard to reproduce the error in isolation, and it can also be confusing to new users.The way to address this issue is that any object which needs random numbers must have its own random number generator. It cannot rely on a global shared random number generator. Setting the seed at the top of a model file will not help -- because that seeds the random number generator at import time. Here is a good article about random number seeding in SystemVerilog:
http://www.doulos.com/knowhow/sysverilog/SNUG13_random_stability/SNUG2013_SV_Random_Stability_paper.pdf
That paper mentions: "In class-based testbenches, objects will represent the components in the testbench and generate the random stimulus that passes into the design. In order to ensure random stability, each object also must have its own independent random number generator, and consequently, its own seed so previous random stimulus can be reproduced."
So I implemented this approach in PyMTL. The key insight is that Python gives you an easy way to create a private, non-shared random number generator. You can just instantiate a Random object:
So I think anywhere in our PyMTL code where we want to generate a random number, we should always create a dedicated Random object for that purpose. So test delay models should have a member which is the random number generator. You can see example here:
https://github.com/cornell-brg/pymtl/blob/52325306/pclib/test/TestRandomDelay.py#L19-L29
Notice that I pass in the seed value as a constructor argument so that a parent can (and probably should) ensure that multiple instances of TestRandomDelay have different seeds. We should probably do the same thing even when we are generating random data in test scripts:
https://github.com/cornell-brg/pymtl/blob/52325306/pclib/cl/adapters_test.py#L74-L78
After making this change, the line traces for the TestRandomDelay_test.py test script look identical regardless of whether or not we run all of the test cases or a single test case in isolation.