search

phanrahan / mantle

mantle library
Other
42 stars 10 forks source link

Testing interface for primitives #48

Open leonardt opened 6 years ago

leonardt commented 6 years ago

Proposal: test(mantle.operator.and_, operator.and_, [Bit, Bits, UInt, SInt])

Where test(mantle_primitive_op, python_op, types) will test mantle_primitive_op in the Python simulator, Verilator, and coreir simulator using python_op as a gold function. type is a list of magma types to test the operations on.

phanrahan commented 6 years ago

This looks good.

I think we should have a good default way of doing this, but maybe we should provide options for how to do the test. For example, what bit width? Or whether to use exhaustive or random testing? Also, what do you propose as the default test?

leonardt commented 6 years ago

That makes sense, we can add support for bit width as a single value or a range of values. Then another parameter indicating a strategy or list of strategies, for now we can start with exhaustive and random as the two supported strategies. The random strategy should have some configuration options (e.g. seed, number of tests).

New proposed syntax: test(mantle.operator.and_, operator.and_, [Bit, Bits, UInt, SInt], strategy=magma.testing.strategy.RandomStrategy(seed=315452, num_tests=500)) I think it's a clean interface to have an abstract class Strategy that can be subclassed to create new testing strategies.

For the default test, perhaps we can do exhaustive up to 4 bits and random up to 16 bits? Will need to think a bit and do some research on a good scheme for randomized testing that will get good coverage. Let me know if you have any references you can suggest.