es-ude / elastic-ai.creator

elastic ai.creator
MIT License
16 stars 2 forks source link

HW simulation tests #317

Closed glencoe closed 1 year ago

glencoe commented 1 year ago

related to: #316 work on this already started on: #315

To help us automate tests that compare hw simulation results to software module results, I propose an approach as follows:

def my_simulation_test():
  unit_under_test = my_software_module()
  inputs = [0, 1, 2, 3]
  expected_outputs = unit_under_test(inputs)
  workdir = Directory("my_simulation_test_dir")
  test_bench = TestBench(unit_under_test.create_design("my_simulation_test"), unit_under_test.prepare_simulation_data(inputs))
  runner = TestBenchRunner(test_bench, workdir=workdir)
  runner.prepare()
  runner.run()
  result = runner.getReportedContent()
  outputs = list(map(float, result))
  assert expected_outputs == outputs

The way the input data has to be prepared can be solved on a case-to-case basis. For TestBench, TestRunner i suggest the interfaces below. Note that the Directory protocol is extended here to return the absolute path, such that the TestBenchRunner can set this as a working directory for the external call to the simulation tool.

class TestBench(Protocol):
    def save_to(self, destination: Directory): ...

    @property
    def name(self) -> str: ...
class Directory(Protocol):
    @property
    def absolute_path(self) -> str: ...

    def make_subdirectory(self) -> "Directory": ...

class TestBenchRunner(Protocol):
    def prepare(self):
        """create directories and save design files based on `workdir`"""
        pass

    def run(self):
        """run the simulation and capture results"""
        pass

    def getReportedContent(self) -> list[str]:
        """Return the content that was captured during simulation. 
           This omits information that is generated by the simulation tool,
           ie., this only returns a list of lines as passed to the `report`
           operation in vhdl.
        """
        pass

    def getFullReport(self) -> list[dict]:
        """Returns a dictionary representing the distinct values parsed from the captured
           simulation output. E.g., for ghdl this would include `"time"`, `"source"`, 
           `"line"`, etc.
        """
        pass

    def getRawReport(self) -> list[str]:
        """Returns the verbatim report line by line as returned by the simulation tool"""
        pass
glencoe commented 1 year ago

@julianhoever @Sillobear now that this is better specified, please feel free contact me and take it over if you want.