Tests failing

[FFY00]

Hey, I am packaging this for archlinux and the tests are failing. I am building from the latest release.

Are the tools needed for the tests? Some of them seem to pass without them installed.

The test output is also a bit fiddly, sometimes test_icarus.py fails, sometimes it doesn't.

============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /build/python-edalize/src/edalize-0.2.2
collected 41 items

tests/test_ascentlint.py .                                               [  2%]
tests/test_diamond.py ..                                                 [  7%]
tests/test_edam.py ......                                                [ 21%]
tests/test_ghdl.py .                                                     [ 24%]
tests/test_icarus.py F.                                                  [ 29%]
tests/test_icestorm.py ......                                            [ 43%]
tests/test_ise.py ..                                                     [ 48%]
tests/test_isim.py .                                                     [ 51%]
tests/test_modelsim.py .                                                 [ 53%]
tests/test_quartus.py .                                                  [ 56%]
tests/test_rivierapro.py .                                               [ 58%]
tests/test_spyglass.py FF                                                [ 63%]
tests/test_vcs.py ...                                                    [ 70%]
tests/test_veribleformat.py .                                            [ 73%]
tests/test_veriblelint.py .                                              [ 75%]
tests/test_verilator.py ...                                              [ 82%]
tests/test_vivado.py ...                                                 [ 90%]
tests/test_vunit.py ..                                                   [ 95%]
tests/test_xcelium.py .                                                  [ 97%]
tests/test_xsim.py .                                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_icarus __________________________________

    def test_icarus():
        import os
        import shutil
        from edalize_common import compare_files, setup_backend, tests_dir

        ref_dir      = os.path.join(tests_dir, __name__)
        paramtypes   = ['plusarg', 'vlogdefine', 'vlogparam']
        name         = 'test_icarus_0'
        tool         = 'icarus'
        tool_options = {
            'iverilog_options' : ['some', 'iverilog_options'],
            'timescale'        : '1ns/1ns',
        }

        (backend, work_root) = setup_backend(paramtypes, name, tool, tool_options, use_vpi=True)
        backend.configure()

        compare_files(ref_dir, work_root, ['Makefile',
                                           name+'.scr',
                                           'timescale.v',
        ])

        backend.build()
        compare_files(ref_dir, work_root, ['iverilog.cmd'])
>       compare_files(ref_dir, work_root, ['iverilog-vpi.cmd'])

tests/test_icarus.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ref_dir = '/build/python-edalize/src/edalize-0.2.2/tests/test_icarus'
work_root = '/tmp/icarus_tzu23ah2', files = ['iverilog-vpi.cmd']

    def compare_files(ref_dir, work_root, files):
        """Check that all *files* in *work_root* match those in *ref_dir*.

        If the environment variable :envvar:`GOLDEN_RUN` is set,
        the *files* in *work_root* are copied to *ref_dir* to become the new reference.
        """

        import os.path
        import shutil

        for f in files:
            reference_file = os.path.join(ref_dir, f)
            generated_file = os.path.join(work_root, f)

            assert os.path.exists(generated_file)

            if 'GOLDEN_RUN' in os.environ:
                shutil.copy(generated_file, reference_file)

            with open(reference_file) as fref, open(generated_file) as fgen:
>               assert fref.read() == fgen.read(), f
E               AssertionError: iverilog-vpi.cmd

tests/edalize_common.py:29: AssertionError
----------------------------- Captured stdout call -----------------------------
iverilog-vpi --name=vpi1 -lsome_lib -Isrc/vpi_1/ src/vpi_1/f1 src/vpi_1/f3
iverilog-vpi --name=vpi2   src/vpi_2/f4
iverilog -stop_module -c test_icarus_0.scr -o test_icarus_0 some iverilog_options
------------------------------ Captured log call -------------------------------
WARNING  edalize.icarus:icarus.py:81 qip_file.qip has unknown file type 'QIP'
WARNING  edalize.icarus:icarus.py:81 qsys_file has unknown file type 'QSYS'
WARNING  edalize.icarus:icarus.py:81 sdc_file has unknown file type 'SDC'
WARNING  edalize.icarus:icarus.py:81 bmm_file has unknown file type 'BMM'
WARNING  edalize.icarus:icarus.py:81 pcf_file.pcf has unknown file type 'PCF'
WARNING  edalize.icarus:icarus.py:81 ucf_file.ucf has unknown file type 'UCF'
WARNING  edalize.icarus:icarus.py:81 tcl_file.tcl has unknown file type 'tclSource'
WARNING  edalize.icarus:icarus.py:81 vhdl_file.vhd has unknown file type 'vhdlSource'
WARNING  edalize.icarus:icarus.py:81 vhdl_lfile has unknown file type 'vhdlSource'
WARNING  edalize.icarus:icarus.py:81 vhdl2008_file has unknown file type 'vhdlSource-2008'
WARNING  edalize.icarus:icarus.py:81 xci_file.xci has unknown file type 'xci'
WARNING  edalize.icarus:icarus.py:81 xdc_file.xdc has unknown file type 'xdc'
WARNING  edalize.icarus:icarus.py:81 bootrom.mem has unknown file type 'mem'
WARNING  edalize.icarus:icarus.py:81 c_file.c has unknown file type 'cSource'
WARNING  edalize.icarus:icarus.py:81 cpp_file.cpp has unknown file type 'cppSource'
WARNING  edalize.icarus:icarus.py:81 config.vbl has unknown file type 'veribleLintRules'
____________________________ test_spyglass_defaults ____________________________

    def test_spyglass_defaults():
        """ Test if the SpyGlass backend picks up the tool defaults """
        import os
        import shutil
        from edalize_common import compare_files, setup_backend, tests_dir

        ref_dir = os.path.join(tests_dir, __name__, 'defaults')
        paramtypes = ['vlogdefine', 'vlogparam']
        name = 'test_spyglass_0'
        tool = 'spyglass'
        tool_options = {}

        (backend, work_root) = setup_backend(
            paramtypes, name, tool, tool_options)
        backend.configure()

        compare_files(ref_dir, work_root, [
            'Makefile',
            'spyglass-run-design_read.tcl',
            'spyglass-run-lint_lint_rtl.tcl',
            name + '.prj',
        ])

        backend.build()
>       compare_files(ref_dir, work_root, [
            'spyglass.cmd',
        ])

tests/test_spyglass.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ref_dir = '/build/python-edalize/src/edalize-0.2.2/tests/test_spyglass/defaults'
work_root = '/tmp/spyglass_4py1xwbp', files = ['spyglass.cmd']

    def compare_files(ref_dir, work_root, files):
        """Check that all *files* in *work_root* match those in *ref_dir*.

        If the environment variable :envvar:`GOLDEN_RUN` is set,
        the *files* in *work_root* are copied to *ref_dir* to become the new reference.
        """

        import os.path
        import shutil

        for f in files:
            reference_file = os.path.join(ref_dir, f)
            generated_file = os.path.join(work_root, f)

            assert os.path.exists(generated_file)

            if 'GOLDEN_RUN' in os.environ:
                shutil.copy(generated_file, reference_file)

            with open(reference_file) as fref, open(generated_file) as fgen:
>               assert fref.read() == fgen.read(), f
E               AssertionError: spyglass.cmd

tests/edalize_common.py:29: AssertionError
----------------------------- Captured stdout call -----------------------------
sg_shell -enable_pass_exit_codes -tcl spyglass-run-design_read.tcl
sg_shell -enable_pass_exit_codes -tcl spyglass-run-lint_lint_rtl.tcl
------------------------------ Captured log call -------------------------------
WARNING  edalize.spyglass:spyglass.py:140 qip_file.qip has unknown file type 'QIP'
WARNING  edalize.spyglass:spyglass.py:140 qsys_file has unknown file type 'QSYS'
WARNING  edalize.spyglass:spyglass.py:140 sdc_file has unknown file type 'SDC'
WARNING  edalize.spyglass:spyglass.py:140 bmm_file has unknown file type 'BMM'
WARNING  edalize.spyglass:spyglass.py:140 pcf_file.pcf has unknown file type 'PCF'
WARNING  edalize.spyglass:spyglass.py:140 ucf_file.ucf has unknown file type 'UCF'
WARNING  edalize.spyglass:spyglass.py:140 xci_file.xci has unknown file type 'xci'
WARNING  edalize.spyglass:spyglass.py:140 xdc_file.xdc has unknown file type 'xdc'
WARNING  edalize.spyglass:spyglass.py:140 bootrom.mem has unknown file type 'mem'
WARNING  edalize.spyglass:spyglass.py:140 c_file.c has unknown file type 'cSource'
WARNING  edalize.spyglass:spyglass.py:140 cpp_file.cpp has unknown file type 'cppSource'
WARNING  edalize.spyglass:spyglass.py:140 config.vbl has unknown file type 'veribleLintRules'
__________________________ test_spyglass_tooloptions ___________________________

    def test_spyglass_tooloptions():
        """ Test passing tool options to the Spyglass backend """
        import os
        import shutil
        from edalize_common import compare_files, setup_backend, tests_dir

        ref_dir = os.path.join(tests_dir, __name__, 'tooloptions')
        paramtypes = ['vlogdefine', 'vlogparam']
        name = 'test_spyglass_0'
        tool = 'spyglass'
        tool_options = {
            'methodology': 'GuideWare/latest/block/rtl_somethingelse',
            'goals': ['lint/lint_rtl', 'some/othergoal'],
            'spyglass_options': ['handlememory yes'],
            'rule_parameters': ['handle_static_caselabels yes'],
        }

        (backend, work_root) = setup_backend(
            paramtypes, name, tool, tool_options)
        backend.configure()

        compare_files(ref_dir, work_root, [
            'Makefile',
            'spyglass-run-design_read.tcl',
            'spyglass-run-lint_lint_rtl.tcl',
            'spyglass-run-some_othergoal.tcl',
            name + '.prj',
        ])

        backend.build()
>       compare_files(ref_dir, work_root, [
            'spyglass.cmd',
        ])

tests/test_spyglass.py:63:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ref_dir = '/build/python-edalize/src/edalize-0.2.2/tests/test_spyglass/tooloptions'
work_root = '/tmp/spyglass_nal23vau', files = ['spyglass.cmd']

    def compare_files(ref_dir, work_root, files):
        """Check that all *files* in *work_root* match those in *ref_dir*.

        If the environment variable :envvar:`GOLDEN_RUN` is set,
        the *files* in *work_root* are copied to *ref_dir* to become the new reference.
        """

        import os.path
        import shutil

        for f in files:
            reference_file = os.path.join(ref_dir, f)
            generated_file = os.path.join(work_root, f)

            assert os.path.exists(generated_file)

            if 'GOLDEN_RUN' in os.environ:
                shutil.copy(generated_file, reference_file)

            with open(reference_file) as fref, open(generated_file) as fgen:
>               assert fref.read() == fgen.read(), f
E               AssertionError: spyglass.cmd

tests/edalize_common.py:29: AssertionError
----------------------------- Captured stdout call -----------------------------
sg_shell -enable_pass_exit_codes -tcl spyglass-run-design_read.tcl
sg_shell -enable_pass_exit_codes -tcl spyglass-run-lint_lint_rtl.tcl
sg_shell -enable_pass_exit_codes -tcl spyglass-run-some_othergoal.tcl
------------------------------ Captured log call -------------------------------
WARNING  edalize.spyglass:spyglass.py:140 qip_file.qip has unknown file type 'QIP'
WARNING  edalize.spyglass:spyglass.py:140 qsys_file has unknown file type 'QSYS'
WARNING  edalize.spyglass:spyglass.py:140 sdc_file has unknown file type 'SDC'
WARNING  edalize.spyglass:spyglass.py:140 bmm_file has unknown file type 'BMM'
WARNING  edalize.spyglass:spyglass.py:140 pcf_file.pcf has unknown file type 'PCF'
WARNING  edalize.spyglass:spyglass.py:140 ucf_file.ucf has unknown file type 'UCF'
WARNING  edalize.spyglass:spyglass.py:140 xci_file.xci has unknown file type 'xci'
WARNING  edalize.spyglass:spyglass.py:140 xdc_file.xdc has unknown file type 'xdc'
WARNING  edalize.spyglass:spyglass.py:140 bootrom.mem has unknown file type 'mem'
WARNING  edalize.spyglass:spyglass.py:140 c_file.c has unknown file type 'cSource'
WARNING  edalize.spyglass:spyglass.py:140 cpp_file.cpp has unknown file type 'cppSource'
WARNING  edalize.spyglass:spyglass.py:140 config.vbl has unknown file type 'veribleLintRules'
=========================== short test summary info ============================
FAILED tests/test_icarus.py::test_icarus - AssertionError: iverilog-vpi.cmd
FAILED tests/test_spyglass.py::test_spyglass_defaults - AssertionError: spygl...
FAILED tests/test_spyglass.py::test_spyglass_tooloptions - AssertionError: sp...
========================= 3 failed, 38 passed in 1.76s =========================

[GCHQDeveloper560]

The tests don't need the actual tools and instead use mock versions of the tools in tests/mock_commands. The GitHub CI setup is just Ubuntu without anything special installed, and tests Python 3.8 so there shouldn't be anything surprising there. Files like iverilog-vpi.cmd referenced in the errors are just one-liners with the command line arguments recorded.

A frequent source of test failures like these is command line arguments getting out of order, so they may be valid but don't match the reference output. I don't know why you might be seeing an intermittent failure. Unfortunately the test output doesn't actually show the differences between the two files reference in the error. Are you able to post them?

[olofk]

Thanks for working on the packaging. Unfortunately I haven't got a clue about what goes wrong here. Ideally the differences between the files should be shown in the output, but thanks to an annoying "feature" in pytest, this doesn't work (happens when the assert is in a different file from the test function).

If you can compare the files between ref_dir and work_root it would help a lot to analzye what goes wrong

[FFY00]

Sticking a print(os.listdir(work_root)) shows:

['Makefile', 'test_icarus_minimal_0.scr']

For test_icarus_minimal, there should be a spyglass.cmd right?

['Makefile', 'spyglass-run-some_othergoal.tcl', 'spyglass-run-lint_lint_rtl.tcl', 'spyglass-run-design_read.tcl', 'test_spyglass_0.prj']

For test_spyglass_tooloptions. Similarly, there should be a spyglass.cmd, right?

[GCHQDeveloper560]

The test failures print the name of the temporary directory used for the output, so you shouldn't have to dive into Python to check the test's output files. In the example you posted the failing tests put their outputs in /tmp/icarus_tzu23ah2/iverilog-vpi.cmd, /tmp/spyglass_4py1xwbp/spyglass.cmd, and /tmp/spyglass_nal23vau/spyglass.cmd.

For test_icarus_minimal I'd expect to see Makefile, test_icarus_minimal_0.scr, iverilog.cmd, and vvp.cmd. The two files you show are created from the configure command and the later two from build and run, so I don't know whether your list is from early in the run before they were created or whether something is going wrong in the build phase that is resulting in them not being created. Similarly for the Spyglass test it looks like the spyglass.cmd file that should be created by the build command is missing.

I may be wrong, but I would expect different errors if the files were totally missing. The earlier failures you posted seem more like the files are there but don't have the expected contents. If that's the case then posting the contents of the failing files from the temporary directory would be helpful.

[olofk]

Do we know if this is still an issue?