Riscof `coverage` command treats warnings as errors

[charmitro]

When running the riscof coverage command, warning are emitted as errors.

> riscof coverage --cgf-file riscv-arch-test/coverage/rvi.cgf --config=config.ini --suite riscv-arch-test/riscv-test-suite/rv32i_m/ --env=riscv-arch-test/riscv-test-suite/env
    INFO | Running Tests on Reference.
   ERROR | /home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S: Assembler messages:
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:37: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:47: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:62: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:137: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:193: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:333: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:343: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:398: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:493: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:608: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:718: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:828: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:933: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1043: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1148: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1253: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1363: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1473: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1583: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1643: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1648: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1653: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1658: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1663: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1668: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1673: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1678: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1683: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1748: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1858: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:1968: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2078: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2188: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2248: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2253: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2253: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2258: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2263: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2268: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2273: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2278: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2283: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2288: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2293: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2298: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2303: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2308: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2363: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2473: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2583: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2693: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2804: Warning: shift count out of range (-1 is not between 0 and 63)
/home/cmitrodimas/repositories/ACT-related/ACTs/riscv-arch-test/riscv-test-suite/rv32i_m/C/src/cadd-01.S:2914: Warning: shift count out of range (-1 is not between 0 and 63)

While on the other side, when running riscof run command, the warning are simply ignored, as they should be, and we are producing a tests report.

This is blocking #321 from producing a coverage report.

[pawks]

These warnings should not be causing any errors while producing the coverage reports. If this is indeed blocking, it is due to an actual error and not the warnings which are being printed.

[allenjbaum]

This is a weird artifact of the tools. These particular warnings are generated when the instructions are inside a

if / #endif clause that is not evaluated.

I thought you should only see them in RV64 because I rearranged code to avoid them in RV32, so I'm a bit surprised here. I also thought they should only appear if there is some real error, which enables all warnings - so you might be getting overwhelmed with these false positives and missing the real error.

On Sun, Apr 9, 2023 at 11:07 PM S Pawan Kumar @.***> wrote:

These warnings should not be causing any errors while producing the coverage reports. If this is indeed blocking, it is due to an actual error and not the warnings which are being printed.

— Reply to this email directly, view it on GitHub https://github.com/riscv-non-isa/riscv-arch-test/issues/331#issuecomment-1501436964, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXVJXVQHIINNCBF5APOHLXAOPQVANCNFSM6AAAAAAWWL6BB4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[charmitro]

This is a weird artifact of the tools. These particular warnings are generated when the instructions are inside a #if / #endif clause that is not evaluated. I thought you should only see them in RV64 because I rearranged code to avoid them in RV32, so I'm a bit surprised here. I also thought they should only appear if there is some real error, which enables all warnings - so you might be getting overwhelmed with these false positives and missing the real error.

Exactly, it turned out that the real error was down below, making me unsure whether the warnings treated as errors or the python errors below it.

Since it's still unclear what the issue is to me, I'm pasting this for reference, in-case you have previously encountered anything similar.

$ ~/repositories/ACT-related/ACTs$ riscv_isac --verbose info coverage --decoder-name rvopcodesdecoder --decoder-path ./rvop-plugin -t trace.log --parser-name spike  -o coverage.rpt -e riscof_work/src/fleq_b1-01.S/ref/ref.elf -c riscv-ctg/sample_cgfs/zfa/fleq.s.cgf
    INFO | ****** RISC-V ISA Coverage 0.17.0 *******
    INFO | Copyright (c) 2020, InCore Semiconductors Pvt. Ltd.
    INFO | All Rights Reserved.
Traceback (most recent call last):
  File "/home/cmitrodimas/.local/bin/riscv_isac", line 8, in <module>
    sys.exit(cli())
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/riscv_isac/main.py", line 136, in coverage
    isac(output_file,elf,trace_file, window_size, expand_cgf(cgf_file,int(xlen),int(flen),log_redundant), parser_name, decoder_name, parser_path, decoder_path, detailed, test_label,
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/riscv_isac/cgf_normalize.py", line 565, in expand_cgf
    cgf = utils.load_cgf(cgf_files)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/riscv_isac/utils.py", line 163, in load_cgf
    safe_yaml.dump(dict(safe_yaml.load(fp)),string_stream)
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/ruamel/yaml/main.py", line 434, in load
    return constructor.get_single_data()
  File "/home/cmitrodimas/.local/lib/python3.10/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 706, in _ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 724, in _ruamel_yaml.CParser._compose_document
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
  File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
  File "_ruamel_yaml.pyx", line 743, in _ruamel_yaml.CParser._compose_node
ruamel.yaml.composer.ComposerError: found undefined alias
  in "<file>", line 9, column 11 

[pawks]

The cgfs are yaml files underneath. Looks like your file has an alias but the file with the alias has not been passed as a cgf argument. Lets take the cgf for fle.s. In this file, the all_fregs, all_regs and sfmt_op_comb are aliases which are defined in dataset.cgf. Inorder for them to be accurately expanded, you will have to pass the dataset followed by fle.s in the CLI(the order matters). ISAC, CTG and RISCOF can take multiple CGF files by using the same option multiple times. Example RISCOF command:

riscof --verbose debug coverage --suite /path/to/suite --env /path/to/suite --cgf /path/to/dataset --cfg /path/to/fle.s

[allenjbaum]

This appear is to be a test-setup issue which exposes warnings that should be suppressed. The warnings will be fixed in the next PR.