VUnit / vunit

VUnit is a unit testing framework for VHDL/SystemVerilog
http://vunit.github.io/
Other
746 stars 263 forks source link

check lib: Prefix binary and hex numbers in log output #451

Open cmarqu opened 5 years ago

cmarqu commented 5 years ago

(Call for comments.)

Right now the check library displays output like "Got 1111_1010 (250). Expected 249 (1111_1001).". I think it would be nice and also better for machine readability if every number could be interpreted without taking context into account, meaning, the "1111_1010" and "1111_1001" should be marked as being binary. Same for hexadecimal numbers.

(Personally, I would use neither VHDL syntax nor Verilog syntax but "0b" and "0x".)

LarsAsplund commented 5 years ago

I'm ok with adding a prefix. First I thought VHDL style would be the correct way but I have to think about that some more. If we end up with more logging support for SV it would be nice with a consistent format.

The current format was designed for human readability

check_equal outputs two representations of the vectors. One nibble string and one integer. So 10001100 becomes 1000_1100 (140). The idea is that regardless if you consider the vector as a number or a set of unrelated bits there should be a suitable format. The nibble string is also fairly easy to translate into a hex number is your head - 8c

Your use case seems to be about machine readability and for that we should really think about other file formats. Today we have CSV (we've also discussed JSON) but that puts all of the message in one "column". A machine would prefer the expected and received values in different columns.

@cmarqu Can you share a bit more about your use case?

tmeissner commented 5 years ago

I would also like the addition of prefixes which show the format more explicitily. Machine readability should also be taken into account, although I don't have an specific format in mind at the moment.

LarsAsplund commented 5 years ago

@cmarqu Any comments on your use case or should I proceed and implement what's been suggested?

cmarqu commented 5 years ago

Use case is actually just that I wanted to write some similar library in Python for use with eg. cocotb, and thought that this number format is about the only thing I would change in my version, the rest is very nice.

While it's true that proper machine readability requires more markup like in JSON, it's still nice to just use grep/sed/etc. on the log output you already have and not accidentally misinterpret a value.

Or, as part of the daily work, you sometimes come across log output where you don't immediately know what tool produced it, and thus have to guess what a "100" is likely to mean. In such cases I think to myself "why couldn't they just make it explicit?"

And it would be a mild form of https://en.m.wikipedia.org/wiki/Poka-yoke

LarsAsplund commented 5 years ago

@cmarqu Thanks for making that clear. That would further motivate why we should use the most commonly used prefix. I don't mind if our format is used for grepping strings. What I do mind is if people start to build more permanent tools that rely on our format. Log parsing is heavily used with UVM to figure out what's going on but I think that's a poor practice to handle missing functionality that I don't want the VUnit community to adopt. If we were to encourage such approaches we would also make our text outputs and formats a public APIs and that's an additional constraint I want to avoid