search

ghdl / docker

Scripts to build and use docker images including GHDL
40 stars 10 forks source link

Code coverage stopped working after switch to Debian Bullseye and gcc 9.1 #42

Closed abyszuk closed 2 years ago

abyszuk commented 3 years ago

I have a CI system setup that automatically runs simulation tests for our codebase and generates artifacts with code coverage reports. Today we've noticed that all of our pipelines started failing all of a sudden. Quick investigation suggests that it's most likely caused by the new version of ghdl/vunit:gcc images that we use. Errors vary wildly between testbenches, but all are caused by the ghdl-gcc inability to compile VHDL, run testbench or generate code coverage data. A few log extracts below (we use Vunit btw.):

Compiling into sci_master_lib: common/vhd/interfaces/wishbone/sci_master/rtl/sci_master_top.vhd                                                      failed
=== Command used: ===
/usr/local/bin/ghdl -a --workdir=/builds/cce/cce/vunit_out/ghdl/libraries/sci_master_lib --work=sci_master_lib --std=08 -P/builds/cce/cce/vunit_out/ghdl/libraries/vunit_lib -P/builds/cce/cce/vunit_out/ghdl/libraries/osvvm -P/builds/cce/cce/vunit_out/ghdl/libraries/sci_master_lib -frelaxed -fprofile-arcs -ftest-coverage /builds/cce/cce/common/vhd/interfaces/wishbone/sci_master/rtl/sci_master_top.vhd
=== Command output: ===
during IPA pass: profile
/builds/cce/cce/common/vhd/interfaces/wishbone/sci_master/rtl/sci_master_top.vhd: In function ‘sci_master_lib__sci_master_top__ARCH__rtl__P2__PROC’:
/builds/cce/cce/common/vhd/interfaces/wishbone/sci_master/rtl/sci_master_top.vhd:165: internal compiler error: in coverage_begin_function, at coverage.c:656
  165 |   sci_mode_o                   <= (sci_clk_i or master_command_clk_high) and (not master_command_clk_low);
      | 
0x60ead2 coverage_begin_function(unsigned int, unsigned int)
    ../../gcc-srcs/gcc/coverage.c:656
0xb01d14 branch_prob(bool)
    ../../gcc-srcs/gcc/profile.c:1233
0xc34b62 tree_profiling
    ../../gcc-srcs/gcc/tree-profile.c:793
0xc34b62 execute
    ../../gcc-srcs/gcc/tree-profile.c:898
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
/usr/local/bin/ghdl: exec error
l_serdes_stop_bit_9770f3ec736766095851a7d3f5f1fec2ae269065/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_zero_bits_607788854b84d43782763aab602c90277ff3345e/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_analogue_patterns_461895921170e375859f53d6a68dd9be965fc4c2/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_gated_transmission_a4d899af41db12640e66feea19a4c2568eae9de0/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_gated_to_normal_transmission_99cc1732306fbbbf9906b29a21db3e6b28fc45bd/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_address_patterns_6cb5d8aa3912c9a556294a7746c1c9f11a9c1979/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_parity_bits_a98bccb3b3c40fb4dba71942347a6558dd8ad898/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_break_sequence_133e6f90afa3241856d0fbd9fcc54bb706740063/coverage
WARNING - Missing coverage directory: /builds/cce/cce/vunit_out/test_output/uut.aol_serdes_vunit_tb.aol_serdes_start_bit_082961e4c64492079a2092f47e66325b09c7d3ce/coverage
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/vunit/ui/__init__.py", line 726, in main
    all_ok = self._main(post_run)
  File "/usr/local/lib/python3.9/dist-packages/vunit/ui/__init__.py", line 772, in _main
    all_ok = self._main_run(post_run)
  File "/usr/local/lib/python3.9/dist-packages/vunit/ui/__init__.py", line 819, in _main_run
    post_run(results=Results(self._output_path, simulator_if, report))
  File "/builds/cce/cce/analog_optical_link/common/tb/run.py", line 33, in post_run
    results.merge_coverage(file_name="coverage_data")
  File "/usr/local/lib/python3.9/dist-packages/vunit/ui/results.py", line 33, in merge_coverage
    self._simulator_if.merge_coverage(file_name=file_name, args=args)
  File "/usr/local/lib/python3.9/dist-packages/vunit/sim_if/ghdl.py", line 439, in merge_coverage
    assert len(gcda_dirs) == 1, "Expected exactly one folder with gcda files"
AssertionError: Expected exactly one folder with gcda files
umarcor commented 3 years ago

Hi @abyszuk! Let's try to track this down!

Quick investigation suggests that it's most likely caused by the new version of ghdl/vunit:gcc images that we use.

I do think there might be a problem as a result of changing from Debian Buster to Debian Bullseye, and from GCC 8 to GCC 9. However...

Errors vary wildly between testbenches, but all are caused by the ghdl-gcc inability to compile VHDL, run testbench or generate code coverage data. A few log extracts below (we use Vunit btw.):

I don't think it's caused by the ghdl-gcc inability to compile VHDL or run testbench. Precisely, the images that are used for building ghdl/vunit:* do explicitly pass the whole test suite successfully. See https://github.com/ghdl/docker/actions/runs/1134401039. Therefore, it is guaranteed that compiling and simulating does work.

I believe the problems are caused by using the coverage features (either in GHDL or in VUnit). Can you please confirm? I.e. can you execute those failing examples disabling coverage features?

Maybe related: ghdl/ghdl#1815 and ghdl/ghdl#1817.

tgingold commented 3 years ago

Yes, this crash was fixed by ghdl/ghdl#1817.

abyszuk commented 3 years ago

Hello, yes I also think it's only related to code coverage features. Unfortunately I've started my vacations a few days ago, so I'll be able to give more feedback in ~2 weeks. If the crash is fixed then that's great news, thanks Tristan! I think there's still issue of missing gcda/gcno files. For now I can only guess that maybe the output file hierarchy changed in newer GCC versions (in that case it would look like more of a Vunit issue), but I can investigate that once I get back to the office.

abyszuk commented 2 years ago

Hi @umarcor ! Sorry for the long wait! I've managed to reproduce the problem with much smaller example. Essentially, it looks that testbench executables made with new GCC version don't create the *.gcda files required by GCOV. I did my tests with two GHDL docker images that I've had at hand:

Steps to reproduce:

  1. Download the code coverage example project from here
  2. Run manually steps from the example 
    cd projects/full_adder
ghdl -a -Wc,-fprofile-arcs -Wc,-ftest-coverage adder.vhd
ghdl -a adder_tb.vhd
ghdl -e -Wl,-lgcov adder_tb
ghdl -r adder_tb
# Generate code coverage report using gcov
gcov -s $PWD adder.vhd
# Generate code coverage report using lcov
lcov -c -d . -o adder_tb.info
genhtml -o html adder_tb.info
  3. Everything looks similar up to ghdl -r adder_tb stage. Old version generates adder.gcda file after simulation, while new version doesn't do that.

Console output for old version:

bash-4.4$ ls
adder.vhd  adder_tb.vhd
bash-4.4$ ghdl -a -Wc,-fprofile-arcs -Wc,-ftest-coverage adder.vhd 
bash-4.4$ ls
adder.gcno  adder.o  adder.vhd  adder_tb.vhd  work-obj93.cf
bash-4.4$ ghdl -a adder_tb.vhd 
bash-4.4$ ghdl -e -Wl,-lgcov adder_tb 
bash-4.4$ ls
adder.gcno  adder.o  adder.vhd  adder_tb  adder_tb.o  adder_tb.vhd  e~adder_tb.o  work-obj93.cf
bash-4.4$ ghdl -r adder_tb
adder_tb.vhd:53:7:@8ns:(assertion note): end of test
bash-4.4$ ls
adder.gcda  adder.gcno  adder.o  adder.vhd  adder_tb  adder_tb.o  adder_tb.vhd  e~adder_tb.o  work-obj93.cf

And for new version:

bash-4.4$ ls
adder.vhd  adder_tb.vhd
bash-4.4$ ghdl -a -Wc,-fprofile-arcs -Wc,-ftest-coverage adder.vhd
bash-4.4$ ls
adder.gcno  adder.o  adder.vhd  adder_tb.vhd  work-obj93.cf
bash-4.4$ ghdl -a adder_tb.vhd
bash-4.4$ ghdl -e -Wl,-lgcov adder_tb
bash-4.4$ ls
adder.gcno  adder.o  adder.vhd  adder_tb  adder_tb.o  adder_tb.vhd  e~adder_tb.o  work-obj93.cf
bash-4.4$ ghdl -r adder_tb
adder_tb.vhd:53:7:@8ns:(assertion note): end of test
bash-4.4$ ls
adder.gcno  adder.o  adder.vhd  adder_tb  adder_tb.o  adder_tb.vhd  e~adder_tb.o  work-obj93.cf
umarcor commented 2 years ago

I'm pinging @jocorso here, since he has been recently dealing with coverage and latest versions of GCC. In fact, he had to remove some gcda files for genhtml to work (see ghdl/ghdl#1815). @jocorso, any guess about what might be going on?

abyszuk commented 2 years ago

Anyway, at this point this looks more like bug in GHDL than the Docker image itself. Should we move this issue to GHDL repo?

jocorso commented 2 years ago

Hello @abyszuk I think you need more invoking arguments when you call ghdl in order to generate gcda and gcno files. If you can check my makefile I have the same flags when we invoke -a argument. Nevertheless I use -Wl,-lgcov and -Wl,--coverage when i invoke -e argument.

When I use these arguments the process generates the following files on workspace folder:

ls workspace/
e~full_adder_tb.o  full_adder.gcda  full_adder.gcno  full_adder_gcov.info  full_adder.o  full_adder_tb  full_adder_tb.gcda  full_adder_tb.gcno  full_adder_tb.o  work-obj08.cf

If I remove these arguments, the process fails and some files are missing on workspace folder:

ls workspace/
e~full_adder_tb.lst  e~full_adder_tb.o  full_adder.gcno  full_adder.o  full_adder_tb.gcno  full_adder_tb.o  work-obj08.cf

And I kindly recommend you to use -i and -m instead of -a and -e

abyszuk commented 2 years ago

Unfortunately, I can't repeat these results when using Docker images from this repo. @jocorso did you do your tests with Docker images or with your own GHDL build? Moreover, I can't use -i and -m switches because Vunit framework calls GHDL with -a and -e steps. I can imagine changing this in Vunit upstream wouldn't be trivial (and Vunit developers can always just say it's GHDL regression bug, nothing to be done on their end).

abyszuk commented 2 years ago

I'm a bit at loss about how to debug this further. I've collected strace output from two different versions, maybe that will be helpful. Please let me know how else I can help debugging this.

Older, custom and working image:

bash-4.4$ strace ./adder_tb                                                                                                                                                                                                                                                              
execve("./adder_tb", ["./adder_tb"], 0x7ffed7bce0c0 /* 10 vars */) = 0                                                                                                                                                                                                                   
brk(NULL)                               = 0x10f2000                                                                                                                                                                                                                                      
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd66a52e80) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7dd1e19000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=19135, ...}) = 0
mmap(NULL, 19135, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7dd1e14000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\20\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=28816, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7dd19eb000
mprotect(0x7f7dd19ee000, 2093056, PROT_NONE) = 0
mmap(0x7f7dd1bed000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7dd1bed000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \305\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2191840, ...}) = 0
mmap(NULL, 3674432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7dd1669000
mprotect(0x7f7dd17ea000, 2093056, PROT_NONE) = 0
mmap(0x7f7dd19e9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x180000) = 0x7f7dd19e9000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@'\0\0\0\0\0\0"..., 832) = 832
lseek(3, 88664, SEEK_SET)               = 88664
read(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\1\0\0\300\4\0\0\0\30\0\0\0\0\0\0\0"..., 48) = 48
fstat(3, {st_mode=S_IFREG|0755, st_size=95416, ...}) = 0
lseek(3, 88664, SEEK_SET)               = 88664
read(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\1\0\0\300\4\0\0\0\30\0\0\0\0\0\0\0"..., 48) = 48
mmap(NULL, 2187272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7dd1452000
mprotect(0x7f7dd1468000, 2093056, PROT_NONE) = 0
mmap(0x7f7dd1667000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f7dd1667000
mmap(0x7f7dd1668000, 8, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7dd1668000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2405\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=3167976, ...}) = 0
lseek(3, 808, SEEK_SET)                 = 808
read(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 3950400, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7dd108d000
mprotect(0x7f7dd1249000, 2093056, PROT_NONE) = 0
mmap(0x7f7dd1448000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bb000) = 0x7f7dd1448000
mmap(0x7f7dd144e000, 14144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7dd144e000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7dd1e12000
arch_prctl(ARCH_SET_FS, 0x7f7dd1e12b80) = 0
mprotect(0x7f7dd1448000, 16384, PROT_READ) = 0
mprotect(0x7f7dd1667000, 4096, PROT_READ) = 0
mprotect(0x7f7dd19e9000, 4096, PROT_READ) = 0
mprotect(0x7f7dd1bed000, 4096, PROT_READ) = 0
mprotect(0x6c6000, 4096, PROT_READ)     = 0
mprotect(0x7f7dd1e1b000, 4096, PROT_READ) = 0
munmap(0x7f7dd1e14000, 19135)           = 0
brk(NULL)                               = 0x10f2000
brk(0x1113000)                          = 0x1113000
brk(NULL)                               = 0x1113000
rt_sigaction(SIGSEGV, {sa_handler=0x421b76, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f7dd10c4400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0                                                                                                 
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7dd10c4400}, NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x421b76, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f7dd10c4400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7dd10c4400}, 8) = 0                                                           
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "adder_tb.vhd:53:7:@8ns:(assertio"..., 53adder_tb.vhd:53:7:@8ns:(assertion note): end of test
) = 53
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7dd10c4400}, NULL, 8) = 0
getpid()                                = 39
openat(AT_FDCWD, "/work/temp/ghdl-coverage/projects/adder/adder.gcda", O_RDWR|O_CREAT, 0666) = 3
fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
read(3, "adcg*58A\343\235\253\343\0\0\0\2439\0\0\0\341\376\213\215#\0\0\0\1\0\0\0"..., 4096) = 728
read(3, "", 3368)                       = 0
lseek(3, 0, SEEK_SET)                   = 0
lseek(3, 12, SEEK_SET)                  = 12
write(3, "\0\0\0\2439\0\0\0\341\376\213\215#\0\0\0\2\0\0\0\222\0\0\0\0\0\0\0\10\0\0\0"..., 716) = 716
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

New image from this repo:

I have no name!@pcte247806:/work/temp/ghdl-coverage/projects/adder$ strace ./adder_tb
execve("./adder_tb", ["./adder_tb"], 0x7ffdf14d27f0 /* 8 vars */) = 0
brk(NULL)                               = 0x5611535df000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=11641, ...}) = 0
mmap(NULL, 11641, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7d975fa000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18688, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d975f8000
mmap(NULL, 20752, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d975f2000
mmap(0x7f7d975f3000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7d975f3000
mmap(0x7f7d975f5000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f7d975f5000
mmap(0x7f7d975f6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f7d975f6000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0203\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=113088, ...}) = 0
mmap(NULL, 115088, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d975d5000
mprotect(0x7f7d975d8000, 98304, PROT_NONE) = 0
mmap(0x7f7d975d8000, 69632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f7d975d8000
mmap(0x7f7d975e9000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7f7d975e9000
mmap(0x7f7d975f0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f7d975f0000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@n\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1839792, ...}) = 0
mmap(NULL, 1852680, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d97410000
mprotect(0x7f7d97435000, 1662976, PROT_NONE) = 0
mmap(0x7f7d97435000, 1355776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f7d97435000
mmap(0x7f7d97580000, 303104, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x170000) = 0x7f7d97580000
mmap(0x7f7d975cb000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7f7d975cb000
mmap(0x7f7d975d1000, 13576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7d975d1000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d9740d000
arch_prctl(ARCH_SET_FS, 0x7f7d9740d740) = 0
mprotect(0x7f7d975cb000, 12288, PROT_READ) = 0
mprotect(0x7f7d975f0000, 4096, PROT_READ) = 0
mprotect(0x7f7d975f6000, 4096, PROT_READ) = 0
mprotect(0x5611516ea000, 8192, PROT_READ) = 0
mprotect(0x7f7d97627000, 4096, PROT_READ) = 0
munmap(0x7f7d975fa000, 11641)           = 0
brk(NULL)                               = 0x5611535df000
brk(0x561153600000)                     = 0x561153600000
rt_sigaction(SIGSEGV, {sa_handler=0x561151633932, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f7d9744bd60}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d9744bd60}, NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x561151633932, sa_mask=[], sa_flags=SA_RESTORER|SA_RESETHAND|SA_SIGINFO, sa_restorer=0x7f7d9744bd60}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d9744bd60}, 8) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "adder_tb.vhd:53:7:@8ns:(assertio"..., 53adder_tb.vhd:53:7:@8ns:(assertion note): end of test
) = 53
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7d9744bd60}, NULL, 8) = 0
exit_group(0)                           = ?
+++ exited with 0 +++
jocorso commented 2 years ago

Hi @abyszuk

Unfortunately, I can't repeat these results when using Docker images from this repo. @jocorso did you do your tests with Docker images or with your own GHDL build?

This gitlab project calls a custom docker based on arch-linux with ghdl nightly generated in august. Maybe you can clone my project and try to execute makefiles on your docker. Or you can tell me what docker are you using, and i can try to branch my project with this docker.

Moreover, I can't use -i and -m switches because Vunit framework calls GHDL with -a and -e steps. I can imagine changing this in Vunit upstream wouldn't be trivial (and Vunit developers can always just say it's GHDL regression bug, nothing to be done on their end).

I've never used Vunit, maybe in a near future, but I understand your situation.

abyszuk commented 2 years ago

Hi @jocorso , thanks for fast reply

This gitlab project calls a custom docker based on arch-linux with ghdl nightly generated in august. Maybe you can clone my project and try to execute makefiles on your docker. Or you can tell me what docker are you using, and i can try to branch my project with this docker.

I use GHDL images from this repo, more specifically ghdl/vunit:gcc

jocorso commented 2 years ago

Hi @abyszuk I created a branch in my gitlab repository that calls ghdl/vunit:gcc docker.

@umarcor It looks like there is something wrong in that docker...

tgingold commented 2 years ago

I think this is due to different gcc version.

Which gcc version (try gcc -v) are you using on your system ? It seems libgcov should match the gcc compiler used.

With this docker image, ghdl uses gcc 9.1 backend to generate object code, but uses the system gcc for the final link. And libgcov comes from the final link.

If you can install and use gcc 9 for the link, I think it should work. Maybe that gcc should be in the docker image ?

I was not able to reproduce this issue using gcc 9 to link the executable.

abyszuk commented 2 years ago

Hi @tgingold, I did the tests you've asked for. Things still don't work, but there's progress. Default GCC version in Debian Bullseye is GCC 10.2.1. I've installed GCC 9 in the container with apt-get install gcc-9 and used update-alternatives to switch to it globally. After installing GCC-9 in this image we have 3 versions available:

I have no name!@pcte247806:/work/temp/ghdl-coverage/projects/adder$ find /usr -name gcc*
(... cut ...)
/usr/bin/gcc-9
/usr/bin/gcc-10
/usr/bin/gcc  # (symlink to /usr/bin/gcc-9)
/usr/local/bin/gcc

However, there's still a minor version mismatch. GHDL backend is GCC 9.1, but the version from Debian repo is 9.3. This results in following error when running simulation:

I have no name!@pcte247806:/work/temp/ghdl-coverage/projects/adder$ ghdl -r adder_tb
libgcov profiling error:/work/temp/ghdl-coverage/projects/adder/adder.gcda:Version mismatch - expected 9.3 (release) (A93*) got 9.1 (release) (A91*)
adder_tb.vhd:53:7:@8ns:(assertion note): end of test

So it really does suggest that GCC versions should match along the full chain (or that GHDL should use only the backend GCC version for everything, if that's possible).

jocorso commented 2 years ago

I pulled ghdl/vunit:gcc and I runned it:

This is the installed ghdl version:

# ghdl --version
GHDL 2.0.0-dev (v1.0.0-749-g324403d7) [Dunoon edition]
 Compiled with GNAT Version: 9.3.0
 GCC back-end code generator
Written by Tristan Gingold.

Copyright (C) 2003 - 2021 Tristan Gingold.
GHDL is free software, covered by the GNU General Public License.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#

This is the installed gcc version:

# gcc --version
gcc (GCC) 9.1.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#

The libgcov path installed in that docker are located at:

# find / -iname libgcov*
/usr/lib/gcc/x86_64-linux-gnu/10/libgcov.a
/usr/local/lib/gcc/x86_64-pc-linux-gnu/9.1.0/libgcov.a
#

So if you add the following argument when you invoke -e it will works:

-Wl,-L/usr/local/lib/gcc/x86_64-pc-linux-gnu/9.1.0 (you can see an example in my makefile)

Now my gitlab branch with ghdl/vunit:gcc docker works and reports code coverage results

tgingold commented 2 years ago

Maybe new docker images should be built with different gcc backends. I suppose PR are welcome!

tgingold commented 2 years ago

This is weird. If your default gcc version is 9.1.0, why it wouldn't link with libgcov from 9.1.0 ?

abyszuk commented 2 years ago

Maybe new docker images should be built with different gcc backends. I suppose PR are welcome!

I can try to do that. Which GCC version can be considered "safe" to use? 10? 11? @umarcor is it enough to change version string in .github/workflows/bullseye.yml or does this have to accompanied by some PR to GHDL repo too?

This is weird. If your default gcc version is 9.1.0, why it wouldn't link with libgcov from 9.1.0 ?

I've noticed that too. There's some strange mixup with what GCC version is called where. It looks like GCC 9.1 (/usr/local/bin/gcc) is the one that's called by default, as reported by which gcc. But the libgcov version error message suggests that ghdl -e -Wl,-lgcov adder_tb calls GCC 9.3 (/usr/bin/gcc)? Or could this be an issue with GCC itself?

jocorso commented 2 years ago

@abyszuk have you tried my workaround?.

I have the same problem with arch-linux. As a rolling-release, it uses a latest version of gcc that ghdl is developed, so you should use this flag and update from time to time, if ghdl is not developed with the last version of gcc.

abyszuk commented 2 years ago

@abyszuk have you tried my workaround?.

Yes, and it works. Thank you for finding this! Unfortunately, I don't think I can use this workaround in our team repo, because it would mean I'd have to modify all VUnit run.py scripts. I'd prefer to wait until we properly fix this Docker image.

tgingold commented 2 years ago

I can try to do that. Which GCC version can be considered "safe" to use? 10? 11?

Both gcc 10 and 11 are supported.

I've noticed that too. There's some strange mixup with what GCC version is called where. It looks like GCC 9.1 (/usr/local/bin/gcc) is the one that's called by default, as reported by which gcc. But the libgcov version error message suggests that ghdl -e -Wl,-lgcov adder_tb calls GCC 9.3 (/usr/bin/gcc)? Or could this be an issue with GCC itself?

Can you try with ghdl -e -v ... to display the command executed ?

abyszuk commented 2 years ago 
@pcte247806$ ghdl -e -v -Wl,-lgcov adder_tb
/usr/local/libexec/gcc/x86_64-pc-linux-gnu/9.1.0/ghdl1 -P/usr/local/lib/ghdl/ieee/v93/ -P/usr/local/lib/ghdl/ --elab adder_tb -l e~adder_tb.lst -quiet -o e~adder_tb.s e~adder_tb
/usr/bin/as -o e~adder_tb.o e~adder_tb.s
/usr/bin/cc -o adder_tb e~adder_tb.o /usr/local/lib/ghdl/std/v93/std_standard.o adder_tb.o adder.o -lgcov /usr/local/lib/ghdl/libgrt.a -ldl -lm /usr/local/lib/ghdl/libbacktrace.a -L./ -lz -ldl -Wl,--version-script=/usr/local/lib/ghdl/grt.ver -Wl,--export-dynamic

It calls cc binary, which isn't provided by GCC 9.1.0 install in /usr/local/bin. Going through the symlink chain I can see it points to /usr/bin/gcc (which is currently 9.3 from repo, 10.2 by default) After changing the symlink with update-alternatives --install /usr/bin/cc cc /usr/local/bin/gcc 9 it started working!

EDIT: Looking at the ghdldrv.adb source inspired me to do one more test and instruct elaborate step to use gcc as a linker by adding --LINK=gcc option. Now GHDL calls (IMO) "proper" binary and simulation generates *.gcda file too!

@pcte247806$ ghdl -e -v --LINK=gcc -Wl,-lgcov adder_tb
/usr/local/libexec/gcc/x86_64-pc-linux-gnu/9.1.0/ghdl1 -P/usr/local/lib/ghdl/ieee/v93/ -P/usr/local/lib/ghdl/ --elab adder_tb -l e~adder_tb.lst -quiet -o e~adder_tb.s e~adder_tb
/usr/bin/as -o e~adder_tb.o e~adder_tb.s
/usr/local/bin/gcc -o adder_tb e~adder_tb.o /usr/local/lib/ghdl/std/v93/std_standard.o adder_tb.o adder.o -lgcov /usr/local/lib/ghdl/libgrt.a -ldl -lm /usr/local/lib/ghdl/libbacktrace.a -L./ -lz -ldl -Wl,--version-script=/usr/local/lib/ghdl/grt.ver -Wl,--export-dynamic

I've tested this also with freshly started Docker image, without additional GCC versions or update-alternatives changes.

Is this an issue with GHDL (maybe using code coverage should also imply --LINK=gcc), or should the Docker image also have updated symlinks?

abyszuk commented 2 years ago

@umarcor @tgingold sorry for bothering you, but can you tell me how do you want this fixed? I can prepare the PR, but I think you're the only ones who can decide what's the best fix here.

Again, I'm sorry for bothering you, but our firmware CI pipelines are broken for a month now, so this is becoming a bit serious :-)

tgingold commented 2 years ago

I would do a new docker image with ghdl using the same gcc version as what you need.

But umarcor is the maintainer!

abyszuk commented 2 years ago

PR ready. @umarcor can you look if this is acceptable for you? I have a feeling that this problem isn't specific only to Debian Bullseye, but I wanted to make the least invasive change that fixes Vunit images

umarcor commented 2 years ago

@umarcor is it enough to change version string in .github/workflows/bullseye.yml or does this have to accompanied by some PR to GHDL repo too?

That is correct. The version of GCC used as a backend of GHDL is defined by the backend argument in the CI workflow. Changing that YAML file is enough.

Until now, we were not taking care of GHDL using exactly the same version as the system GCC. On the one hand, we are trying to use different versions of GCC just to increase the chances of finding regressions. On the other hand, GHDL should not depend on the GCC version available on the system. However, due to the issues found in here, we might consider being more careful about that, and using the same version for GHDL as the one on the system. @tgingold, what do you think? Is that necessary or should it be enough with setting CC in the containers?

I have a feeling that this problem isn't specific only to Debian Bullseye, but I wanted to make the least invasive change that fixes Vunit images

I believe it applies to all distros. However, I don't know if this is exclusive to GCC or it might be problematic with LLVM backends as well. I guess it's not, because when using the LLVM backend we don't need the sources of clang/gcc, we use the system version only.