search

linux-test-project / lcov

LCOV
GNU General Public License v2.0
867 stars 235 forks source link

[macOS][flutter test]branch coverage missing since 2.0.0 later #221

Closed LongXiangGuo closed 1 year ago

LongXiangGuo commented 1 year ago

The genhtml could not generate branch coverage since 2.0.0 later. 

  lines......: 98.9% (18843 of 19054 lines)
  functions..: no data found
  branches...: no data found

Test with 1.1.6 it works well

Writing directory view page.
Overall coverage rate:
  lines......: 98.9% (18843 of 19054 lines)
  functions..: no data found
  branches...: 98.4% (2660 of 2703 branches)

There are both use the same test data. lcov.info.zip

ENV macOS: 13.3.1 (a) Flutter: 3.7.12

henry2cox commented 1 year ago

How was that .info file generated? Its syntax is not quite right - so it appears not to have been generated by lcov/geninfo.

I haven't yet checked why lcov 2.0 is not parsing it - but I suspect that the issue would go away if the file was correct. It might also be possible to make lcov/genhtml handle this syntax - but I would like to understand why it exists, first. Thanks Henry

henry2cox commented 1 year ago

OK...found a bug. Fixed in b73440cb70.

Note that your branch coverage data is rather odd. For example, the branches in 'lib/src/images/branded_asset_image.dart' look like:

BRDA:9,0,0,1 BRDA:15,0,0,1 BRDA:17,0,0,1 BRDA:19,0,0,1 BRDA:21,0,0,1

..so there are 5 lines which contain branches, but each branch has only one block. Normally, we expect branches to have an even number of blocks: taken/not taken (or into/over).

These would be filtered out by the exception branch filter - as they look like orphan branches that get left behind after marked exception branches are removed.

The bug was affecting the summary count printed by lcov and genhtml - but was not affecting the generated info or HTML. Annoying - but not critical.

Henry

LongXiangGuo commented 1 year ago

OK...found a bug. Fixed in b73440c.

Note that your branch coverage data is rather odd. For example, the branches in 'lib/src/images/branded_asset_image.dart' look like:

BRDA:9,0,0,1 BRDA:15,0,0,1 BRDA:17,0,0,1 BRDA:19,0,0,1 BRDA:21,0,0,1

..so there are 5 lines which contain branches, but each branch has only one block. Normally, we expect branches to have an even number of blocks: taken/not taken (or into/over).

These would be filtered out by the exception branch filter - as they look like orphan branches that get left behind after marked exception branches are removed.

The bug was affecting the summary count printed by lcov and genhtml - but was not affecting the generated info or HTML. Annoying - but not critical.

Henry

Very thanks! The genhtml command now works well now, Can you also fix the lcov --summary comand for it? 

 lcov --summary coverage/lcov.info  
Summary coverage rate:
  lines......: 98.9% (18843 of 19054 lines)
  functions..: no data found
  branches...: no data found
henry2cox commented 1 year ago

Can you also fix the lcov --summary comand for it?

Branch metrics are not enabled by default - unless you change the setting in your lcovrc config file(s). Thus, you saw:

$ ./bin/lcov --summary ~/lcov.info <- branches not enabled Summary coverage rate: lines......: 98.9% (18843 of 19054 lines) functions..: no data found branches...: no data found <- so no branches reported

Whereas what you needed was:

$ ./bin/lcov --summary ~/lcov.info --branch <- turn on branch reporting Summary coverage rate: lines......: 98.9% (18843 of 19054 lines) functions..: no data found functions..: no data found branches...: 98.4% (2660 of 2703 branches)

It might have been better if the tool omitted the title if some category was disabled - rather than reporting no data. Not sure if that is worthwhile to change. No idea if somebody might have a script somewhere which depends on legacy behaviour.

I think that this one is fixes - so the issue can be closed.

Henry

henry2cox commented 1 year ago

It might have been better if the tool omitted the title if some category was disabled - rather than reporting no data.

Above change is in SHA 33e06215f661 genhtml had already implemented similar behaviour.

LongXiangGuo commented 1 year ago

Thanks, I have verified it, both lcov and genhtml are works well now. I have overlooked the latest help command document --(no-)branch-coverage Enable (disable) branch coverage collection.

lcov --branch-coverage --summary coverage/lcov.info
Summary coverage rate:
  lines......: 98.9% (18843 of 19054 lines)
  functions..: no data found
  branches...: 98.4% (2660 of 2703 branches)
LongXiangGuo commented 1 year ago

Hi @henry2cox Could you help publish the new version to apt-get, CICD are required the new fix.

henry2cox commented 1 year ago

Could you explain that? The bug was just in the value printed in the summary - not in the data, and not in the numbers shown by genhtml. I understand that fewer bugs are better than more bugs, but I do not think that this one rises to the level of "serious" - leave alone "critical". But there may be something about the bug or about your environment that I do not know/do not understand. Thanks Henry

LongXiangGuo commented 1 year ago

We use the apt-get to got the latest lcov and build a new docker to check branch coverage threshold like this.

RUN apt-get update --allow-unauthenticated && \
    apt-get --allow-unauthenticated --no-install-recommends install -f -y software-properties-common \
    lcov
   ...

If the latest fix not publish will not be able to got the correctly, because the current release version had now branch coverage data

lcov --branch-coverage --summary coverage/lcov.info
Summary coverage rate:
  lines......: 98.9% (18843 of 19054 lines)
  functions..: no data found
  branches...: no data found
henry2cox commented 1 year ago

If the latest fix not publish will not be able to got the correctly, because the current release version had now branch coverage data

No.
You have misunderstood. The lcov 2.0 version tells you that it extracted no branch data - but in fact the data is there in the output file, and will be displayed if you generate HTML.

If you really need the patch, then you can get that by changing your docker install to clone from github and 'make install' that.

I still think that this bug is annoying - but not serious.

Henry

LongXiangGuo commented 1 year ago

If the latest fix not publish will not be able to got the correctly, because the current release version had now branch coverage data

No. You have misunderstood. The lcov 2.0 version tells you that it extracted no branch data - but in fact the data is there in the output file, and will be displayed if you generate HTML.

If you really need the patch, then you can get that by changing your docker install to clone from github and 'make install' that.

I still think that this bug is annoying - but not serious.

Henry

Thanks for your solution, I have tried it before, I have tried it before the ubuntu:16.04 works well. but our CICID still failed for the Capture::Tiny module, I'will contact with our CICD team with this issue.

Can't locate Capture/Tiny.pm in @INC (you may need to install the Capture::Tiny module) (@INC contains: /app/lcov/bin/../lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.34.0 /usr/local/share/perl/5.34.0 /usr/lib/x86_64-linux-gnu/perl5/5.34 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.34 /usr/share/perl/5.34 /usr/local/lib/site_perl) at /app/lcov/bin/../lib/lcovutil.pm line 14.
BEGIN failed--compilation aborted at /app/lcov/bin/../lib/lcovutil.pm line 14.
Compilation failed in require at lcov/bin/lcov line 103.
BEGIN failed--compilation aborted at lcov/bin/lcov line 103.

RUN apt-get update ; apt-get install -y git\ make\ libdatetime-perl

lcov depes

RUN make git clone -b master --single-branch --depth=1 https://github.com/dagolden/Capture-Tiny.git RUN cd Capture-Tiny && perl Makefile.PL && make && make install RUN cd ..

RUN echo "Capture-Tiny installed"

work around for the latest lcov not published ->https://github.com/linux-test-project/lcov/issues/221

RUN git clone -b master --single-branch --depth=1 https://github.com/linux-test-project/lcov.git lcov ENV PATH=$PATH:/lcov/bin RUN echo lcov --version

RUN echo "success"·