kcov tweaking

[pzmarzly]

After #207, I tried tweaking kcov settings further on, to understand how they work. I was running kcov on my local machine, using perf stat ./kcov-script.sh.

--{include,exclude}-{path,pattern}= options are used for filtering source files. Per kcov man page, they take comma-separated list of patterns. When no such options are present, all files are used (that includes dependencies downloaded in $HOME/.cargo). Include and exclude are filtering list of files kcov finds in DWARF informations. They kinda work like this (Rust pseudocode):

fn include(files: Vec, text: String) -> Vec {
    files.iter().filter(|x| x.contains(text)).collect()
}
fn exclude(files: Vec, text: String) -> Vec {
    files.iter().filter(|x| !x.contains(text)).collect()
}

{include,exclude}_path are much slower than {include,exclude}_pattern. --include_path=components adds 60 seconds to kcov process on my machine, and it would probably be more on Travis.

Since we use include_pattern=components, only files which path contains components are picked. If any dependency of offst ever happens to have components directory, its test coverage would influence ours. That's why --exclude-path=/usr,~/.cargo may be a good idea. --exclude-pattern=/usr/,/.cargo/ may be an even better one (see paragraph above), as long as we don't create folder called "usr" in offst codebase.

--verify flag makes kcov 3-4 times slower, but it was needed to prevent it from segfaulting when working with Rust binaries. I tried removing it and nothing broke. I'm using the latest version of kcov (CI is not). This may be worth investigating.

kcov runs single-threaded. I tried using GNU parallel like this:

find \
    target/${TARGET}/debug \
    -maxdepth 1 -executable -type f \
    | parallel \
    -j0 \
    'kcov ${FLAGS} target/kcov {}'

-j0 means "spawn as many threads as there are CPUs". This was not a good idea - kcov crashes a lot when multiple processes try to access target/kcov at once.

So, given these pieces of information, I would suggest trying:

for exe in ${exes}; do
    ${HOME}/install/kcov-${TARGET}/bin/kcov \
        --exclude-pattern=/usr/,/.cargo/ \
        --include-pattern=/components/ \
        target/kcov \
        ${exe}
done

Or maybe --exclude-pattern=/.cargo/ in case we ever create usr directory. Or maybe just --exclude-path=/usr --exclude-pattern=/.cargo/, it is costly, but it still should be better than what we have now, especially without --verify.

For now I'm just writing what I know, asking for opinion.

[pzmarzly]

Here is some raw perf data, though I stopped writing these down after a while

exclude-path=/usr/lib, include-pattern=components: 502 722 540 131 cycles:u # 2,628 GHz 938 866 219 880 instructions:u # 1,87 insn per cycle 196 347 602 771 branches:u # 1026,580 M/sec 944 884 723 branch-misses:u # 0,48% of all branches 192,142756433 seconds time elapsed 171,348535000 seconds user 19,665885000 seconds sys 508 614 330 007 cycles:u # 2,602 GHz 956 887 911 221 instructions:u # 1,88 insn per cycle 199 661 484 191 branches:u # 1021,454 M/sec 974 111 279 branch-misses:u # 0,49% of all branches 205,933535450 seconds time elapsed 174,700801000 seconds user 20,496185000 seconds sys exclude-pattern=.cargo,/usr/lib,/usr/include, include-pattern=components: 527 363 364 339 cycles:u # 2,665 GHz 1 020 114 023 227 instructions:u # 1,93 insn per cycle 219 241 979 840 branches:u # 1108,083 M/sec 959 441 722 branch-misses:u # 0,44% of all branches 198,525452254 seconds time elapsed 178,927531000 seconds user 18,662000000 seconds sys 521 149 352 588 cycles:u # 2,681 GHz 1 020 128 746 640 instructions:u # 1,96 insn per cycle 219 169 578 086 branches:u # 1127,297 M/sec 954 710 720 branch-misses:u # 0,44% of all branches 195,037822089 seconds time elapsed 175,606380000 seconds user 18,568160000 seconds sys 534 233 029 385 cycles:u # 2,623 GHz 1 039 055 998 492 instructions:u # 1,94 insn per cycle 222 766 246 980 branches:u # 1093,891 M/sec 1 005 201 666 branch-misses:u # 0,45% of all branches 214,076562770 seconds time elapsed 182,133361000 seconds user 21,183972000 seconds sys no exluding, include-pattern=components: 484 364 587 437 cycles:u # 2,631 GHz 896 546 598 743 instructions:u # 1,85 insn per cycle 184 306 478 649 branches:u # 1001,245 M/sec 935 019 075 branch-misses:u # 0,51% of all branches 184,719943556 seconds time elapsed 164,056328000 seconds user 19,781724000 seconds sys 484 506 001 646 cycles:u # 2,608 GHz 896 392 646 798 instructions:u # 1,85 insn per cycle 184 241 117 317 branches:u # 991,917 M/sec 935 686 658 branch-misses:u # 0,51% of all branches 186,310238671 seconds time elapsed 164,075560000 seconds user 21,448010000 seconds sys no exluding, include-pattern=components, no --verify: 125 316 008 058 cycles:u # 2,118 GHz 257 735 773 328 instructions:u # 2,06 insn per cycle 53 752 244 390 branches:u # 908,676 M/sec 243 768 284 branch-misses:u # 0,45% of all branches 60,037292691 seconds time elapsed 43,022770000 seconds user 16,022707000 seconds sys 137 964 867 712 cycles:u # 2,051 GHz 277 232 735 130 instructions:u # 2,01 insn per cycle 57 384 474 140 branches:u # 853,080 M/sec 282 029 913 branch-misses:u # 0,49% of all branches 77,665052384 seconds time elapsed 50,387391000 seconds user 16,689920000 seconds sys 125 757 150 324 cycles:u # 2,035 GHz 257 753 640 643 instructions:u # 2,05 insn per cycle 53 747 234 096 branches:u # 869,874 M/sec 246 906 156 branch-misses:u # 0,46% of all branches 62,577967780 seconds time elapsed 43,139634000 seconds user 18,549244000 seconds sys no exluding, include-path=components, no --verify: 169 819 872 622 cycles:u # 1,568 GHz 329 052 244 303 instructions:u # 1,94 insn per cycle 66 823 697 408 branches:u # 617,162 M/sec 349 166 910 branch-misses:u # 0,52% of all branches 128,141396496 seconds time elapsed 66,956083000 seconds user 41,113681000 seconds sys exlude-path=/usr/lib, include-pattern=components, no --verify: 181 960 949 127 cycles:u # 1,694 GHz 388 474 214 750 instructions:u # 2,13 insn per cycle 85 292 009 590 branches:u # 794,146 M/sec 319 028 745 branch-misses:u # 0,37% of all branches 107,968285312 seconds time elapsed 67,742567000 seconds user 39,503044000 seconds sys no excluding, no including - tested dependencies in .cargo: 844 235 831 447 cycles:u # 2,305 GHz 1 395 705 577 507 instructions:u # 1,65 insn per cycle 271 353 559 049 branches:u # 740,828 M/sec 1 663 067 006 branch-misses:u # 0,61% of all branches 367,885771958 seconds time elapsed 302,566904000 seconds user 63,210475000 seconds sys no excluding, include-path=components: 518 250 495 990 cycles:u # 2,291 GHz 946 301 339 101 instructions:u # 1,83 insn per cycle 193 243 362 302 branches:u # 854,231 M/sec 1 003 924 977 branch-misses:u # 0,52% of all branches 226,966495513 seconds time elapsed 181,909074000 seconds user 44,039785000 seconds sys 513 380 789 124 cycles:u # 2,274 GHz 942 930 097 539 instructions:u # 1,84 insn per cycle 192 717 280 378 branches:u # 853,730 M/sec 980 727 974 branch-misses:u # 0,51% of all branches 226,464357247 seconds time elapsed 179,707696000 seconds user 45,766969000 seconds sys