Use tarpaulin for code coverage

[fulmicoton]

https://github.com/xd009642/tarpaulin

[lnicola]

I'm not sure tarpaulin works with multi-threaded programs yet: https://github.com/xd009642/tarpaulin/issues/35. I wasn't able to test it myself as it doesn't run on my distro.

[xd009642]

Just because I was linked to this on reddit @lnicola if you're distro is a Linux distro and not BSD based if you open an issue on the tarpaulin repo I'll look into it. Also https://github.com/xd009642/tarpaulin/issues/35 only affects the accuracy of hit count (which kcov doesn't offer) so for consistent results use the --no-count option :smile:

[lnicola]

@xd009642 oh, I see your point about https://github.com/xd009642/tarpaulin/issues/35. The build issue is https://github.com/xd009642/tarpaulin/issues/23.

[xd009642]

So Arch works and there are tarpaulin users on Arch. The issue is with the order of linker options in projects with certain dependencies serde_derive is a notable one. It's a bit more of a rustc option in my mind as it's caused by patched versions of ld on some distros so is unfortunately not something I've found an easy fix for tarpaulin side. So TL;DR tarpaulin can work on Arch but depends on dependencies unfortunately.

[lnicola]

Right. tantivy actually uses serde_derive, which is how I ended up on https://github.com/xd009642/tarpaulin/issues/23. I haven't tried it lately, but I assumed it's still not working. That's unfortunate, indeed.

[xd009642]

Ah that is unfortunate. Using the gold linker might have a different result but that's just a shot in the dark.

If you don't mind trying it out and posting if it makes things better/worse/no-different in the issue that would be greatly appreciated. If not I understand and I will look into tackling one of the fixes to the issue internal to rustc when I've got time

[lnicola]

An idea worth trying, but alas, using gold makes no difference. On the other hand...

value.push_str("-C link-arg=-no-pie -C link-arg=-shared ");

$ ~/tarpaulin/target/release/cargo-tarpaulin tarpaulin -v --skip-clean
Running Tarpaulin
   Compiling quote v0.3.15
   Compiling bitflags v0.4.0
   Compiling stable_deref_trait v1.0.0
   Compiling unicode-xid v0.0.4
   Compiling rustc-demangle v0.1.5
   Compiling version v2.0.1
   Compiling void v1.0.2
   Compiling either v1.4.0
   Compiling byteorder v0.5.3
   Compiling utf8-ranges v0.1.3
   Compiling byteorder v1.1.0
   Compiling log v0.3.8
   Compiling utf8-ranges v1.0.0
   Compiling regex-syntax v0.4.1
   Compiling rust-stemmers v0.1.0
   Compiling regex-syntax v0.3.9
   Compiling num-traits v0.1.40
   Compiling serde v1.0.20
   Compiling crossbeam v0.3.0
   Compiling winapi v0.2.8
   Compiling bit-vec v0.4.4
   Compiling libc v0.2.33
   Compiling cfg-if v0.1.2
   Compiling itoa v0.3.4
   Compiling futures v0.1.17
   Compiling maplit v0.1.6
   Compiling lazy_static v0.1.16
   Compiling lazy_static v0.2.10
   Compiling winapi-build v0.1.1
   Compiling dtoa v0.4.2
   Compiling cc v1.0.3
   Compiling bitflags v0.5.0
   Compiling getopts v0.2.15
   Compiling semver v0.1.20
   Compiling gcc v0.3.54
   Compiling ascii v0.7.1
   Compiling synom v0.11.3
   Compiling owning_ref v0.3.3
   Compiling unreachable v1.0.0
   Compiling itertools v0.5.10
   Compiling bit-set v0.4.0
   Compiling time v0.1.38
   Compiling rand v0.3.18
   Compiling num_cpus v1.7.0
   Compiling memchr v1.0.2
   Compiling tinysegmenter v0.1.0
   Compiling kernel32-sys v0.2.2
   Compiling pulldown-cmark v0.0.8
   Compiling backtrace-sys v0.1.16
   Compiling tantivy v0.5.0-dev (file:///~/tantivy)
   Compiling combine v2.5.2
   Compiling rustc_version v0.1.7
   Compiling syn v0.11.11
   Compiling thread_local v0.3.4
   Compiling serde_json v1.0.6
   Compiling bincode v0.8.0
   Compiling lz4-sys v1.8.0
   Compiling futures-cpupool v0.1.7
   Compiling aho-corasick v0.6.3
   Compiling uuid v0.5.1
   Compiling tempdir v0.3.5
error: failed to run custom build command for `kernel32-sys v0.2.2`
process didn't exit successfully: `~/tantivy/target/debug/build/kernel32-sys-9e6da616b79a58fb/build-script-build` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
Error: failed to compile: build failed
Error during run
$ ~/tarpaulin/target/release/cargo-tarpaulin tarpaulin -v --skip-clean
Running Tarpaulin
   Compiling nix v0.7.0
   Compiling chan v0.1.19
   Compiling tempfile v2.2.0
   Compiling kernel32-sys v0.2.2
   Compiling lz4-sys v1.8.0
   Compiling backtrace-sys v0.1.16
   Compiling serde_derive_internals v0.17.0
   Compiling skeptic v0.9.0
   Compiling regex v0.2.2
   Compiling fs2 v0.2.5
   Compiling backtrace v0.3.4
error: failed to run custom build command for `nix v0.7.0`
process didn't exit successfully: `~/tantivy/target/debug/build/nix-78fb40fa3c7fe661/build-script-build` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
Error: failed to compile: build failed
Error during run
$ ~/tarpaulin/target/release/cargo-tarpaulin tarpaulin -v --skip-clean
Running Tarpaulin
   Compiling serde_derive v1.0.20
   Compiling nix v0.7.0
   Compiling env_logger v0.4.3
   Compiling lz4 v1.22.0
   Compiling error-chain v0.8.1
error: failed to run custom build command for `lz4 v1.22.0`
process didn't exit successfully: `~/tantivy/target/debug/build/lz4-45ede134a9e593fe/build-script-build` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
$ ~/tarpaulin/target/release/cargo-tarpaulin tarpaulin -v --skip-clean
Running Tarpaulin
   Compiling atomicwrites v0.1.3
   Compiling memmap v0.4.0
   Compiling lz4 v1.22.0
   Compiling fst v0.1.38
   Compiling tantivy v0.5.0-dev (file:///~/tantivy)
    Finished dev [unoptimized + debuginfo] target(s) in 103.54 secs
Processing tantivy
Launching test
running ~/tantivy/target/debug/deps/tantivy-eaddc236195dfb6a
ERROR: Tarpaulin cannot find code addresses check that pie is disabled for your linker. If linking with gcc try adding -C link-args=-no-pie to your rust flags

[xd009642]

I don't suppose you mind copying this comment over to the issue on tarpaulin. Just to keep things in one place. And it may kickstart some work on that issue again from those of us involved. Plus I don't want to hijack your issue with my stuff :blush:, although this is kind of a tarpaulin issue so I don't feel too bad..

[fulmicoton]

That did not work too well.

$ travis-cargo doc-upload
$ bash ./script/tarpaulin.sh
download tarpaulin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   149  100   149    0     0   1285      0 --:--:-- --:--:-- --:--:--  1295
cargo-tarpaulin
exec tarpaulin
Building project
Launching test
running /home/travis/build/tantivy-search/tantivy/target/debug/deps/tantivy-e65a60e7fd7e9ade
running 161 tests
..Code is mulithreaded, disabling hit count
.............................................................
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

[xd009642]

I'll have a look at this, the solution might be to use the argument --no-count multithreading is a bit of a pain point because of how ptrace works. But I'll get back to you :smile:

[xd009642]

OH just tried to run tarpaulin on tantivy, unfortunately a regression was added into rustc which means tarpaulin doesn't work properly on beta or nightly https://github.com/rust-lang/rust/issues/46467 there is a PR to address this, that will hopefully be merged in soon once some review comments have been addressed

[xd009642]

So said issue is now closed, looking at your tarpaulin script I'd recommend this instead, there's no reason to do additional runs to get extra output formats it accepts multiple outputs:

echo "download tarpaulin";
bash <(curl https://raw.githubusercontent.com/xd009642/tarpaulin/master/travis-install.sh);
echo "exec tarpaulin";
cargo tarpaulin --ciserver travis-ci --coveralls $TRAVIS_JOB_ID --out Xml --no-count;
echo "upload tarpaulin";
bash <(curl -s https://codecov.io/bash);

EDIT: Tarpaulin is only working on nightly as the PR fix didn't make the deadline to get into stable or beta, it will be in 1.24 but until then you'll have to run tarpaulin in nightly.

[PSeitz]

We have coverage with cargo-llvm-cov now