Build with clean incremental cache slower than without incremental compilation

[bjorn3]

$ rustc -V
rustc 1.36.0 (a53f9df32 2019-07-03)

$ git clone https://github.com/bjorn3/fractals
$ cd fractals
$ git checkout 0c4929c80c55cadf3a2576f288bfde463ffb8e18
$ cargo build # build deps

$ # build without incr comp
$ touch src/main; CARGO_INCREMENTAL=0 cargo build
   Compiling fractals v0.1.0 (/home/bjorn/Documenten/fractals)
    Finished dev [unoptimized + debuginfo] target(s) in 2.74s

$ # incremental cache was thrown away by the previous command; rebuild it
$ touch src/main.rs; CARGO_INCREMENTAL=1 cargo build
   Compiling fractals v0.1.0 (/home/bjorn/Documenten/fractals)
    Finished dev [unoptimized + debuginfo] target(s) in 13.47s

$ # incremental cache exists. yet the build is slower than without incr comp
$ touch src/main.rs; CARGO_INCREMENTAL=1 cargo build
   Compiling fractals v0.1.0 (/home/bjorn/Documenten/fractals)
    Finished dev [unoptimized + debuginfo] target(s) in 4.45s

[cyplo]

Can still reproduce on stable rustc 1.40.0 (73528e339 2019-12-16)