Generalized dot test sensitve to numerical precision

[martinholters]

The tests at https://github.com/JuliaLang/Compat.jl/blob/e667af7e6e31c090776249f9a1b5df2684a571d2/test/runtests.jl#L171-L173 sometimes fail for Float32 eltype with an error that can probably be attributed to numerical precision issues, e.g.

  Expression: dot(x, transpose(A), y) ≈ dot(x, transpose(A) * y) ≈ x' * transpose(A) * y ≈ (x' * transpose(A)) * y
   Evaluated: -0.0029978678f0 ≈ -0.0029971108f0 ≈ -0.0029982424f0 ≈ -0.0029982424f0

The same tests appear in the LinearAlgebra tests: https://github.com/JuliaLang/julia/blob/29826c2c0893488dce0d4ee92b457f7f1a586dfb/stdlib/LinearAlgebra/test/generic.jl#L472-L474 So Julia is probably effected as well.

The probability of failing seems to be quite low (around 0.02% in a brief experiment), but it would still be nice to sort this out. CC @dkarrasch who added these tests in julia and @mcabbott who brought them here.

[dkarrasch]

Perhaps we should set a random seed somewhere close to the tests, and change that in case RNGs start to produce bad numbers.

[mcabbott]

I thought testsets did set a consistent seed, although the result isn't fixed across versions.

For example, this passes for me every time on Julia 1.5.0-rc1, and fails every time on 1.4.2:

using Test
@testset "stuff" begin
    @show rand()
    @test rand()>0.5
end

[stevengj]

Any dot product test will be badly conditioned (very sensitive to roundoff error and hence likely to fail the ≈ test of relative error) if you pick two vectors that happen to be nearly orthogonal.

Fortunately, this is a low-probability event, so it should suffice to set the seed. (Even if Julia changes the random-number generator in a future version, there is a low probability that we will need to change the seed.)

(Similarly for pretty much any ≈ test that involves computation with random numbers.)

@mcabbott, my understanding is that @testset sets the seed to the global seed, but this does not result in a deterministic test across multiple runs unless you call Random.seed! explicitly in the global scope first.

[mcabbott]

Thanks, sadly I did not get as far as actually reading the docs...

Should be simple to fix the seed though, will make a PR unless someone beats me to it.