search

flame / blis

BLAS-like Library Instantiation Software Framework
Other
2.25k stars 365 forks source link

Netlib BLAS test failures should cause CI failure #518

Open devinamatthews opened 3 years ago

devinamatthews commented 3 years ago

See e.g. here. The build passes even though many BLAS tests fail. In this case, I have no idea why they fail but since they do so only for shared builds and not static it is probably a real bug that the CI workflow should indicate.

For context, similar BLAS test failures happen on Windows/AVX512 (#514) and are not reflected in the exit code of make check.

isuruf commented 3 years ago

The way the BLAS tests are setup, it's impossible to have them run on windows with shared builds.

This is because a local xerbla implementation needs to override the one in the blis library which works in Unix and in windows static builds, but with windows DLLs, the xerbla implementation in the DLL takes precedence.

fgvanzee commented 3 years ago

Ah yes, I forgot about that highly unfortunate feature of the BLAS test drivers. Thanks for that reminder, Isuru.

devinamatthews commented 3 years ago

So is the symptom of this seemingly random test failures or is there a more specific outcome?

isuruf commented 3 years ago

So is the symptom of this seemingly random test failures

Yes

devinamatthews commented 3 years ago

Well, that sucks. I'll revert the changes to .appveyor.yml but I still think that BLAS tests should trigger CI failure. I've have them fail in other circumstances where the BLIS tests did not and there was indeed a bug.

isuruf commented 3 years ago

With static libraries, it should work though.

fgvanzee commented 3 years ago

I recommend that we insert a comment somewhere (in the .appveyor.yml file?) that will remind future readers of this issue.

devinamatthews commented 3 years ago

I recommend that we insert a comment somewhere (in the .appveyor.yml file?) that will remind future readers of this issue.

Done

devinamatthews commented 3 years ago

Apparently all test failures are ignored in the Makefile (prepended with "- "). @fgvanzee can you remind me what the rationale for this is? Makes it hard for other people to incorporate "make check" into their CI pipelines.

fgvanzee commented 3 years ago

Apparently all test failures are ignored in the Makefile (prepended with "- "). @fgvanzee can you remind me what the rationale for this is?

I probably did this because I wanted make to continue whatever rule is being executed, even if an error is encountered.

Here's the relevant part of the GNU make documentation, for those who are curious.