three failing tests (in19, in20, in22)

[janusw]

After building today's flang master on Ubuntu 16.04 (x86_64) using LLVM 5, I see two test failures:

Failing Tests (2):
    Flang :: f90_correct/lit/wa03.sh
    Flang :: f90_correct/lit/wa14.sh

  Expected Passes    : 81
  Unexpected Failures: 2

For details, see below.

Cheers, Janus

$ make check-flang
Built target not
Built target FileCheck
Built target count
Running Flang regression tests
lit.py: /home/janus/llvm5/flang/test/lit.cfg:196: note: using flang: '/home/janus/llvm5/install/bin/flang'
FAIL: Flang :: f90_correct/lit/wa03.sh (60 of 83)
******************** TEST 'Flang :: f90_correct/lit/wa03.sh' FAILED ********************
Script:
--
FLAGS="" TEST_SRC=/home/janus/llvm5/flang/test/f90_correct/lit/wa03.sh MAKE_FILE_DIR=/home/janus/llvm5/flang/test/f90_correct/lit/.. bash /home/janus/llvm5/flang/test/f90_correct/lit/runmake | tee /home/janus/llvm5/flang/build/test/f90_correct/lit/Output/wa03.sh.tmp
cat /home/janus/llvm5/flang/build/test/f90_correct/lit/Output/wa03.sh.tmp | /home/janus/llvm5/flang/build/./bin/FileCheck /home/janus/llvm5/flang/test/f90_correct/lit/runmake
--
Exit Code: 1

Command Output (stdout):
--
Test name: wa03
rm -f wa03.out core *.d *.mod FOR*.DAT FTN* ftn* fort.*
------------------------------------ building test build
clang -c  /home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c -o check.o
flang -c -I/home/janus/llvm5/flang/test/f90_correct/lit/../src      /home/janus/llvm5/flang/test/f90_correct/lit/../src/wa03.f90 -o wa03.o
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa03.mk:24: recipe for target 'build' failed
flang -I/home/janus/llvm5/flang/test/f90_correct/lit/../src      wa03.o check.o  -o wa03.out
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa03.mk:24: recipe for target 'build' failed
------------------------------------ executing test wa03
wa03.out
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa03.mk:32: recipe for target 'run' failed
make: 'verify' is up to date.

--
Command Output (stderr):
--
/home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c:114:9: warning: implicitly declaring library function 'abs' with type 'int (int)' [-Wimplicit-function-declaration]
        diff = abs(ires - iexp);
               ^
/home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c:114:9: note: include the header <stdlib.h> or explicitly provide a declaration for 'abs'
1 warning generated.
F90-F-0004-Corrupt or Old Module file ./m.mod (/home/janus/llvm5/flang/test/f90_correct/lit/../src/wa03.f90: 25)
F90/x86-64 Linux Flang - 1.5 2017-05-01: compilation aborted
make: [build] Error 1 (ignored)
clang-5.0: error: no such file or directory: 'wa03.o'
make: [build] Error 1 (ignored)
make: wa03.out: Command not found
make: *** [run] Error 127
/home/janus/llvm5/flang/test/f90_correct/lit/runmake:29:10: error: expected string not found in input
# CHECK: {{([1-9][0-9]* tests PASSED\. 0 tests failed|^PASS[[:space:]])}}
         ^
<stdin>:1:1: note: scanning from here
Test name: wa03
^

--

********************
FAIL: Flang :: f90_correct/lit/wa14.sh (68 of 83)
******************** TEST 'Flang :: f90_correct/lit/wa14.sh' FAILED ********************
Script:
--
FLAGS="" TEST_SRC=/home/janus/llvm5/flang/test/f90_correct/lit/wa14.sh MAKE_FILE_DIR=/home/janus/llvm5/flang/test/f90_correct/lit/.. bash /home/janus/llvm5/flang/test/f90_correct/lit/runmake | tee /home/janus/llvm5/flang/build/test/f90_correct/lit/Output/wa14.sh.tmp
cat /home/janus/llvm5/flang/build/test/f90_correct/lit/Output/wa14.sh.tmp | /home/janus/llvm5/flang/build/./bin/FileCheck /home/janus/llvm5/flang/test/f90_correct/lit/runmake
--
Exit Code: 1

Command Output (stdout):
--
Test name: wa14
rm -f wa14.out core *.d *.mod FOR*.DAT FTN* ftn* fort.*
------------------------------------ building test build
clang -c  /home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c -o check.o
flang -c -I/home/janus/llvm5/flang/test/f90_correct/lit/../src      /home/janus/llvm5/flang/test/f90_correct/lit/../src/wa14.f90 -o wa14.o
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa14.mk:24: recipe for target 'build' failed
flang -I/home/janus/llvm5/flang/test/f90_correct/lit/../src      wa14.o check.o  -o wa14.out
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa14.mk:24: recipe for target 'build' failed
------------------------------------ executing test wa14
wa14.out
/home/janus/llvm5/flang/test/f90_correct/lit/../inc/wa14.mk:32: recipe for target 'run' failed
make: 'verify' is up to date.

--
Command Output (stderr):
--
/home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c:114:9: warning: implicitly declaring library function 'abs' with type 'int (int)' [-Wimplicit-function-declaration]
        diff = abs(ires - iexp);
               ^
/home/janus/llvm5/flang/test/f90_correct/lit/../src/check.c:114:9: note: include the header <stdlib.h> or explicitly provide a declaration for 'abs'
1 warning generated.
F90-F-0004-Unable to open MODULE file test.mod (/home/janus/llvm5/flang/test/f90_correct/lit/../src/wa14.f90: 31)
F90/x86-64 Linux Flang - 1.5 2017-05-01: compilation aborted
make: [build] Error 1 (ignored)
clang-5.0: error: no such file or directory: 'wa14.o'
make: [build] Error 1 (ignored)
make: wa14.out: Command not found
make: *** [run] Error 127
/home/janus/llvm5/flang/test/f90_correct/lit/runmake:29:10: error: expected string not found in input
# CHECK: {{([1-9][0-9]* tests PASSED\. 0 tests failed|^PASS[[:space:]])}}
         ^
<stdin>:1:1: note: scanning from here
Test name: wa14
^

--

[sscalpone]

Looks like tests are overwriting each others module files. You should be able to get success today with lit -j 1. We have a fix in the works that doesn't involve rewriting all of the test files.

[janusw]

At c8cacad7f3d8840267cfc905ab03c0f8740a829d, the failures seen earlier have gone away and "make check-flang" instead reports a different set of failures:

Failing Tests (3):
    Flang :: f90_correct/lit/in19.sh
    Flang :: f90_correct/lit/in20.sh
    Flang :: f90_correct/lit/in22.sh

  Expected Passes    : 899
  Unexpected Failures: 3

[pawosm-arm]

More should appear when you build flang with today's LLVM master:

Failing Tests (8):
    Flang :: f90_correct/lit/bs00.sh
    Flang :: f90_correct/lit/fa07.sh
    Flang :: f90_correct/lit/fs22406.sh
    Flang :: f90_correct/lit/fs22408.sh
    Flang :: f90_correct/lit/ia00.sh
    Flang :: f90_correct/lit/in19.sh
    Flang :: f90_correct/lit/in20.sh
    Flang :: f90_correct/lit/in22.sh

  Expected Passes    : 896
  Unexpected Failures: 8

[janusw]

I have rebuilt with today's flang master (and the release_50 branch of LLVM) and still see the same three failures as before, but no more than that.

make check-flang and make flang-test automatically use 12 threads on my system. It is not clear to me how to run the tests with a single thread only.

[sscalpone]

The problem running tests with threads is fixed. To run with a single thread, you run lit with -j 1. I don't know how to do that from the makefile.

Those tests show as failing; however, the actual test results are ok. I'll fix up the tests or the script that we use to check for success.

[pawosm-arm]

fs22406 and fs22408 were failing due to internally used bugfix for issue #178