search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.65k stars 1.64k forks source link

`meson test` doesn't show the error log for failed test on Windows while doing so on any *nix system #9195

Closed XVilka closed 3 years ago

XVilka commented 3 years ago

Describe the bug The same Meson configuration and command line prints the output of failed unit tests on meson test -C build or ninja -C build test on any *nix systems - Linux, FreeBSD, OpenBSD, NetBSD, etc. At the same time it fails to do so on Windows systems. It should do that consistently on all supported systems.

On Linux:

Run ninja -C build test
ninja: Entering directory `build'
[0/1] Running all tests.
 1/89 addr_interval                  OK              0.02s
 2/89 agraph                         OK              0.10s
 3/89 analysis_cc                    OK              0.05s
 4/89 analysis_class_graph           OK              0.10s
 5/89 analysis_function              OK              0.04s
 6/89 analysis_hints                 OK              0.04s
 7/89 analysis_meta                  OK              0.07s
 8/89 analysis_op                    OK              0.05s
 9/89 analysis_var                   OK              0.04s
10/89 analysis_global_var            OK              0.03s
11/89 analysis_xrefs                 OK              0.03s
12/89 annotated_code                 OK              0.02s
13/89 analysis_block                 OK              1.19s
14/89 base64                         OK              0.02s
15/89 autocmplt                      OK              0.96s
16/89 bin                            OK              0.04s
17/89 bin_lines                      OK              0.22s
18/89 binheap                        OK              0.02s
19/89 bitmap                         OK              0.02s
20/89 buf                            OK              0.17s
21/89 cmd                            OK              0.02s
22/89 config                         OK              0.02s
23/89 cons                           OK              0.02s
24/89 contrbtree                     OK              0.02s
25/89 core_bin                       OK              0.91s
26/89 core_cmd                       OK              0.31s
27/89 core_seek                      OK              0.52s
28/89 big                            OK              2.69s
29/89 debruijn                       OK              0.02s
30/89 debug                          OK              0.02s
31/89 debug_session                  OK              0.02s
32/89 diff                           OK              0.03s
33/89 core_task                      OK              0.19s
34/89 dwarf_info                     OK              0.03s
35/89 dwarf                          OK              0.05s
36/89 endian                         OK              0.02s
37/89 event                          OK              0.02s
38/89 file                           OK              0.02s
39/89 flags                          OK              0.02s
40/89 glob                           OK              0.02s
41/89 graph                          OK              0.02s
42/89 hash                           OK              0.02s
43/89 hex                            OK              0.02s
44/89 id_storage                     OK              0.02s
45/89 idpool                         OK              0.02s
46/89 idstorage                      OK              0.02s
47/89 intervaltree                   OK              0.30s
48/89 io                             OK              0.02s
49/89 itv                            OK              0.02s
50/89 io_ihex                        OK              0.09s
51/89 json                           OK              0.02s
52/89 list                           OK              0.02s
53/89 ovf                            OK              0.02s
54/89 pdb                            OK              0.23s
55/89 pj                             OK              0.02s
56/89 project_migrate                OK              0.59s
57/89 queue                          OK              0.02s
58/89 rbtree                         OK              0.10s
59/89 reg                            OK              0.02s
60/89 run                            OK              0.02s
61/89 rz_test                        OK              0.02s
62/89 rzpipe                         OK              0.16s
63/89 serialize_analysis             OK              0.16s
64/89 serialize_config               OK              0.02s
65/89 serialize_flag                 OK              0.02s
66/89 serialize_spaces               OK              0.02s
67/89 serialize_types                OK              0.05s
68/89 dwarf_integration              OK              2.23s
69/89 skiplist                       OK              0.02s
70/89 skyline                        OK              0.02s
71/89 spaces                         OK              0.02s
72/89 sparse                         OK              0.02s
73/89 stack                          OK              0.02s
74/89 str                            OK              0.02s
75/89 strbuf                         OK              0.02s
76/89 subprocess                     OK              0.04s
77/89 table                          OK              0.02s
78/89 task                           OK              0.02s
79/89 tree                           OK              0.02s
80/89 sign                           OK              0.38s
81/89 type                           OK              0.09s
82/89 uleb128                        OK              0.02s
83/89 unum                           OK              0.02s
84/89 util                           OK              0.02s
85/89 vector                         OK              0.02s
86/89 inflate_deflate                FAIL            0.03s   exit status 1
87/89 bin_vfiles                     OK              0.04s
88/89 cpu_platform_profiles          OK              0.70s
89/89 open_analyse_save_load_project OK              2.26s

The output from the failed tests:

86/89 inflate_deflate                FAIL            0.03s   exit status 1
--- command ---
10:49:49 /home/runner/work/rizin/rizin/build/test/unit/test_inflate_deflate
--- stdout ---
test_rz_inflate ERR
[XX] Fail at line 31: Windows inflate OS bits: expected 11, got 3.

test_rz_deflate OK
test_rz_inflate_buf OK
test_rz_deflate_buf OK

-------

Summary of Failures:

86/89 inflate_deflate                FAIL            0.03s   exit status 1

Ok:                 88  
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/runner/work/rizin/rizin/build/meson-logs/testlog.txt
FAILED: meson-test 
/usr/local/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.

On Windows:

%PYTHON%\Scripts\meson test -C build -t 10
ninja: Entering directory `C:\projects\rizin\build'
ninja: no work to do.
 1/89 addr_interval                  OK               0.02s
 2/89 agraph                         OK               0.11s
 3/89 analysis_cc                    OK               0.04s
 4/89 analysis_class_graph           OK               0.11s
 5/89 analysis_function              OK               0.03s
 6/89 analysis_hints                 OK               0.03s
 7/89 analysis_meta                  OK               0.05s
 8/89 analysis_op                    OK               0.05s
 9/89 analysis_var                   OK               0.03s
10/89 analysis_global_var            OK               0.02s
11/89 analysis_xrefs                 OK               0.02s
12/89 annotated_code                 OK               0.02s
13/89 analysis_block                 OK               0.80s
14/89 base64                         OK               0.00s
15/89 autocmplt                      OK               0.93s
16/89 big                            OK               0.64s
17/89 bin                            OK               0.06s
18/89 binheap                        OK               0.02s
19/89 bitmap                         OK               0.02s
20/89 bin_lines                      OK               0.20s
21/89 cmd                            OK               0.02s
22/89 config                         OK               0.02s
23/89 cons                           OK               0.02s
24/89 contrbtree                     OK               0.02s
25/89 buf                            OK               0.23s
26/89 core_cmd                       OK               0.34s
27/89 core_seek                      OK               0.62s
28/89 core_task                      OK               0.19s
29/89 debruijn                       OK               0.03s
30/89 debug                          OK               0.03s
31/89 debug_session                  OK               0.02s
32/89 diff                           OK               0.03s
33/89 dwarf                          OK               0.03s
34/89 dwarf_info                     OK               0.03s
35/89 dwarf_integration              OK               1.98s
36/89 endian                         OK               0.02s
37/89 event                          OK               0.02s
38/89 file                           OK               0.02s
39/89 flags                          OK               0.02s
40/89 glob                           OK               0.02s
41/89 graph                          OK               0.02s
42/89 hash                           OK               0.02s
43/89 hex                            OK               0.02s
44/89 id_storage                     OK               0.02s
45/89 idpool                         OK               0.02s
46/89 idstorage                      OK               0.02s
47/89 intervaltree                   OK               0.19s
48/89 io                             OK               0.02s
49/89 itv                            OK               0.02s
50/89 io_ihex                        OK               0.30s
51/89 json                           OK               0.02s
52/89 list                           OK               0.02s
53/89 ovf                            OK               0.00s
54/89 pdb                            OK               0.28s
55/89 pj                             OK               0.02s
56/89 core_bin                       OK               4.65s
57/89 queue                          OK               0.02s
58/89 rbtree                         OK               0.08s
59/89 reg                            OK               0.02s
60/89 run                            OK               0.02s
61/89 rz_test                        OK               0.02s
62/89 rzpipe                         OK               0.02s
63/89 serialize_analysis             OK               0.16s
64/89 serialize_config               OK               0.00s
65/89 serialize_flag                 OK               0.00s
66/89 serialize_spaces               OK               0.00s
67/89 serialize_types                OK               0.06s
68/89 project_migrate                OK               0.73s
69/89 skiplist                       OK               0.02s
70/89 skyline                        OK               0.02s
71/89 spaces                         OK               0.02s
72/89 sparse                         OK               0.01s
73/89 stack                          OK               0.00s
74/89 str                            OK               0.02s
75/89 strbuf                         OK               0.02s
76/89 subprocess                     OK               0.12s
77/89 table                          OK               0.02s
78/89 task                           OK               0.05s
79/89 tree                           OK               0.02s
80/89 sign                           OK               0.41s
81/89 uleb128                        OK               0.02s
82/89 unum                           OK               0.02s
83/89 type                           OK               0.09s
84/89 util                           OK               0.02s
85/89 vector                         OK               0.02s
86/89 inflate_deflate                FAIL             0.03s   exit status 1
>>> PATH=C:/projects/rizin/build/librz/analysis;C:/projects/rizin/build/librz/asm;C:/projects/rizin/build/librz/bin;C:/projects/rizin/build/librz/bp;C:/projects/rizin/build/librz/config;C:/projects/rizin/build/librz/cons;C:/projects/rizin/build/librz/core;C:/projects/rizin/build/librz/diff;C:/projects/rizin/build/librz/crypto;C:/projects/rizin/build/librz/debug;C:/projects/rizin/build/librz/egg;C:/projects/rizin/build/librz/flag;C:/projects/rizin/build/librz/hash;C:/projects/rizin/build/librz/io;C:/projects/rizin/build/librz/lang;C:/projects/rizin/build/librz/magic;C:/projects/rizin/build/librz/main;C:/projects/rizin/build/librz/parse;C:/projects/rizin/build/librz/reg;C:/projects/rizin/build/librz/search;C:/projects/rizin/build/librz/socket;C:/projects/rizin/build/librz/syscall;C:/projects/rizin/build/librz/type;C:/projects/rizin/build/librz/util;C:\projects\rizin\rizin-vs2019_64-v0.3.0-git\bin;C:\Python38-x64;C:\msys64\mingw64\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\\Extensions\Microsoft\IntelliCode\CLI;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Team Tools\Performance Tools\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Team Tools\Performance Tools;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\\x64;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\FSharp\Tools;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\devinit;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\\MSBuild\Current\Bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files\Git\cmd;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\PowerShell\7\;C:\Program Files\7-Zip;C:\Program Files\Microsoft\Web Platform Installer\;C:\Tools\NuGet;C:\Tools\PsTools;C:\Program Files\Git\usr\bin;C:\Program Files\Git LFS;C:\Program Files\Mercurial\;C:\Program Files (x86)\Subversion\bin;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin;C:\Tools\xUnit;C:\Tools\xUnit20;C:\Tools\NUnit\bin;C:\Tools\NUnit3\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform;C:\Ruby193\bin;C:\Tools\WebDriver;C:\Python27;C:\Python27\Scripts;C:\Program Files (x86)\nodejs\;C:\Program Files\nodejs;C:\Program Files (x86)\iojs;C:\Program Files\iojs;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files (x86)\Yarn\bin\;C:\go\bin;C:\Program Files\Java\jdk1.8.0\bin;C:\Program Files\erl10.7\bin;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\Tools\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Amazon\AWSCLI\;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\Microsoft DirectX SDK;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Tools\Doxygen;C:\Tools\Graphviz\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\LLVM\bin;C:\Tools\vcpkg;C:\Tools\Coverity\bin;C:\Program Files (x86)\NSIS;C:\Tools\Octopus;C:\Program Files\Meson\;C:\Program Files (x86)\Apache\Maven\bin;C:\Tools\GitVersion;C:\Users\appveyor\AppData\Local\Microsoft\WindowsApps;C:\Users\appveyor\.dotnet\tools;C:\Users\appveyor\AppData\Roaming\npm;C:\Users\appveyor\AppData\Local\Yarn\bin;C:\Program Files\AppVeyor\BuildAgent\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\Llvm\x64\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe MALLOC_PERTURB_=21 C:\projects\rizin\build\test/unit\test_inflate_deflate.exe
87/89 bin_vfiles                     OK               0.05s
88/89 cpu_platform_profiles          OK               0.86s
89/89 open_analyse_save_load_project OK               1.90s
Ok:                 88  
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0

To Reproduce

Expected behavior The same output on any system:

The output from the failed tests:

86/89 inflate_deflate                FAIL            0.03s   exit status 1
--- command ---
10:49:49 /home/runner/work/rizin/rizin/build/test/unit/test_inflate_deflate
--- stdout ---
test_rz_inflate ERR
[XX] Fail at line 31: Windows inflate OS bits: expected 11, got 3.

test_rz_deflate OK
test_rz_inflate_buf OK
test_rz_deflate_buf OK

-------

system parameters

See https://github.com/rizinorg/rizin/issues/1556

eli-schwartz commented 3 years ago

Huh, odd. Is the file created anyway? e.g. try listing C:\projects\rizin\build\meson-logs\ in the build runner.

XVilka commented 3 years ago

Ok, just recreated an error in a custom PR and listed log files:

Full log written to C:\projects\rizin\build\meson-logs\testlog.txt
dir C:\Projects\rizin\build\meson-logs
 Volume in drive C has no label.
 Volume Serial Number is 0CA6-1C58
 Directory of C:\Projects\rizin\build\meson-logs
08/31/2021  05:25 AM    <DIR>          .
08/31/2021  05:25 AM    <DIR>          ..
08/31/2021  05:25 AM            42,889 install-log.txt
08/31/2021  05:14 AM           176,636 meson-log.txt
08/31/2021  05:25 AM         1,764,002 testlog.json
08/31/2021  05:25 AM            34,323 testlog.junit.xml
08/31/2021  05:25 AM           602,866 testlog.txt
               5 File(s)      2,620,716 bytes
               2 Dir(s)  67,561,287,680 bytes free
type C:\Projects\rizin\build\meson-logs\testlog.txt
Log of Meson test suite run on 2021-08-31T05:25:45.102936
Inherited environme

Then I print the whole of the testlog.txt and it's looong, here are the lines that reported on other systems:

iles\CMake\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\LLVM\bin;C:\Tools\vcpkg;C:\Tools\Coverity\bin;C:\Program Files (x86)\NSIS;C:\Tools\Octopus;C:\Program Files\Meson\;C:\Program Files (x86)\Apache\Maven\bin;C:\Tools\GitVersion;C:\Users\appveyor\AppData\Local\Microsoft\WindowsApps;C:\Users\appveyor\.dotnet\tools;C:\Users\appveyor\AppData\Roaming\npm;C:\Users\appveyor\AppData\Local\Yarn\bin;C:\Program Files\AppVeyor\BuildAgent\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\Llvm\x64\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\VC\Linux\bin\ConnectionManagerExe C:\projects\rizin\build\test/unit\test_vector.exe
----------------------------------- output -----------------------------------
Gillian Seed: 1630387552
test_vector_init OK
test_vector_new OK
test_vector_fini ERR
[XX] Fail at line 75: init len: expected 6, got 1.
test_vector_clear OK
test_vector_free OK
test_vector_clone OK
test_vector_empty OK
test_vector_remove_at OK
test_vector_remove_range

See https://ci.appveyor.com/project/rizinorg/rizin/builds/40584734/job/gefpw4225gtlyi56#L3474

XVilka commented 3 years ago

Any updates on this one? Let me know if you need more information.

eli-schwartz commented 3 years ago

I just went and actually looked into it.

This is an intentional behavior change in commit bb9d8b5b80e5f4aa421b4ad549d71fd103f2b1dc.

You are NOT using the same version of meson on both systems -- or at least you are not using meson 0.59.1 on one of them. :)

Also, I just noticed you are running ninja test on Linux, and meson test on Windows. The former invokes meson test --print-errorlogs but you did not add that --print-errorlogs option to the Windows invocation.