Parsing build output with separate <path> and <filename>

[KamilSzczygiel]

Some build tools execute compiler from many various paths. For example tup ( https://github.com/gittup/tup ) executes the compiler from the folder of currently parsed Tupfile, which means that in a project with multiple Tupfiles (this is a typical scenario - in the project used as example I have 22 of them), the compiler will be invoked from multiple folders. Thus it is not possible to just set one working directory, which means that error parsing will not work. An example build output (slightly edited to remove useless compiler flags) with error message looks like this:

*  43% 1) test/Queue: arm-none-eabi-g++  [...]  -c QueueWrappers.cpp -o ../../output/test/Queue/QueueWrappers.o
QueueWrappers.cpp: In member function 'virtual int distortos::test::FifoQueueWrapper::emplace(uint8_t, distortos::test::OperationCountingType::Value) const':
QueueWrappers.cpp:58:1: error: expected ';' before '}' token
 }
 ^
 *** tup errors ***
 *** Command ID=835 failed with return value 1
tup error: Expected to write to file '../../output/test/Queue/QueueWrappers.o' from cmd 835 but didn't
 *** Additionally, command 835 failed to process input dependencies. These should probably be fixed before addressing the command failure.
 *** tup: 1 job failed.

The full path of file (relative to project root) is test/Queue/QueueWrappers.cpp. In this case compiler was run from test/Queue/ folder, but in the whole build process it is run from dozen of different folders. Thus I think it is not possible to actually use current error matching method.

The first idea I had was to introduce another match-group - <cwd>, which could easily be parsed from the error above (assuming the package allows parsing multiple-lines with single regex) - the final path to file would then be "<cwd>/<file>".

I guess the same issue would be present with recursive Makefiles, but it couldn't be solved like that. In that case maybe it would be possible to allow cwd variable to be a regex? This could also solve the issue with parsing of tup output.

Would it be possible to implement something like this? Maybe such error is possible to parse with current version of the package?

[fstiewitz]

build-tools needs absolute path names or paths that are relative to the directory where you invoked the build command (in your case tup). I don't know much (I don't know anything) about tup, so I have no idea if there is a way in tup to call g++ with absolute path names (other build systems do this by default). Should this tool not allow absolute path names, then someone could try to create a profile module for build-tools that reads cwd from tup's output (but that requires some coding). I'll try tup later and see how it works in one of my c++ projects.

[KamilSzczygiel]

build-tools needs absolute path names or paths that are relative to the directory where you invoked the build command (in your case tup). I don't know much (I don't know anything) about tup, so I have no idea if there is a way in tup to call g++ with absolute path names (other build systems do this by default). Should this tool not allow absolute path names, then someone could try to create a profile module for build-tools that reads cwd from tup's output (but that requires some coding).

The problem here is that you invoke tup from project's root, but then this tool invokes compiler from the folder where it found a Tupfile (this is a file which describes build - you can consider this to be a makefile fragment). If there are many Tupfiles (and this would be a typical use case), then the compiler is run from many different paths. The complete path can be parsed from each command without problems, but it is split into two parts. For example see here (almost all compiler flags omitted for clarity):

...
81% 116) [0.557s] source/threads: arm-none-eabi-g++ [...] -c DynamicThread.cpp -o ../../output/source/threads/DynamicThread.o
82% 117) [0.504s] source/scheduler: arm-none-eabi-g++ [...] -c statistics.cpp -o ../../output/source/scheduler/statistics.o
...

These are two commands that compile two source files from two different folders. These files, with "full" paths, are:

• source/threads/DynamicThread.cpp
• source/scheduler/statistics.cpp

These paths could be parsed from the output without problems - you could parse "source/threads" or "source/scheduler" into <path>, and then parse "DynamicThread.cpp" or "statistics.cpp" into <file> - the full path would be <path>/<file>.

(please note, that <file> doesn't need to be in the form "filename.exension" - it can as well be "some/sub/folder/filename.extension")

So in case of tools like tup, parsing the output would be pretty easy and this would need only support for new optional regex name - <path> or sth like that.

But compilation with recursive makefiles cannot be solved like that and this case is much much harder... An example of compilation using recursive make (I have removed compiler flags as well):

...
make[2]: Entering directory '/home/freddie/Elektronika/ARM/nuttx/nuttx/mm'
arm-none-eabi-gcc [...] -c  mm_initialize.c -o  bin/mm_initialize.o
arm-none-eabi-gcc [...] -c mm_sem.c -o  bin/mm_sem.o
...

In this case this is much harder - the full path of first and second file both need the first line - "make[x]: Entering directory ..." - parsed and saved for the <path> component...

I'll try tup later and see how it works in one of my c++ projects.

If you need any help with setting tup up, please let me know. I can also provide you with a simple test project that you could use to test your plugin or as a base for adding relevant files to your own test project. I could also provide you with a test project using recursive Makefile.

BTW - I consider "recursive Makefile" approach to be completely wrong, but there are quite a lot of projects that use it. Actually cmake generates recursive makefiles.

P.S. I see that my first post has some formatting issues related to "<" and ">" - I'll fix it in a moment, because without them it may seem completely cryptic.

[fstiewitz]

First of all, from my package's perspective, there is no difference between tup and recursive Makefiles. Both change their working directory mid-command and both indicate this change on another line than the actual error message.

However, the difference between "ordinary" recursive Makefiles and cmake's recursive Makefiles is that cmake calls g++ with the full path of the source file(s).

• g++ with absolute paths results in error messages with absolute paths (which are supported in the current version of this package)
• g++ with relative paths results in error messages with relative paths (which are supported if they are relative to the working directory specified in .build-tools.cson)

That aside, here is the real problem: build-tools parses errors line by line and you can only set one custom regular expression or one highlighting profile.

I think the best way to support your scenario is to allow users to specify more than one regular expression / profile. The user could have a regular expression with the cwd group I'm about to implement and a profile for the compiler they are using (in your case GCC/Clang).

This also fixes recursive Makefiles.

• for tup: [\d]+%\s[\d]+\)\s\[[\d]+\.[\d]+s\]\s(?<cwd> .+?):\s.+
• for recursive Makefiles: Entering directory '(?<cwd> .+?)'

With that being said, I have a few projects using cmake and autotools, but I have no projects using tup. If you could provide a simple tup project for testing, that'd be great :smile:

[KamilSzczygiel]

I think the best way to support your scenario is to allow users to specify more than one regular expression / profile. The user could have a regular expression with the cwd group I'm about to implement and a profile for the compiler they are using (in your case GCC/Clang).

Great - I'm looking forward to testing these new features (;

With that being said, I have a few projects using cmake and autotools, but I have no projects using tup. If you could provide a simple tup project for testing, that'd be great :smile:

Here's a test project that shows the basics of how tup works. Most of all it shows compilation of files from current directory (with "current directory" being the top-level folder - main.c - and some subfolder - A/A.c and B/B.c) and files from subfolders of current directory (B/1/B1.c and B/2/B2.c). It should work on both Linux and Windows with the most recent stable tup version (0.7.3).

https://www.dropbox.com/s/8dzs2xd1b7a6z5j/tup-example.tar.xz?dl=0&s=sl

After extracting just run tup:

$ tup
Initializing .tup in /home/freddie/test/tup-example
.tup repository initialized.
[ tup ] [0.006s] Scanning filesystem...
[ tup ] [0.013s] Reading in new environment variables...
[ tup ] [0.020s] Parsing Tupfiles...
 1) [0.001s] .
 2) [0.001s] A                                                                                                         
 3) [0.000s] B/1                                                                                                       
 4) [0.000s] B/2
 5) [0.001s] B
 [     ] 100%                                                                                                          
[ tup ] [0.024s] No files to delete.                                                                                   
[ tup ] [0.024s] Checking circular dependencies among groups...
[ tup ] [0.024s] Generating .gitignore files...
[ tup ] [0.031s] Executing Commands...
 1) [0.029s] gcc  -O2 -Wall -Wstrict-prototypes -Wextra -Wshadow -std=gnu99 -g -ggdb3 -ffunction-sections -fdata-sections -I.  -c main.c -o ./output/./main.c.o
 2) [0.042s] B: gcc  -O2 -Wall -Wstrict-prototypes -Wextra -Wshadow -std=gnu99 -g -ggdb3 -ffunction-sections -fdata-sections -I.  -c 2/B2.c -o ../output/B/2/B2.c.o
 3) [0.043s] B: gcc  -O2 -Wall -Wstrict-prototypes -Wextra -Wshadow -std=gnu99 -g -ggdb3 -ffunction-sections -fdata-sections -I.  -c B.c -o ../output/B/B.c.o
 4) [0.059s] A: gcc  -O2 -Wall -Wstrict-prototypes -Wextra -Wshadow -std=gnu99 -g -ggdb3 -ffunction-sections -fdata-sections -I.  -c A.c -o ../output/A/A.c.o
 5) [0.059s] B: gcc  -O2 -Wall -Wstrict-prototypes -Wextra -Wshadow -std=gnu99 -g -ggdb3 -ffunction-sections -fdata-sections -I.  -c 1/B1.c -o ../output/B/1/B1.c.o
 6) [0.050s] g++  -g -Wl,--gc-sections %<objects> -o ./output/./executable                                             
 [      ] 100%
[ tup ] [0.148s] Updated.

If you have problems running it - just let me know. This example is a stripped down version of the build files I use for my RTOS project ( https://github.com/DISTORTEC/distortos ).

[fstiewitz]

Better late than never (sorry for that). Recursive make files should work just fine, but tup requires a bit of configuration.

• The regular expression appears to be \d+%\s\d+\)\s(\[\d+\.\d+s\]\s)?(?<cwd> .+?):\s.+
• I recommend disabling standard output for tup. This way you'll only see errors from gcc (or an empty console pane on success).