Library auto-detection breaks if headers from libraries are needed and g++ does not output English error messages

nicoroeser commented 3 years ago

Bug Report

Current behavior

Tested with LANG=de_DE.UTF-8:

Compilation of a project from Arduino IDE or from the command line fails because the necessary include paths (for used libraries) are not appended to the g++ command. The auto-detection does not work properly because it expects a hard-coded string, which is not present in the compiler output in all locales.

Expected behavior

Compilation works independently from the current locale.

The code should set LC_MESSAGES before executing g++.

Environment

CLI version (output of arduino-cli version): tested with arduino-builder-1.5.4, which links arduino-cli as of 2020-06-12T15:26:41-00:00 (3b18571924405b9175cd914d4ef20b07618b92d4).
OS and platform: tested on a Gentoo system.

Additional context

Hard-coded error message is in legacy/builder/includes_finder_with_regexp.go. The regular expression may be safe and match in all locales. However, the (more) correct way is to run the compiler in a known locale (POSIX or C at best, because a machine will be reading the output).

silvanocerza commented 3 years ago

This is an arduino-builder issue right? If that's the case am gonna move it into that repo and probably we'll never fix it.

The arduino-builder is deprecated, unmaintained, old and dying. I strongly suggest you use the Arduino CLI instead of the builder.

By the way I couldn't reproduce this issue with arduino-cli if you can provide an example to reproduce it I'll be happy to fix it.

nicoroeser commented 3 years ago

Well… depends on how you see it.

We have an IDE (arduino) which calls a tool (arduino-builder) which uses code from another project (arduino-cli).

As the more recent IDE (arduino-ide) still does beta releases, let’s assume that many people still use arduino (version 1.8.x). (May also be true for some time in future, depending on their operating system distribution.) And all of us probably want user’s Arduino IDE to work as expected.

Either the issue is in arduino – because it uses arduino-builder for building instead of arduino-cli.
Or the issue is in arduino-builder – because it pulls in (legacy) code from arduino-cli, and that code is known to be buggy – or because it pulls in the code without working around the locale issue itself.
Or the issue is in arduino-cli where some input text is compared to a locale-specific string without forcing the input text to be in the same locale. That’s the source of the problem (the broken code), see the link to the hard-coded error message in the original message.

From my programmer’s point of view, a fix should be made in arduino-cli: set LC_MESSAGES before calling g++. I’m just not fluent enough in Go to submit a patch, sorry.

From another angle, I agree that arduino-builder should be left dying, and arduino-cli used instead. But then again, we should probably not drastically change the toolchain that arduino uses without a big version update.

We could argue that arduino-cli is almost fine, and just its interface is not specified properly, because the code in question ought to be called only with the C locale. (A bit unexpected for the calling code, if you ask me.) That would move the bug to arduino-builder.

arduino-builder pulls in the problem from arduino-cli when the former is built, as far as I could find out.

arduino uses arduino-builder for its compilations; at least in version 1.8.13, which is the (a bit dated) version I tested with.

No matter how you look at it, starting compilation in arduino will produce error messages for some users (who typically do not know or care about toolchain details). We can rely on distributions to work around this, but I think it’s better solved upstream.

Steps to reproduce

Ensure you have the LC_MESSAGES files for avr-gcc for the test locale (de_DE) installed for this to work, or test with another locale (which has non-English avr-g++ error messages). (I have tested with avr-g++ version 10.2.0; yes, also not the most recent version.)
We’ll be using an external library. In this example, Servo. Make sure it is properly installed.

Create a tiny sketch like this one:

#include <Servo.h>
void setup() {}
void loop() {}

Close the Arduino IDE if it is running.
Open the Arduino IDE with our non-English test locale:
```
env LC_MESSAGES=de_DE.UTF-8 arduino
```
Open the sketch from above.
Start compilation (by clicking the checkmark button, pressing Ctrl+R, or from the menu).

Result should be similar to the following:

/usr/share/arduino/arduino-builder -dump-prefs -logger=machine -hardware /usr/share/arduino/hardware -tools /usr/share/arduino/hardware/tools/avr -libraries /home/user/Arduino/libraries -fqbn=arduino:avr:uno -ide-version=10813 -build-path /tmp/arduino_build_752874 -warnings=more -build-cache /tmp/arduino_cache_314535 -prefs=build.warn_data_percentage=75 -verbose /home/user/Arduino/arduino-cli-issue1488/arduino-cli-issue1488.ino
/usr/share/arduino/arduino-builder -compile -logger=machine -hardware /usr/share/arduino/hardware -tools /usr/share/arduino/hardware/tools/avr -libraries /home/user/Arduino/libraries -fqbn=arduino:avr:uno -ide-version=10813 -build-path /tmp/arduino_build_752874 -warnings=more -build-cache /tmp/arduino_cache_314535 -prefs=build.warn_data_percentage=75 -verbose /home/user/Arduino/arduino-cli-issue1488/arduino-cli-issue1488.ino
Using board 'uno' from platform in folder: /usr/share/arduino/hardware/arduino/avr
Using core 'arduino' from platform in folder: /usr/share/arduino/hardware/arduino/avr
Detecting libraries used...
/usr/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10813 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -I/usr/share/arduino/hardware/arduino/avr/cores/arduino -I/usr/share/arduino/hardware/arduino/avr/variants/standard /tmp/arduino_build_752874/sketch/arduino-cli-issue1488.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE
Error while detecting libraries included by /tmp/arduino_build_752874/sketch/arduino-cli-issue1488.ino.cpp
Generating function prototypes...
/usr/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10813 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -I/usr/share/arduino/hardware/arduino/avr/cores/arduino -I/usr/share/arduino/hardware/arduino/avr/variants/standard /tmp/arduino_build_752874/sketch/arduino-cli-issue1488.ino.cpp -o /tmp/arduino_build_752874/preproc/ctags_target_for_gcc_minus_e.cpp -DARDUINO_LIB_DISCOVERY_PHASE
/home/user/Arduino/arduino-cli-issue1488/arduino-cli-issue1488.ino:1:10: schwerwiegender Fehler: Servo.h: Datei oder Verzeichnis nicht gefunden
    1 | #include <Servo.h>
      |          ^~~~~~~~~
Kompilierung beendet.
exit status 1
Fehler beim Kompilieren für das Board Arduino Uno.

For comparison, close the Arduino IDE, and restart it, but this time use env LC_MESSAGES=C arduino. The GUI will show another language, of course, and compilation of the tiny sketch will succeed.

Workaround

Ensure that a small wrapper script is used instead of the original arduino-builder.

#!/bin/sh
# This is /usr/bin/arduino-builder.
# Workaround for <https://github.com/arduino/arduino-cli/issues/1488>.

exec env LC_MESSAGES=C /usr/bin/arduino-builder.orig "$@"

cmaglie commented 2 years ago

Can you paste the output of:

/usr/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10813 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -I/usr/share/arduino/hardware/arduino/avr/cores/arduino -I/usr/share/arduino/hardware/arduino/avr/variants/standard /tmp/arduino_build_752874/sketch/arduino-cli-issue1488.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE

with and without the LC_MESSAGE var set?

I'd like to understand better the origin of the problem because the regexp that extracts the missing include is this one:

(?ms)^\s*#[ \t]*include\s*[<"](\S+)[">]

So it seems it should be independent from the language, but it will fail if some other things are printed around the include line, for example the 1 | before the #include in:

    1 | #include <Servo.h>
      |          ^~~~~~~~~

https://regex101.com/r/NgQaBu/1

nicoroeser commented 2 years ago

I tried:

args='-c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10813 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -I/usr/share/arduino/hardware/arduino/avr/cores/arduino -I/usr/share/arduino/hardware/arduino/avr/variants/standard /tmp/arduino_build_testing/sketch/arduino-cli-issue1488.ino.cpp -o /dev/null -DARDUINO_LIB_DISCOVERY_PHASE'

for l in de_DE.UTF-8 en_US.UTF-8 ; do
    printf '===== %s =====\n' "$l"
    env "LC_MESSAGES=$l" /usr/bin/avr-g++ $args 2>&1 | cat
    env "LC_MESSAGES=$l" /usr/bin/avr-g++ $args 2>&1 | hexdump -C
    printf \\n
done

I added cat so that the output would not go to a terminal directly (in order to disable colored output). This should resemble how g++ is called by arduino-builder, I think. I also added hex dumps so that it is clear where space and/or tab characters are (if any).

Output:

===== de_DE.UTF-8 =====
/home/user/Arduino/arduino-cli-issue1488/arduino-cli-issue1488.ino:1:10: schwerwiegender Fehler: Servo.h: Datei oder Verzeichnis nicht gefunden
Kompilierung beendet.
00000000  2f 68 6f 6d 65 2f 75 73  65 72 2f 41 72 64 75 69  |/home/user/Ardui|
00000010  6e 6f 2f 61 72 64 75 69  6e 6f 2d 63 6c 69 2d 69  |no/arduino-cli-i|
00000020  73 73 75 65 31 34 38 38  2f 61 72 64 75 69 6e 6f  |ssue1488/arduino|
00000030  2d 63 6c 69 2d 69 73 73  75 65 31 34 38 38 2e 69  |-cli-issue1488.i|
00000040  6e 6f 3a 31 3a 31 30 3a  20 73 63 68 77 65 72 77  |no:1:10: schwerw|
00000050  69 65 67 65 6e 64 65 72  20 46 65 68 6c 65 72 3a  |iegender Fehler:|
00000060  20 53 65 72 76 6f 2e 68  3a 20 44 61 74 65 69 20  | Servo.h: Datei |
00000070  6f 64 65 72 20 56 65 72  7a 65 69 63 68 6e 69 73  |oder Verzeichnis|
00000080  20 6e 69 63 68 74 20 67  65 66 75 6e 64 65 6e 0a  | nicht gefunden.|
00000090  4b 6f 6d 70 69 6c 69 65  72 75 6e 67 20 62 65 65  |Kompilierung bee|
000000a0  6e 64 65 74 2e 0a                                 |ndet..|
000000a6

===== en_US.UTF-8 =====
/home/user/Arduino/arduino-cli-issue1488/arduino-cli-issue1488.ino:1:10: fatal error: Servo.h: No such file or directory
compilation terminated.
00000000  2f 68 6f 6d 65 2f 75 73  65 72 2f 41 72 64 75 69  |/home/user/Ardui|
00000010  6e 6f 2f 61 72 64 75 69  6e 6f 2d 63 6c 69 2d 69  |no/arduino-cli-i|
00000020  73 73 75 65 31 34 38 38  2f 61 72 64 75 69 6e 6f  |ssue1488/arduino|
00000030  2d 63 6c 69 2d 69 73 73  75 65 31 34 38 38 2e 69  |-cli-issue1488.i|
00000040  6e 6f 3a 31 3a 31 30 3a  20 66 61 74 61 6c 20 65  |no:1:10: fatal e|
00000050  72 72 6f 72 3a 20 53 65  72 76 6f 2e 68 3a 20 4e  |rror: Servo.h: N|
00000060  6f 20 73 75 63 68 20 66  69 6c 65 20 6f 72 20 64  |o such file or d|
00000070  69 72 65 63 74 6f 72 79  0a 63 6f 6d 70 69 6c 61  |irectory.compila|
00000080  74 69 6f 6e 20 74 65 72  6d 69 6e 61 74 65 64 2e  |tion terminated.|
00000090  0a                                                |.|
00000091

This makes me think that probably function findIncludeForOldCompilers causes the problem, because it cannot find the line introducing the error message: it looks for "fatal error".

arduino / arduino-cli