Non-ASCII characters in library installation or cache path breaks library compilation caching

[vvb333007]

Describe the problem

I am using Windows 10, and my user name is in Cyrillic letters.

Every library gets recompiled every time I press "Compile button" even if no files were changed. This happens because .d (dependencies) files contain wrong path. Example below shows the problem win RTCLib but you can use any library/code to reproduce

I checked .d files in a temp folder and this is what I found inside:

(must be C:\Users\ֲВячеслав....)

C:\Users\ֲקוסכאג\AppData\Local\Temp\arduino\sketches\5E52C855D96EEA0FECA98993020C8EE1\libraries\RTClib\RTC_DS1307.cpp.o: \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\RTClib\src\RTC_DS1307.cpp \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\RTClib\src\RTClib.h \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\Adafruit_BusIO/Adafruit_I2CDevice.h \

So no wonder that is "up-to-date" check fails leading to full recompiling. Some paths are Ok tho.

PS: All the .o and .a files do exist in temp folder but get recompiled every time

To reproduce

Setup environment

$ arduino-cli version

arduino-cli  Version: git-snapshot Commit: c86ca3012 Date: 2024-07-31T11:02:30Z

$ mkdir -p "/tmp/Håkan/arduino-cli-directories/user"  # Note that the path contains the "å" character.

$ export ARDUINO_DIRECTORIES_USER="/tmp/Håkan/arduino-cli-directories/user"

$ arduino-cli lib install Servo

Downloading Servo@1.2.2...
Servo@1.2.2 Servo@1.2.2 already downloaded
Installing Servo@1.2.2...
Installed Servo@1.2.2

$ arduino-cli core install arduino:avr

Platform arduino:avr@1.8.6 already installed

$ mkdir "/tmp/ServoSketch"

$ printf '#include <Servo.h>\nvoid setup() {}\nvoid loop() {}\n' > "/tmp/ServoSketch/ServoSketch.ino"

Demo

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library will be compiled from source on this first compile command.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library objects cached during the previous compile will be used this time.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

[...]

🐛 The "Servo" library was compiled from source instead of using cached objects produced by the previous compile command.

Now we try again, but this time with a directories.user path that only contains ASCII characters:

$ mkdir -p "/tmp/Hakan/arduino-cli-directories/user"  # Note that the path contains only ASCII characters.

$ export ARDUINO_DIRECTORIES_USER="/tmp/Hakan/arduino-cli-directories/user"

$ arduino-cli lib install Servo

Downloading Servo@1.2.2...
Servo@1.2.2 Servo@1.2.2 already downloaded
Installing Servo@1.2.2...
Installed Servo@1.2.2

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library will be compiled from source on this first compile command.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Hakan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Hakan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

[...]

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library objects cached during the previous compile will be used this time.

[...]

Compiling library "Servo"
Using previously compiled file: C:\Users\per\AppData\Local\Temp\arduino\sketches\BFDE784884ED2EECC432F1D25AE2F6D2\libraries\Servo\nrf52\Servo.cpp.o

[...]

🙂 When directories.user was changed to a path that only contains ASCII characters, the library compilation caching system worked as expected.

Expected behavior

Expected behaviour: not to recompile files which were not modified

Arduino CLI version

c86ca3012

Operating system

Windows

Operating system version

• 10
• 11

Additional context

Originally reported at https://forum.arduino.cc/t/ide-2-3-2-very-slow-compiling/1281480

Additional reports

• https://forum.arduino.cc/t/lgvldemo-recompiles-libraries-every-time/1286840

Issue checklist

• [X] I searched for previous reports in the issue tracker
• [X] I verified the problem still occurs when using the latest nightly build
• [X] My report contains all necessary details

[vvb333007]

JFYI: I am compiling code for an "ESP32 Dev Board" target.