Installing contributed board package breaks other boards packages

[elektor-labs]

After wrestling many hours with this problem I can only conclude that I have hit a bug. I have created my own contributed boards package that includes a platform and a toolchain. Installation of it (Windows only) works fine, all the files are placed where they should go, "hardware" and "tools" folders are created, etc., boards are available in the IDE. Perfect, no warnings, no errors. Only one problem remains: the compiler path from platform.txt is not recognized and remains empty. That this happens for my package I can accept as I may have made a mistake (even though I have tried everything possible), but after installing my package other packages that I have installed like Intel Arduino 101 stop working with the same problem: empty compiler path.

Arduino 101 IDE output sample with my package not installed, the arc32 compiler is found (the base path is C:\Users\CPV\AppData\Local\Arduino15\packages\Intel\tools\arc-elf32\1.6.4+1.0/):

C:\work\Arduino\arduino-1.6.7\arduino-builder -dump-prefs -logger=machine -hardware "C:\work\Arduino\arduino-1.6.7\hardware" -hardware "C:\Users\CPV\AppData\Local\Arduino15\packages" -tools "C:\work\Arduino\arduino-1.6.7\tools-builder" -tools "C:\work\Arduino\arduino-1.6.7\hardware\tools\avr" -tools "C:\Users\CPV\AppData\Local\Arduino15\packages" -built-in-libraries "C:\work\Arduino\arduino-1.6.7\libraries" -libraries "C:\Users\CPV\Documents\Arduino\libraries" -fqbn=Intel:arc32:arduino_101 -ide-version=10607 -build-path "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp" -warnings=none -prefs=build.warn_data_percentage=75 -verbose "C:\Users\CPV\AppData\Local\Temp\untitled921661166.tmp\sketch_feb19a\sketch_feb19a.ino"
C:\work\Arduino\arduino-1.6.7\arduino-builder -compile -logger=machine -hardware "C:\work\Arduino\arduino-1.6.7\hardware" -hardware "C:\Users\CPV\AppData\Local\Arduino15\packages" -tools "C:\work\Arduino\arduino-1.6.7\tools-builder" -tools "C:\work\Arduino\arduino-1.6.7\hardware\tools\avr" -tools "C:\Users\CPV\AppData\Local\Arduino15\packages" -built-in-libraries "C:\work\Arduino\arduino-1.6.7\libraries" -libraries "C:\Users\CPV\Documents\Arduino\libraries" -fqbn=Intel:arc32:arduino_101 -ide-version=10607 -build-path "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp" -warnings=none -prefs=build.warn_data_percentage=75 -verbose "C:\Users\CPV\AppData\Local\Temp\untitled921661166.tmp\sketch_feb19a\sketch_feb19a.ino"
"C:\Users\CPV\AppData\Local\Arduino15\packages\Intel\tools\arc-elf32\1.6.4+1.0/bin/arc-elf32-g++"  -c -mARCv2EM -mav2em -mlittle-endian -g -Os -Wall -fno-reorder-functions -fno-asynchronous-unwind-tables -fno-omit-frame-pointer -fno-defer-pop -Wno-unused-but-set-variable -Wno-main -ffreestanding -fno-stack-protector -mno-sdata -ffunction-sections -fdata-sections -fsigned-char  -fno-rtti -fno-exceptions -D__ARDUINO_ARC__ -std=c++11 -w -x c++ -E -CC -DF_CPU=32000000L -DARDUINO=10607 -DARDUINO_ARC32_TOOLS -DARDUINO_ARCH_ARC32 -D__CPU_ARC__ -DCLOCK_SPEED=32 -DCONFIG_SOC_GPIO_32 -DCONFIG_SOC_GPIO_AON -DINFRA_MULTI_CPU_SUPPORT -DCFW_MULTI_CPU_SUPPORT -DHAS_SHARED_MEM "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/common" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/drivers" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/bootcode" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/framework/include"        "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\cores\arduino" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\variants\arduino_101" "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp\sketch\sketch_feb19a.ino.cpp" -o "nul"
etc.

Example, IDE Arduino 101 output sample with my package installed, the arc32 compiler is no longer found (the base path is empty):

C:\work\Arduino\arduino-1.6.7\arduino-builder -dump-prefs -logger=machine -hardware "C:\work\Arduino\arduino-1.6.7\hardware" -hardware "C:\Users\CPV\AppData\Local\Arduino15\packages" -tools "C:\work\Arduino\arduino-1.6.7\tools-builder" -tools "C:\work\Arduino\arduino-1.6.7\hardware\tools\avr" -tools "C:\Users\CPV\AppData\Local\Arduino15\packages" -built-in-libraries "C:\work\Arduino\arduino-1.6.7\libraries" -libraries "C:\Users\CPV\Documents\Arduino\libraries" -fqbn=Intel:arc32:arduino_101 -ide-version=10607 -build-path "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp" -warnings=none -prefs=build.warn_data_percentage=75 -verbose "C:\Users\CPV\AppData\Local\Temp\untitled921661166.tmp\sketch_feb19a\sketch_feb19a.ino"
C:\work\Arduino\arduino-1.6.7\arduino-builder -compile -logger=machine -hardware "C:\work\Arduino\arduino-1.6.7\hardware" -hardware "C:\Users\CPV\AppData\Local\Arduino15\packages" -tools "C:\work\Arduino\arduino-1.6.7\tools-builder" -tools "C:\work\Arduino\arduino-1.6.7\hardware\tools\avr" -tools "C:\Users\CPV\AppData\Local\Arduino15\packages" -built-in-libraries "C:\work\Arduino\arduino-1.6.7\libraries" -libraries "C:\Users\CPV\Documents\Arduino\libraries" -fqbn=Intel:arc32:arduino_101 -ide-version=10607 -build-path "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp" -warnings=none -prefs=build.warn_data_percentage=75 -verbose "C:\Users\CPV\AppData\Local\Temp\untitled921661166.tmp\sketch_feb19a\sketch_feb19a.ino"
"/bin/arc-elf32-g++"  -c -mARCv2EM -mav2em -mlittle-endian -g -Os -Wall -fno-reorder-functions -fno-asynchronous-unwind-tables -fno-omit-frame-pointer -fno-defer-pop -Wno-unused-but-set-variable -Wno-main -ffreestanding -fno-stack-protector -mno-sdata -ffunction-sections -fdata-sections -fsigned-char  -fno-rtti -fno-exceptions -D__ARDUINO_ARC__ -std=c++11 -w -x c++ -E -CC -DF_CPU=32000000L -DARDUINO=10607 -DARDUINO_ARC32_TOOLS -DARDUINO_ARCH_ARC32 -D__CPU_ARC__ -DCLOCK_SPEED=32 -DCONFIG_SOC_GPIO_32 -DCONFIG_SOC_GPIO_AON -DINFRA_MULTI_CPU_SUPPORT -DCFW_MULTI_CPU_SUPPORT -DHAS_SHARED_MEM "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/common" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/drivers" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/bootcode" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\system/libarc32_arduino101/framework/include"        "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\cores\arduino" "-IC:\Users\CPV\AppData\Local\Arduino15\packages\Intel\hardware\arc32\1.0.4\variants\arduino_101" "C:\Users\CPV\AppData\Local\Temp\build3238de90840fcb3f69070fa35eb446f0.tmp\sketch\sketch_feb19a.ino.cpp" -o "nul"
exec: "/bin/arc-elf32-g++": file does not exist
Erreur lors de la compilation.

My JSON file can be found here so you can try for yourself: https://raw.githubusercontent.com/ElektorLabs/arduino/master/package_elektor_111_index.json

First install "Intel Curie boards by Intel version 1.0.4", compile an empty sketch (empty loop & setup functions as created by the IDE when opened) for the Arduino 101 board: OK. Then install my package "Elektor AVR Boards by Elektor.Labs version 1.1.1", compile the empty sketch again for the Arduino 101 board: FAIL. Uninstall my package, recompile the empty sketch once more for the Arduino 101: OK.

This is driving me crazy, what the diggins is wrong?

Clemens

[per1234]

The problem is caused by the file Arduino15\packages\Elektor\tools\avr-gcc\4.9.2-elektor\avr\builtin_tools_versions.txt. If you get rid of that file then compilation for the Arduino 101 starts working again. There's more information on the purpose of builtin_tools_versions.txt at https://groups.google.com/a/arduino.cc/forum/m/#!msg/developers/TDYnunLEqXU/w6jRV_FcBAAJ:

Paul Stoffregen ha scritto il 18/09/2015 alle 15:25: I can't seem to get {runtime.ide.path} to work at all in the new builder. I do need some way to construct a {compiler.path} that is the location where I install my customized toolchain.....

There are two ways

First one (preferred) is to create a folder structure such as "tools > name of tool > version of tool > every other file"

This is what boards manager does. For example

tools/bossac/1.6-arduino

This will create runtime.tools.bossac.path and runtime.tools.bossac-1.6.path

Second one is to put a builtin_tools_versions.txt file where your tools are installed. That's what the IDE you download does, because it bundles the AVR toolchain You can find that file in IDE/hardware/tools/avr/

So it sounds like since the folder structure of your tools is already creating runtime.tools.avr-gcc-4.9.2-elektor.path there's no need for builtin_tools_versions.txt.

You will also need to change the line in your platform.txt compiler.path={runtime.tools.avr-gcc-4.9.2-elektor.path}/tools/avr/bin/ to compiler.path={runtime.tools.avr-gcc-4.9.2-elektor.path}/avr/bin/

I think it would be a very good idea to make the IDE be able to handle errors in a hardware package without breaking other packages. We have beginners installing packages made by 3rd parties so the IDE needs to be fairly resilient to things like this.

[per1234]

After those changes when I try to compile for Arduino/Genuino Uno(using Arduino IDE 1.6.7 or 1.6.8 2016/02/19 10:33) I get:

exec: "C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\Elektor\\tools\\avr-gcc\\4.9.2-elektor/bin/avr-g++": file does not exist

So I changed C:\Users\per\AppData\Local\Arduino15\packages\Elektor\tools\avr-gcc to C:\Users\per\AppData\Local\Arduino15\packages\Elektor\tools\avr-gcc-elektor and changed:

compiler.path={runtime.tools.avr-gcc-4.9.2-elektor.path}/avr/bin/

to:

compiler.path={runtime.tools.avr-gcc-elektor-4.9.2-elektor.path}/avr/bin/

in C:\Users\per\AppData\Local\Arduino15\packages\Elektor\hardware\avr\1.1.1\platform.txt and that solves that issue and the Elektor boards still compile. For it to install like this using Boards Manager you would need to change the line:

"name": "avr-gcc",

to:

"name": "avr-gcc-elektor",

in package_elektor_111_index.json

I don't encounter the same problem with avrdude.

Again, this seems less than ideal that one package can break another package so easily, would be nice if the IDE was able to handle this.

[elektor-labs]

Excellent! Your suggestions solved all the problems I was seeing. You also confirmed my suspicion that tool folders must have unique names even though they reside on different paths.

The board manager & library manager will need better documentation because it is pretty complicated to get something working properly.

[per1234]

@ElektorLabs I'm not sure you should close this issue. Sure we have solved your problem(hopefully), but maybe the developers could look at this and see if there is a way to make the IDE handle these sorts of things without breaking other packages. It seems reasonable to allow multiple packages to have the same tool name.

[rei-vilo]

The cause is up again. See http://forum.arduino.cc/index.php?topic=391429.msg2695845#msg2695845

[per1234]

I have created a new issue for the problem caused by builtin_tools_versions.txt: https://github.com/arduino/Arduino/issues/6474 since this issue report got a bit long from us figuring out the problem and it's difficult to reproduce the issue only from the information given here.

I can't reproduce the second issue that was caused by a non-unique tool name (https://github.com/arduino/Arduino/issues/4593#issuecomment-187064959). I remember that was fixed in Arduino IDE 1.6.11.