Closed KurtE closed 2 years ago
Thanks so much for your report @KurtE!
I was able to determine that this is actually a bug in Arduino CLI, so I have transferred the issue to the appropriate repository.
This is not in any way specific to the Teensy Arduino boards platform, but that platform does provide a dramatic demonstration of the bug due to the Library Manager installation of "SD" being given priority over the platform bundled installation. In most cases (e.g., "USBHost" lib and arduino:samd
platform), the bundled library gets priority, so the bug only manifests in the oddity of the library removed from <directories.user>/libraries
still appearing under a "Multiple libraries were found for" section of the verbose compilation output.
I bisected the bug to f0245bc2da6a56fccea7b2c9ea09e85fdcc52cb8
In case it might assist the developers in the investigation, I'll provide a demo of the issue using Arduino CLI directly:
$ arduino-cli version
arduino-cli.exe Version: 0.23.0 Commit: 899dc91b Date: 2022-06-11T07:40:48Z
$ mkdir "/tmp/FooSketch"
$ printf "#include <SD.h>\nvoid setup(){}\nvoid loop(){}" > "/tmp/FooSketch/FooSketch.ino"
$ export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS="https://www.pjrc.com/teensy/td_156/package_teensy_index.json"
$ arduino-cli core update-index
[...]
$ arduino-cli core install teensy:avr # platform chosen because it has bundled lib w/ header filename also present in an LM lib
[...]
$ arduino-cli lib install SD # this lib also bundled w/ teensy:avr
[...]
$ arduino-cli daemon
Run these grpcurl
commands in another terminal from the root of the Arduino CLI repository:
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Create
{
"instance": {
"id": 1
}
}
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
-d '{"instance": {"id": 1}}' \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Init # use the instance ID provided by the Create response in this and all subsequent commands
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
-proto cc/arduino/cli/commands/v1/compile.proto \
-d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Compile # compile the sketch
[...]
$ echo $? # compilation was successful as expected (the LM library will compile for teensy:avr:teensy41 with this simple sketch)
0
$ arduino-cli lib uninstall SD # remove the LM installation of the lib
Uninstalling SD@1.2.4...
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
-proto cc/arduino/cli/commands/v1/compile.proto \
-d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Compile # compile the sketch again
[...]
$ echo $? # compilation failed unexpectedly, it should have passed, using the platform bundled SD library this time
77
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
-d '{"instance": {"id": 1}}' \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Init # reinitialize the instance
$ grpcurl \
-plaintext \
-import-path ./rpc \
-proto cc/arduino/cli/commands/v1/commands.proto \
-proto cc/arduino/cli/commands/v1/compile.proto \
-d '{"instance": {"id": 1}, "fqbn": "teensy:avr:teensy41", "sketch_path": "/tmp/FooSketch/FooSketch.ino"}' \
127.0.0.1:50051 \
cc.arduino.cli.commands.v1.ArduinoCoreService.Compile # compile the sketch
[...]
$ echo $? # compilation now passes
0
🐛 The second compilation failed. It was necessary to reinitialize Arduino CLI in order to get it to recognize the removal of the library.
@per1234 @cmaglie
Sorry maybe it is just me, but I think there is something missing in the transition from Arduino 1.x to Arduino 2.x as it relates to library search order.
That is with for example Arduino 1.8.19, The Arduino installer, did install a version of the SD library.
In: C:\arduino-1.8.19\libraries
When I install Teensyduino: there is a version of it installed at: C:\arduino-1.8.19\hardware\teensy\avr\libraries Likeise ESP32 has it's version installed at: C:\Users\kurte\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.3\libraries
Now if I am working on the SD library, I may put a version up at: C:\Users\kurte\Documents\Arduino\libraries
And if I am not mistaken, the library search order is: a) Highest priority: C:\Users\kurte\Documents\Arduino\libraries b) Second Priority: C:\arduino-1.8.19\hardware\teensy\avr\libraries c) lowest priority: C:\arduino-1.8.19\libraries
Which makes sense.
But now with the Current RC for Arduino V2 you got rid of lowest priority and now install the default IDE libraries in the sketch folder (highest priority), which breaks all of the setups that use board specific libraries.
Also as per my other issue (https://github.com/arduino/arduino-ide/issues/798)
The install at least at that time, wiped out my custom library I had installed in the
the library search order is:
This is a subject I absolutely love to discuss, but that discussion would be completely off topic for this issue. I will point you to the complete documentation here in case you would like to learn about how it really works:
https://arduino.github.io/arduino-cli/dev/sketch-build-process/#dependency-resolution
If you have any questions or would like to discuss that more, please post over on the Arduino forum. I'll be happy to give you the full "-v" over there.
with the Current RC for Arduino V2 you got rid of lowest priority and now install the default IDE libraries in the sketch folder (highest priority), which breaks all of the setups that use board specific libraries.
That is off topic for discussion in this issue and for discussion in this repository. If there are bug report or feature request you would like to make that is distinct from the existing one at https://github.com/arduino/arduino-ide/issues/798, please do submit them. But let's keep the discussion in this issue focused exclusively on Arduino CLI not recognizing library removals during a session.
@per1234 - At least from this 1 quick test, Arduino CLI is properly recognizing library removals. I'm pretty sure this is an IDE 2.0 issue, not a problem with Arduino CLI. I'd recommend moving this back to the IDE repository. Perhaps the title should be edited too, to more clearly emphasize the real problem that the wrong library ends up getting used.
I'm not quite sure what "during a session" means regarding Arduino CLI, but I here's the test I tried just now to check whether Arduino CLI is not recognizing library removals. But here's the test I ran just now.
First, I'm using the latest nightly, Linux x86-64 build.
paul@preston:/tmp > /tmp/arduino-cli version
arduino-cli Version: nightly-20220612 Commit: 899dc91 Date: 2022-06-12T01:40:11Z
My machine's current packages are a result of having recently run IDE 2.0.0-rc7.
When I run it to compile a sketch which uses the SD library, I get this:
paul@preston:/tmp > /tmp/arduino-cli compile --clean -b teensy:avr:teensy36 ~/teensy/sketch/T40_Tester
Sketch uses 119748 bytes (11%) of program storage space. Maximum is 1048576 bytes.
Global variables use 14952 bytes (5%) of dynamic memory, leaving 247192 bytes for local variables. Maximum is 262144 bytes.
Used library Version Path
SPI 1.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SPI
SD 1.2.4 /home/paul/Arduino/libraries/SD
EEPROM 2.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/EEPROM
Entropy /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Entropy
FreqMeasure 1.2 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/FreqMeasure
Wire 1.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Wire
Used platform Version Path
teensy:avr 1.56.1 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1
The real problem is use of /home/paul/Arduino/libraries/SD. I don't understand why this happens well enough to create a narrowly focused issue.
But I do know I can test whether Arduino CLI recognizes its removal.
paul@preston:~ > rm -rf /home/paul/Arduino/libraries/SD
Then when I run the same Arduino CLI command again, it does indeed use the SD library from Teensy's package.
paul@preston:/tmp > /tmp/arduino-cli compile --clean -b teensy:avr:teensy36 ~/teensy/sketch/T40_Tester
Sketch uses 149088 bytes (14%) of program storage space. Maximum is 1048576 bytes.
Global variables use 15712 bytes (5%) of dynamic memory, leaving 246432 bytes for local variables. Maximum is 262144 bytes.
Used library Version Path
SPI 1.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SPI
SD 2.0.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SD
SdFat 2.1.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SdFat
EEPROM 2.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/EEPROM
Entropy /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Entropy
FreqMeasure 1.2 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/FreqMeasure
Wire 1.0 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/Wire
Used platform Version Path
teensy:avr 1.56.1 /home/paul/.arduino15/packages/teensy/hardware/avr/1.56.1
Again, I don't fully understand this problem. But from this test, I'm pretty confident it's not within Arduino CLI. Or if some bug within Arduino CLI was the cause of a conflicting copy of SD appearing at /home/paul/Arduino/libraries/SD, it definitely did respond properly when I deleted that errant copy, and in using only CLI that errant SD library has not magically reappeared. So I can't 100% rule out CLI, but all signs point to CLI is working as expected and the cause of this conflict is most likely something going wrong with library management in the IDE.
Recommend moving this issue back to IDE.
At least from this 1 quick test, Arduino CLI is properly recognizing library removals.
That is because you are using the command line interface. The library removal is recognized on initialization, which happens every time you run an arduino-cli
command. This is why you must use the gRPC interface like the Arduino IDE 2.x does in to reproduce the bug, as shown in the demo I provided above. That is the only way to run multiple compile operations in a single session.
Ah, ok. I'm not familiar with the gRPC interface. Will put it on my bucket list...
Where would be the appropriate place to discuss the problem that something is causing installation of the generic SD library at /home/paul/Arduino/libraries/SD, which overrides the platform specific SD library? Is an issue already open for that problem?
something is causing installation of the generic SD library at /home/paul/Arduino/libraries/SD
This is the intended behavior for the first run of the Arduino IDE: https://github.com/arduino/arduino-ide/pull/663
which overrides the platform specific SD library
This can be fixed by simply adjusting the metadata of the Teensy platform's bundled SD library. @KurtE has already explained it over on the forum thread:
https://forum.pjrc.com/threads/53548-Arduino-CLI-Alpha-Release-Teensy-Support/page9
If you have any questions about that, I will be happy to answer them over there.
Is an issue already open for that problem?
There isn't anything other than the only peripherally related https://github.com/arduino/arduino-ide/issues/798 about the built-in libraries installation. If you have a proposal about refining its behavior separate from the overwrite issue tracked by https://github.com/arduino/arduino-ide/issues/798 (which clearly needs to be fixed), feel free to submit an issue over on the arduino/arduino-ide
repository.
As for the override issue, this was discussed extensively back when Library Manager was first implemented. I'm sure there are multiple issues about it, but I found this one:
If you have any questions about that, I will be happy to answer them over there.
@per1234 - Any chance I can ask for you help (over on the forum) regarding Servo? The fix works on SD, Ethernet, LiquidCrystal, but not Servo.
If you have any questions about that, I will be happy to answer them over there.
@per1234 - Any chance I can ask for you help (over on the forum) regarding Servo? The fix works on SD, Ethernet, LiquidCrystal, but not Servo.
Sorry not sure if best to talk about the servo library here or on the forum thread. https://forum.pjrc.com/threads/53548-Arduino-CLI-Alpha-Release-Teensy-Support?p=307713&viewfull=1#post307713
As I mentioned, it works for SD, and LiquidCrystal and the like, as the libraries installed by the IDE2 (CLI) typically had
architecture=*
So by us putting a more specific one in the teensy boards section works.
However the servo library you install, has architecture defined something like:
architectures=avr, megaavr, sam, samd, nrf5, stm32f4, mbed, mbed_nano, mbed_portenta,mbed_rp2040
So it is unclear how any of these board types could take priority over the one you now install in the
Describe the problem
I installed RC7 on new machine (RPI4 running Ubuntu 22.04). But pretty sure other setups would work the same.
I then installed the Teensy boards, which installs it's own SD library (~/.arduino15/packages/teensy/hardware...)
I then opened a sketch and tried to compile it, I received several strange errors and then remembered it was using the old SD library. So opened up Arduino/libraries folder and deleted SD.
I then tried to build again.
The compiler then failed with error saying that it used: SD.h from the Arduino/libraries/SD folder and not the one from the teensy install. And then gave error that the file did not exist.
To reproduce
Described above.
Expected behavior
Would expect it to see that the folder did not exist and then do the find of it again.
Note: I also tried closing the window and reopen it and it still gave same error.
Arduino IDE version
RC7
Operating system
Linux
Operating system version
Ubuntu 22.04 (RPI 64 bit)
Additional context
Sort of related to my arduino/arduino-ide#798, where I would hope if I had a custom version already installed it would not overwrite.
Also tried to search for duplicate but hard to find search terms.
Issue checklist