search

arduino / Arduino

Arduino IDE 1.x
https://www.arduino.cc/en/software
Other
14.12k stars 7k forks source link

lib manager overrides core libraries included with 3rd party boards #4064

Closed soundanalogous closed 8 years ago

soundanalogous commented 8 years ago

I'm encountering several library compatibility issues lately. I believe these are due in part to the library manager and that if you update a library via the lib manager (the Servo library for example) that is also included by 3rd party boards (via the board manager or 3rd party installers such as the Teensy installer) the library installed by the lib manager will take priority.

A couple of specific examples: Arduino Zero required a newer Servo library than was included in the latest Arduino release. This could be resolved by using the library manager to install Servo v1.1.0. However this version of Servo was incompatible with non SAMD architectures (although this appears to have been resolved in Servo v1.1.1). However 3rd party boards such as Teensy include their own version of the Servo library. Updating Servo via lib manager however will now include that version rather than the version of the lib included with the 3rd party board. Same problem with Intel Galileo and Servo library.

I've noticed the same incompatibility issues across boards with the Ethernet and SPI libraries as well (I believe it's related to this issue: https://github.com/arduino/Arduino/issues/3961).

So there are actually 2 issues here:

  1. lib manager overrides core libraries included with 3rd party and board manager installed boards
  2. core libraries are not compatible across all official arduino boards (when they should be) - due to issues in core library support for new boards (is anyone testing this before release?)

Some possible solutions:

lmihalkovic commented 8 years ago

I've been playing with the IDE code for a couple days. One of the things I was considering was to change the way libraries are stored. The current layout is something like this:

libraries
   libABC
       file1
       file2
   libDEF
       file1
       file2

I was thinking about following the unix model and doing something like:

libraries
   libABC
       1.0.1
           file1
           file2
       1.0.8
           file1
           file2
   libDEF
       2.0.0
           file1
           file2

This would take care of keeping multiple versions of any given library. The problem is that it requires a new mechanism to let the compiler know which one to use. 

{ 
  ' this is a simplified JSON file following the Ninja format
  depends [
    {
      name: libABC
      version: 1.0.8
    }
  ]
}

No matter what, I may need something like that for the other changes I am making. In the following screenshot, the 'references' section contains a list of the system libraries that the sketch uses. This test sketch only requires Wire but the list is still missing filtering code (the 'libraries' section shows the libraries stored locally inside the sketch).

screen shot 2015-11-01 at 11 06 05 pm

https://github.com/arduino/Arduino/issues/4050

NicoHood commented 8 years ago

This makes sense. However I would suggest to integrate git somehow. Or only apply this to extrnal downloaded libraries via download manager. So that you can develop you libraries via git like we are used to.

lmihalkovic commented 8 years ago

Yes, my thought for that multi-versions scheme is only for the lib manager, to address cases where someone might have different sketches that would require different versions of the same lib. Or for the case where some uses a given version, and bumping up presents a risk until things are fully tested. Then the lib manager cen be instructed to download the newer version, but keeping the existing one.

The sketch-local libraries do not follow this proposed layout, because by definition they would be version controlled (could be fun to have a bare minumum GIT/HG/SVN awareness in the ide).

In all cases, displaying the lib version in the outline view (I think it makes sense to christen it like that!?) is just informative. Ideally, switching to a different version in the REFERENCE section could be done with a simple right-click context menu showing all the versions that exist on the machine. Then it is easy to recompile/test. And then go to lib manager and tell it to delete a given version if it is no longer used... Having all the sketches inside a single parent folder means it is even possible to verify which versions are no longer cureently in use. It would also be nice to be able to say 'make a copy of this lib into my sketch-local dependencies (like in my example where I took the ADFNeoPixels lib and started modifying it).

Then the last nice step would be to right click on a sketch-local lib, thereby pushing it descriptor to someplace that the lib-manager can see, to 'Puclish Library'. The publishing process would require validation the descriptor validating the presense of keywd Adding a 'stars' evaluation system to the lib manager (pre-req to be traced back to github account for eg) would then give users feedback about the libs.

I'm a newb to arduino, so maybe some of this is already there and i don't know it.

Future improvement thouhts include:

....time permitting

criticisms most welcome for any/all of it!!!

per1234 commented 8 years ago
  1. lib manager overrides core libraries included with 3rd party and board manager installed boards

Library Manager installs to sketchbook/libraries so the issue is actually that libraries in the sketchbook/libraries folder now take precedence over bundled libraries. This appears to have been caused by https://github.com/arduino/arduino-builder/commit/7f7eb7a8a044b983803cea7682262183e02602ff or maybe https://github.com/arduino/arduino-builder/commit/4e6db8cb185e5e2d393079c6e8e5e5cf97d36e22 the issue doesn't occur in Arduino IDE 1.6.6 2015/10/29 11:41 and the problem occurs in Arduino IDE 1.6.6 2015/10/30 09:58.

The order of priority used to be(1.6.6 2015/10/26 and earlier): bundled library for selected 3rd party board > sketchbook/libraries > Arduino/libraries

Edit: this information is correct but caused by a bug introduced by arduino-builder.

ffissore commented 8 years ago

@NicoHood and @lmihalkovic you're going off topic. Please open a separate issue to discuss your enhancement proposal @soundanalogous and @per1234 what you're experiencing is the correct and expected IDE behaviour. Libraries installed through library manager get installed in SKETCHBOOK/libraries, which is where libraries have the highest priority. This is to give user/developer an easy way to override a core/builtin library, whatever the reason might be (she's developing a new version of it, she needs some customization) As for the incompatibility of Serve with other cores, I'm assigning this issue to @cmaglie who knows more about core stuff and who can deal with it

per1234 commented 8 years ago

~~@ffissore I just edited my previous comment to state this but in case you miss it: The order of priority used to be(1.6.6 2015/10/26 and earlier): bundled library for selected 3rd party board > sketchbook/libraries > Arduino/libraries~~

Edit: this information is correct but caused by a bug introduced by arduino-builder.

ffissore commented 8 years ago

@per1234 that was a bug, fixed with https://github.com/arduino/arduino-builder/issues/38. try with 1.6.5 (which doesn't ship arduino-builder): it works like recent hourly builds (which bundle arduino-builder)

ffissore commented 8 years ago

For the record, correct order is built in libs, parent platform (if any) libs, target platform libs, sketchbook libs

per1234 commented 8 years ago

Ok, so then the issue is that once a library is updated using Library Manager it is installed to sketchbook/libraries so the behavior changes from the 3rd party bundled library taking precedence to the Library Manager installed library taking precedence. The solution would be for Library Manager to install built in libs to the arduino15/libraries folder and for that folder to be higher in priority than built in libs but lower in priority than the rest.

ffissore commented 8 years ago

Could you rewrite my priority list above so that it features your proposed lib?

per1234 commented 8 years ago

In order of lowest to highest priority: built in libs, updated built in libs(arduino15/libraries folder), parent platform (if any) libs, target platform libs, sketchbook libs

This issue is a major problem for 3rd party cores that use bundled libraries that replace built in libraries because the replacement library needs to be only used when the associated board is selected so requiring the user to install the library as is into the sketchbook/libraries folder is not an option. The replacement library names could be changed but that will require the user to change the includes in their sketches back and forth depending on which board they are using. An example of this is Mighty 1284P where Ethernet, Firmata, GSM, SD, and Servo libraries were patched to make them compatible with ATmega1284P.

lmihalkovic commented 8 years ago

For every example where order BUILTIN-LIBS/arduino15/libraries/PLATFORM/TARGET/SKETCHBOOK-LIBS/SKETCH-LIBS will work, someone will have a particular situation that may break. So why not go towards more transparency and make it 1) clear (as in directly visible) AND 2) selectable ?

All it takes is to have a dependencies.txt file with

{ 
  ' this is a simplified JSON file following the Ninja format
  depends [
    {
      name: libABC
      version: 1.0.8              // in the short term this can be ignored if you don't want to allow side by side versions to exist
      source: XXXXXXX
    }
  ]
}

The value of XXXXXXXX could be picked from a java Enum that would normalize once and for all the name of these different locations. For some sort of symmetry it could be something like:

public enum SketchCodeLocation {
   SKETCH  // I am the only one using this for now
   , SKETCHBOOK
   , LIBMANAGER
   , ......

I did that this morning to keep track of the known file types for the editor:

package processing.app;
public enum SketchCodeType {
  SKETCH(true, true, ".ino", ".pde")
  , CODE(true, true, ".cpp", ".c")
  , HEADER(true, true, ".hpp", ".h", ".hh")
  , ASSEMBLY(true, true, ".s")
  , README(false, false, "README.md", "readme.md")
  , LIB_KEYWORDS(true, false, "keywords.txt")
  , LIB_PROPERTIES(true, false, "library.properties")
  , UNKNOWN(false, false)
  ;

Normalizing the location names means that you can even store the link priority heuristic as a config param somewhere in the IDE... which means it is easy to alter it, and it is easy to experiment/change.

Ok, so then the issue is that once a library is updated using Library Manager it is installed to sketchbook/libraries so the behavior changes from the 3rd party bundled library taking precedence to the Library Manager installed library taking precedence. The solution would be for Library Manager to install built in libs to the arduino15/libraries folder and for that folder to be higher in priority than built in libs but lower in priority than the rest.

This would go away with one more bit on info the dependencies.txt

{ 
  ' this is a simplified JSON file following the Ninja format 
  depends [
    {
      name: libABC
      version: 1.0.8              // in the short term this can be ignored if you don't want to allow side by side versions to exist
      source: XXXXXXX
    }
  ]
  resolution {
     order: [BUILTIN, PLATFORM_PARENT, PLATFORM_TARGET, LIBMANAGER, SKETCHBOOK, SKETCH]
    conflicts: ERROR // or WARN  
  }  
}
soundanalogous commented 8 years ago

One simple fix (although not the most ideal) would be to add an "uninstall" option in the library manager. This would remove an installed library from the user's Arduino/libraries/ folder and clear any cached version of that library. I've found that manually removing the library from Arduino/libraries/ does not work as it appears to be cached. Strangely it will even effect Arduino v1.0.6 (in that the IDE will pick up the cached version). An "uninstall" option would clear whatever is cached for that lib.

Ideally though I think it may make sense for patched core libs that are bundled with 3rd party boards to have higher priority than libs in the users Arduino/libraries/ directory. So if a user updates Servo via the library manager, then tries to compile a sketch for a 3rd party board (Teensy for example) that uses Servo, the correct patched Servo library will be installed.

Or perhaps there is a way to use the architectures attribute from the library.properties file. If a library has been updated via lib manager and the architecture listed does not match that of the board a sketch is being compiles for, then give priority to the lib that matches the board architecture. However I'm not sure if the architectures attribute is very reliable in this respect.

per1234 commented 8 years ago

One simple fix (although not the most ideal) would be to add an "uninstall" option in the library manager.

I don't want to remove the built in libraries, only to override then. If you did a full uninstall of a built in library then any sketch that included it would cease to work for any board that didn't have a platform specific version of that library. So I think you are referring to a roll back of the version to the one in Arduino/libraries. This is already possible but the user would need to know what the original version number was and then they're using an outdated version of the built in library when using a board without a platform specific version of the library. The other problem with that is in 1.6.6 you get update notifications over and over again until you upgrade to the latest version. If you want some other ideas for workarounds I put together a list here.

manually removing the library from Arduino/libraries/ does not work as it appears to be cached.

Definitely not cached for me. Arduino builder does skip rebuilds of libraries when there have been no modifications since the last build but even then it still fails to compile if I manually remove the library between compiles.

it may make sense for patched core libs that are bundled with 3rd party boards to have higher priority than libs in the users Arduino/libraries/ directory.

It already does, this is the current order of precedence from lowest priority to highest: built in libs, parent platform (if any) libs, target platform libs, sketchbook libs As you see built in libs(these are the libs in Arduino/libraries) have lowest priority.

use the architectures attribute from the library.properties file

That still doesn't help for a case such as Mighty 1284P where it's adding support for an AVR that isn't supported by the Arduino core.

soundanalogous commented 8 years ago

Each time in my last comment that I said "user Arduino/library/" I meant "sketchbook libs".

When you install a library via library manager it is installed into sketchbook libs. This overrides libraries included by any 3rd party board packages in their core libraries directory (either via the boards manager or via other installers).

It also appears that if you remove a library from sketchbook libs (after installing it via lib manager), the IDE (or arduino builder or whatever handles this) still somehow does not target the patch lib included with the 3rd party board (I wonder if this could be because a 3rd party patched lib may have the same version number as a core lib even though they are different because of the arch specific patch). Thus my suggestion for adding an "uninstall option" that will do whatever cleanup is necessary. Then the user compiles for the 3rd party board and then if they switch back to a 1st party board they can reinstall the library. Not ideal I know, but better slightly than the current situation.

Best option IMO is still to change the priority order to make 3rd party core libs highest. Developers of 3rd party cores would need to be mindful of this in how they choose which libraries they need to patch since there would no longer be an easy way for a user to override them (unless someone has a clever idea for that).

soundanalogous commented 8 years ago

So my priority list would be (from lowest to highest): built in libs, parent platform (if any) libs, sketchbook libs, target platform libs (3rd party) where "target platform libs" for any 1st party boards would still be sketchbook libs

lmihalkovic commented 8 years ago

it is this feeling of lack-of-control/lack-of-visibility that prompted me to add libraries directly inside the sketch and the outline view. The builder is cleverly crafted, but it also seems to make some seemingly simple things more difficult than they ought to be.

PaulStoffregen commented 8 years ago

Well, 1.6.6 just released, so we're going to have to live with whatever it does for a while, it seems.

soundanalogous commented 8 years ago

Looks like 1.6.6 will fix the Servo issue since it includes v1.1.1 (that works for Zero and all other 1st party boards). That way it's no longer necessary to update Servo via the lib manager (at least for now). It does however force anyone who uses a Zero to upgrade to Arduino 1.6.6 if they are using the Servo lib (if they want to avoid conflicts with 3rd party boards that bundle their own arch-specific version of Servo). May have been better to include a samd patch of Servo with the Zero core/libraries.

cmaglie commented 8 years ago

The builder on 1.6.6 doesn't change anything: it behaves exactly as in 1.6.5 and any older version since 1.5.x (and maybe 1.0.x too?). The order of priority (from the lower to the highest) is

The rationale is that users should have the ability to override any selection made by the IDE, so the sketchbook is at the highest priority.

It does however force anyone who uses a Zero to upgrade to Arduino 1.6.6 if they are using the Servo lib (if they want to avoid conflicts with 3rd party boards that bundle their own arch-specific version of Servo).

The issue has nothing to do with the Zero, the same problem will appear every time the Servo library is upgraded and the user upgrades the library via library manager, moving it from the "bundled" folder to the "sketchbook" folder.

Maybe the correct question is if the Servo library should be a "core" library? SPI and Wire are core libraries, there is no doubt about that, but Servo is quite borderline. What's the "requirement" that makes a library a "core library"? Don't we have the same problem for all the bundled libraries?

it is this feeling of lack-of-control/lack-of-visibility that prompted me to add libraries directly inside the sketch and the outline view. The builder is cleverly crafted, but it also seems to make some seemingly simple things more difficult than they ought to be.

Putting yet another copy of a library inside the sketch doesn't seem to simplify things IMHO because it increases the number of folders to look into from 4 to 5.

For the record: at the times, maybe in an excess of ingenuity, I proposed to keep only the sketchbook libraries folder and drop bundled and cores folders, but this solution received very negative feedback from the community so we had to step back and stick to the current system. This just to say that the current system is a delicate balance of many requirements, not something that has been vertically imposed just to complicate things.

per1234 commented 8 years ago

Does anyone have an opinion on my previously proposed solution:

The solution would be for Library Manager to install built in libs to the arduino15/libraries folder and for that folder to be higher in priority than built in libs but lower in priority than the rest. In order of lowest to highest priority: built in libs, updated built in libs(arduino15/libraries folder), parent platform (if any) libs, target platform libs, sketchbook libs

I've never liked the way that Library Manager "pollutes" my sketchbook/libraries folder with updates of built in libraries as I expect to see only 3rd party libraries in that location.

soundanalogous commented 8 years ago

The issue has nothing to do with the Zero

Yes in terms of the board itself, etc but it's still tangentially related. The version of Servo that was included with Arduino 1.6.5 did not yet support SAMD boards. Therefore in order to compile a library for Zero that included Servo.h, you had to update Servo to the latest version via the library manager. However doing this made sketches using Servo no longer compile on 3rd party boards that bundle their own version of servo.

If you want to recreate to understand the frustration do this:

  1. Make sure you don't already have a newer version (1.1.0+) of Servo installed in you sketchbook lib. If so, remove it temporarily.
  2. Install a clean version of Arduino 1.6.5
  3. Install Zero via the boards manager
  4. Install teensyduino or another 3rd party board package that includes Servo in its core.
  5. Select Zero and try to compile a sketch that includes the servo library. It will fail.
  6. At this point you could do what I did and use the library manager to get an updated version of Servo.
  7. Try to compile again, it will succeed
  8. Select a teensy board and try to compile the same sketch, it will fail on Servo.
  9. Try to figure out how to get back to a state where you can compile the sketch on Teensy.

However this should now be resolved in Arduino 1.6.6 (I haven't had time to verify yet). Overall, having libraries installed in the sketchbook lib has been helpful, especially in ease of updating libraries. However, it would be nice to figure out a way to fix this library compatibility issue at some point to avoid having to install and reinstall different library versions when switching between boards.

soundanalogous commented 8 years ago

Maybe the correct question is if the Servo library should be a "core" library? SPI and Wire are core libraries, there is no doubt about that, but Servo is quite borderline. What's the "requirement" that makes a library a "core library"? Don't we have the same problem for all the bundled libraries?

Yes this will be an issue (in terms of compatibility) for all bundled libraries. While removing Servo from the core may make sense semantically, it has been included for so long that it likely cause more trouble to remove it than keep it... at least until Arduino 2.0.

damellis commented 8 years ago

On Wed, Nov 4, 2015 at 3:32 AM, Cristian Maglie notifications@github.com wrote:

The issue has nothing to do with the Zero, the same problem will appear every time the Servo library is upgraded and the user upgrades the library via library manager, moving it from the "bundled" folder to the "sketchbook" folder.

To me, this is the confusing part. Why does updating a bundled library move it to the sketchbook? It seems to me that they should update in place (i.e. while remaining "bundled"). I know there's a lot of complicating factors here and that this has been discussed before, but that still seems like the solution, at least to me. What do you think?

soundanalogous commented 8 years ago

I think David's suggestion is great and would solve these compatibility issues. Update bundled libraries in place and update non-bundled libraries to the sketchbook lib directory. Then if someone want's to manually override a bundled library (and they understand the potential consequences) they can manually install the library in their sketchbooks lib and it will have highest priority.

per1234 commented 8 years ago

cmaglie:

Bundled libraries (the folder is ${IDE_PATH}/libraries)

There seems to be some differences in terminology here, ffisore was referring to those libraries as "built in libs"

damellis:

It seems to me that they should update in place (i.e. while remaining "bundled")

I'm happy with that but I think one of the reasons why Boards Manager installs to the arduino15 folder is so that if multiple users use the same installation of Arduino IDE then one user's actions won't affect the other users. That's the reason why I suggested installing to arduino15 because it's already the established way of doing this sort of thing.

damellis commented 8 years ago

I agree, installing to the Arduino preferences folder (where downloaded hardware packages and tools go) would be better than updating the library that's actually inside of the Arduino application itself. I just meant that the libraries should go in one of these places rather than the sketchbook folder.

On Wed, Nov 4, 2015 at 11:42 AM, per1234 notifications@github.com wrote:

cmaglie:

Bundled libraries (the folder is ${IDE_PATH}/libraries)

There seems to be some differences in terminology here, ffisore was referring to those libraries as "built in libs"

damellis:

It seems to me that they should update in place (i.e. while remaining "bundled")

I'm happy with that but I think one of the reasons why Boards Manager installs to the arduino15 folder is so that if multiple users use the same installation of Arduino IDE then one user's actions won't affect the other users. That's the reason why I suggested installing to arduino15 because it's already the established way of doing this sort of thing.

— Reply to this email directly or view it on GitHub https://github.com/arduino/Arduino/issues/4064#issuecomment-153842138.

soundanalogous commented 8 years ago

The prompting to updating libraries makes this issue worse. Every once in a while (when using Arduino 1.6.6) I get a prompt to update libraries. One of these libraries is the Ethernet library. If any user updates Ethernet via the library manager, they will no longer be able compile sketches for boards such as Galileo, Edison, etc that use Ethernet since the linker will grab the AVR/SAM/SAMD version of Ethernet from the sketchbook libs folder.

per1234 commented 8 years ago

It appears that this issue also affects Galileo, Edison and the Arduino 101.

bperrybap commented 8 years ago

Here is a summary of my views: There is a well defined priority of how library locations are searched/used: (from lowest to highest priority)

and so when a library for a component (IDE, referenced core, core, etc...) is updated it should not stomp on or trump another library that is above its priority. For example, if updating one of the IDE specific "bundled libraries", (the lowest of priorities) that update should not trump any other library that might exist in one of the other 3 higher priority locations.

The other other thing that I mentioned below, is that it might be useful to have 5th priority location which would be a a "libraries" directly under the sketch directory itself.

More below

It seems bad to allow the IDE to ever update a library in a way the trumps a higher priority library with its own IDE specific "bundled" libraries. This is particularly true when dumping IDE "bundled" library updates into the users {sketchbook}/libraries area.

Keep in mind that for the user, the {sketchbook}/libraries area is common to ALL IDEs and all cores within those IDEs installed on the machine, not just the IDE that is currently trying to do an update. For example, I have 18 different IDEs that I have on a machine that I test against and some of them are not even arduino.cc IDEs. They all share the same {sketchbook}/libraries area which is great for regression testing of a library.

This is an extreme example but other people may have more than a single IDE version installed.

There are also cases where a user may have a personal version of a library or a core specific version that they want to always override the bundled library and don't want an updated version of the IDE bundled library for a specific version of an IDE to stomp on their library.

If the IDE is going to be updating itself, then it should really update itself and do it in a way that will not impact other IDEs or even other components like core libraries.

I think it is bad practice to have the IDE putting its IDE specific stuff down under the users area. It should be updating the actual IDE files where the IDE is installed. There is a potential issue of permissions since a user that is wanting an update may not have admin rights or write access to the area where the IDE is stored.

My guess is that this is quite rare since most people are running the IDE on their person Windows PC machines and have admin rights for the machine. And for the cases where you do have multiple users sharing an install area, then the admins of that system and install area probably would prefer that users don't muck around with the original install area and also don't clutter up their home directories with IDE specific updates that could be added to the original install area for everyone to use.

So I think that the best way to resolve the situation where the user can't update the area where the IDE is installed is simply to fail the update and tell the user to see their administrator to handle the update. (this kind of stuff is the norm for many other types of s/w) For multi-user environments like schools, this can ensure that all the students are using the same versions of the s/w as well as offering an easy way to update the s/w for all the students.

The way it is now where IDE specific bundled library updates and added tools are loaded under the users area seems a bit silly.

One exception that I see as being valuable for certain projects and for temporary situations, would be to add support for a new highest priority library location. This would be higher priority than the {sketchbook}/libraries area. It would be a "libraries" directory under the currently building sketch directory. This structure would allow users to install libraries specific to that sketch. It can be useful for project/product bundling, special testing, as well as for temporary work arounds until an admin could be found to properly update the IDE bundled libraries.

lmihalkovic commented 8 years ago 
To me, this is the confusing part. Why does updating a bundled library
move it to the sketchbook? It seems to me that they should update in place

@damellis there is couple reasons why it probably should not happen that way. 1) the executable folder on a mac should not be messed with, so replacing a file is not a good idea (aplle might even ban it in the future). 2) keeping the original material means it should always be possible for a user to 'restore' their ide without having to delete it. Just a thought.

bperrybap commented 8 years ago

dave, About 1), then how do applications ever update on the mac? At some point there must be an apple official way of updating the executable of application. In my view, IDE updates of an IDE bundled library is no different than updating an application. The IDE is essentially updating itself. So since it is updating itself why not really update itself instead of stomping on some other area which has the potential to clobber something else?

About 2), That is a hypothetical at this point. However, the current way the IDE is updating bundled libraries by doing it outside its own area in a area that is used for sometime else, is corrupting things in certain situations. And again, I come back to an application updating itself. If there is a desire to roll back, then do it correctly and allow a real roll back, that would include ways to roll back to previous versions. i.e. roll back from 1.6.7 to 1.6.5 etc...

In the broader picture, the issue the IDE is having is that in the past the IDE was a just a gigantic blob that was essentially an atomic static install and the only way to update it was to install a new "blob". Now that this is no longer the case, it requires a different way of handling things since each update no matter how small is essentially a new version of the IDE. This problem is that the IDE attempted to solve the issue of incremental updates by using existing library override mechanisms. This was not the appropriate solution. To handle things like updating bundled libraries, The IDE needs to have its own library area that it can update that trumps the IDE's bundled libraries that came with the IDE that is separate and unique from the areas the user uses to trump the IDE bundled library area.

i.e. the user should always be able to trump an IDE bundled library without having his version of the library stomped on by the IDE doing an update. If the IDE ever stomps on a user library, that is just plain wrong.

To handle this, if there is a desire to not do the update in place, then the IDE needs to have another library directory in the library path that in sits in priority just above the IDE bundled libraries but below the end user libraries.

facchinm commented 8 years ago

I pushed a proposal to solve the issue here https://github.com/arduino/arduino-builder/pull/123, could you take a look if it can work for all the corner cases exposed in this thread? @soundanalogous @bperrybap @per1234 @damellis @cmaglie @PaulStoffregen @NicoHood

per1234 commented 8 years ago

In the discussion on https://github.com/arduino/arduino-builder/pull/123 bperrybap suggested that the include priorities should be documented. This is my attempt at documenting the current priorities:

Arduino IDE(1.6.8 2016/03/07 08:33) include priorities(high to low)

Location Folder Name Match Architecture Match
sketch yes/no NA
core NA NA
variant NA NA
highest previous include folder yes/no yes/no
compiler NA NA
sketchbook yes yes
core libraries yes yes
bundled yes yes
sketchbook yes no
core libraries yes no
bundled yes no
sketchbook no yes
core libraries no yes
bundled no yes
sketchbook no no
core libraries no no
bundled no no

Definitions:

I suggest that any proposal for a solution to this issue that involves a change in include priorities present an updated include priorities table and use the defined location terms.

Any corrections or suggestions for the best way to present this information are welcome and I will edit this comment accordingly.

I propose to add this information to https://github.com/arduino/Arduino/wiki/Build-Process once finalized.

markdown source for the table:

Location | Folder Name Match | Architecture Match
---|---|---|---
sketch | yes/no | NA
core | NA | NA
variant | NA | NA
highest previous include folder | yes/no | yes/no
compiler | NA | NA
sketchbook | yes | yes
core libraries | yes | yes
bundled | yes | yes
sketchbook | yes | no
core libraries | yes | no
bundled | yes | no
sketchbook | no | yes
core libraries | no | yes
bundled | no | yes
sketchbook | no | no
core libraries | no | no
bundled | no | no

To do:

bperrybap commented 8 years ago

@per1234 I think for the most part we are in agreement that library updates for each layer need to be kept separate and not stomp on a library in an other layers. The issue is that the current library manager update methodology is too simplistic and is unworkable, since it places all updates in the same single location. In my view this is not a library prioritization issue, but rather how updates are handled and where they are installed. If library updates were done in place then much of this mess goes away, but regardless of whether they are done in place or in other locations that logically extend the layer, it requires that the library manger be able to deal a repository for each individual library in each layer. Until there is a way for each individual library to update itself, there will always be issues. I wrote more about this in the other thread: https://github.com/arduino/arduino-builder/pull/123#issuecomment-193988198

One area that would be good to refine is should each IDE have its own update area for its bundled libraries, or should there be an area that is shared between all IDEs. I would prefer that each IDE do its own bundled updates so that one IDE does not ever update or get affected by another IDEs updates in the case of multiple IDEs installed. Essentially logically treating all library updates as "in place" even if the actual update is not physically stored in the same location as the original library. I think that sharing bundled updates across IDEs will eventually run into issues and not sharing them allows quick and easy regression testing since each IDE is separate.