search

arduino / Arduino

Arduino IDE 1.x
https://www.arduino.cc/en/software
Other
14.11k stars 7k forks source link

Library manager: Library dependencies #5795

Closed Avamander closed 4 years ago

Avamander commented 7 years ago

Is there a possibility that in the future versions of Arduino IDE library could define it's dependencies thus making sure it will work.

Something like this:

  1. Library has a line something similar to requires=<other library's name>:<optional minimum version> and/or requiresIDE=<version> in it's library.properties
  2. User presses download but it does not have a required library installed, there is one installed but without library.properties or the library is outdated (with requiresIDE the IDE's version incompatibility is also warned about)
  3. User is prompted to download or update all the required libraries

This would definetly for example improve how people use libraries like RF24, RF24Network and RF24Mesh (I am sure there are more) making sure both the library and it's dependencies are up-to-date when any of those are downloaded.

carlosperate commented 7 years ago

It would also be quite useful if a library could have an Arduino core dependency (ideally a list with all compatible Arduino cores), as some libraries might be processor-specific.

scls19fr commented 5 years ago

Coming from https://github.com/arduino/Arduino/issues/7567#issuecomment-451193270

What about using a dependency manager?

There's currently:

name url repository
Buckaroo http://www.buckaroo.pm/ https://github.com/LoopPerfect/buckaroo
Conan https://conan.io/ https://github.com/conan-io/conan
Hunter https://github.com/ruslo/hunter
cpm http://www.cpm.rocks/ https://github.com/iauns/cpm
cget https://cget.readthedocs.io/ https://github.com/pfultz2/cget
sw (previously known as cppan) https://software-network.org/ https://github.com/SoftwareNetwork/sw
vcpkg https://github.com/Microsoft/vcpkg
conda https://conda.io/ https://github.com/conda/conda
build2 https://build2.org https://github.com/build2/build2

and probably more.

They are more or less coupled to a build system.

See also https://www.reddit.com/r/cpp/comments/8t0ufu/what_is_a_good_package_manager_for_c/ http://pfultz2.com/blog/2017/10/27/universal-package-manager/

scls19fr commented 5 years ago

Library has a line something similar to requires=<other library's name>: and/or requiresIDE= in it's library.properties

@Avamander specifying optionally minimum version may not be enough because if API of a library have some breaking changes you will also need to restrict library version with a maximum version to use latest previous major version

We could need to specify for example:

ArduinoJson<6.0.0,>=5.0.0

But I'm wondering if all registered Arduino libraries uses semantic versioning. Because we need to be able to order version numbering without error / uncertainty.

Related https://github.com/arduino/arduino-cli/issues/105

per1234 commented 5 years ago

I'm wondering if all registered Arduino libraries uses semantic versioning.

They don't. There is a required version format, which includes semver, but also allows non-semver compliant version numbers.

From https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format

version - version of the library. Version should be semver compliant. 1.2.0 is correct; 1.2 is accepted; r5, 003, 1.1c are invalid

I think this is a programmatic definition of the requirement: https://github.com/bugst/relaxed-semver

njlr commented 5 years ago

Hi! :wave:

One of the authors of Buckaroo here.

I had a look at the library.json format and I think it will be feasible to add native support for this in Buckaroo. From the looks of things, these libraries will build very cleanly! :+1:

However, I could not see any dependency information in the library files that I found.

Is the dependency information encoded anywhere?

I am looking for something like:

This library depends on package X, which can be found at github.com/a/x

scls19fr commented 5 years ago

Thanks @njlr for coming here to help us.

I wonder if this kind of information shouldn't be available (first) in library.properties (or somewhere else). It's currently not the case.

I think this file should contain dependencies for compiling a sketch ie you need for example LibraryXYZ latest registered version.

But there is differents cases to manage.

Is LibraryXYZ a "registered" library or not?

If it's a registered library library_index.json could (partially) help (to know where to download it)

Version number restriction could also be required for installing some libraries (because sometimes, installing latest version of a library doesn't work because API of a library changed)

Being able to install an (unregistered) library from a Git repository at a given commit, or a given branch should also be considered.

I opened some days ago issues to manage this with arduino-cli

but the main problem here is to first define new data file to define dependencies.

Is adding new fields to library.properties a good idea? should we have a separate file (a YAML or a JSON file) to manage only dependencies?

I don't really know where the Arduino team want to go with this but from my perspective I will use a separate human readable file for this (and I think that YAML file is great for this).

We should be aware that some examples of a library could need more dependencies that a library itself so we should have a mechanism to override this config file for a given directory. (this problem is quite comparable to https://github.com/ianfixes/arduino_ci/issues/109 )

njlr commented 5 years ago

Hi @scls19fr thanks for the response.

If it's a registered library library.json could (partially) help (to know where to download it)

By "registered" I assume you mean registered on https://www.arduinolibraries.info?

Being able to install an (unregistered) library from a Git repository at a given commit, or a given branch should also be considered.

Buckaroo already works directly on GitHub (or any Git) repositories, so there are no issues there :+1:

but the main problem here is to first define new data file to define dependencies.

Is adding new fields to library.properties a good idea? should we have a separate file (a YAML or a JSON file) to manage only dependencies?

Buckaroo has a TOML file (similar benefits to YAML) to define this.

Here is what it might look like:

[[dependency]]
package = "github.com/bblanchon/ArduinoJson"
version = "6.8.0"

[[dependency]]
package = "github.com/arduino-libraries/ArduinoHttpClient"
version = "revision=b6424e430db57fa0c67f82333e80d58b1521a97b"

Buckaroo supports sem-ver ranges, branches, tags and even commits.

It will examine each dependency it finds and recursively fetch its dependencies, assuming there is a TOML file.

The big question is how to integrate this with the Arduino workflow. Currently, Buckaroo puts dependencies into ./buckaroo and wires them together using Buck (Buck is a build system similar to Make, but more powerful). We could add Arduino as an additional back-end option.

I saw in the IDE that packages are put into ./libraries. However, using ino at the command-line, libraries seem to go inside ./lib.

scls19fr commented 5 years ago

By registered I mean that is available in library_index.json (sorry about my mistake of library.json ... library.json is a PlatformIO config file)

ArduinoJson https://github.com/bblanchon/ArduinoJson is a quite large open source project

njlr commented 5 years ago

So library.properties would be the Arduino package file?

For example: https://github.com/bblanchon/ArduinoJson/blob/0685a36f0e51ba71b96c441697e0432ed2a6b176/library.properties

I see that the file does not contain any header or source-file lists. Does this mean that there is a convention about the layout of the files?

scls19fr commented 5 years ago

This article https://www.arduino.cc/en/guide/libraries describes how libraries are installed and this one deals with creating Arduino lib https://www.arduino.cc/en/hacking/libraries

You can also have a look at some wiki articles such as https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ

Code is in a src directory Examples are in an examples directory and are automatically shown in IDE Unit tests can be in a test directory (when using arduino_ci but that's a quite new project currently not widely used).

When a user wants to build a project he must ensure (manually) that all libraries are installed correctly (and with the correct version). I personally think the current situation can be problematic as the concept of environment doesn't exist https://github.com/arduino/arduino-cli/issues/108 (but that's just my own opinion).

sandeepmistry commented 4 years ago

This was added in https://github.com/arduino/Arduino/pull/8600.

ladyada commented 4 years ago

yaaaaaaaaaaaay

scls19fr commented 4 years ago

Good news but I'm wondering if dependencies version are taken into account.

I only see

      "requires": [
        { "name": "WiFi101" },
        { "name": "ArduinoSound" }
      ],

nothing about a possible version restriction of a library (see https://github.com/arduino/arduino-cli/issues/105 )

cmaglie commented 4 years ago

Good news but I'm wondering if dependencies version are taken into account.

This is being implemented, the general dependency-resolution (with version restrictions) requires a much more complex backtracking algorithm, for now we decided to deploy the "light" version of lib dependecy (that should cover 99% of the cases anyway).

ArminJo commented 4 years ago

Great work, a feature I waited for so long...

What must I do to describe the library dependencies in my library.properties file? I cannot find any reference in https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification