search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.86k stars 790 forks source link

Sharing dependencies across environments #4451

Open robsonos opened 1 year ago

robsonos commented 1 year ago

What kind of issue is this?

Configuration

Operating system: Windows 11 Pro

PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.4

Description of problem

Hi, I was wondering if it is possible to create a feature to allow sharing dependencies across environments. I have a project with several dependencies with the same version and they end up getting downloaded multiple times.

Steps to Reproduce

pio run

Actual Results

The above command will create:

.pio
'-libdeps
  '-env1
     `-ArduinoJson
  '-env2
     `-ArduinoJson

Expected Results

Something like:

.pio
'-libdeps
  '-env-shared
     `-ArduinoJson

If problems with PlatformIO Build System:

The content of platformio.ini:

[env]
platform = nordicnrf52
board = adafruit_feather_nrf52840
framework = arduino
lib_deps = 
    bblanchon/ArduinoJson@^6.19.4

[env:env1]

[env:env2]

Source file to reproduce issue:

#include <Arduino.h>

void setup() {
  // put your setup code here, to run once:
}

void loop() {
  // put your main code here, to run repeatedly:
}

Additional info

ivankravets commented 1 year ago

with several dependencies with the same version and they end up getting downloaded multiple times.

PlatformIO does not download dependencies multiple times, it uses a cache for HTTP and binary requests. So, don't worry about this.

Regarding common dependencies. You can put them in the lib folder. Libraries located here are sharable between environments.

robsonos commented 1 year ago

Hi @ivankravets,

I see. However, PlatformIO doesn't seem to cache dependencies from git (private) repositories (I apologise for not mentioning that before), which makes sense but it drastically slows down continuous integration. I am aware of the lib folder alternative, but I still wanna treat the libraries as dependencies as they are versioned per release (but not per environment).

Would it be able to achieve dependency sharing across environments with Advanced Scripting?

ivankravets commented 1 year ago

Yes, https://docs.platformio.org/en/latest/scripting/index.html is the right solution. Just use the PRE script and pre-install dependencies to the lib folder. Also, you can keep declarations in the config and re-use them, see example https://docs.platformio.org/en/latest/scripting/examples/platformio_ini_custom_options.html

Another idea is to use the LibraryPackageManager API directly but I don't know how you are good in Python. See part of the code https://github.com/platformio/platformio-core/blob/develop/platformio/package/commands/install.py#L220

mcspr commented 1 year ago

^ I believe something like this was previously mentioned after lib_extra_dirs stopped working with lib_deps in #4206, so in our (espurna) case the only real option now is to manually install everything through the API

robsonos commented 1 year ago

Thanks, guys. I was considering using LibraryPackageManager though was not sure how to approach this in extra_scripts as my understanding is that extra_scripts run after the Library Manager. What you sent looks promising @mcspr.

robsonos commented 1 year ago

Hi guys. I ended up going with something like this:

platformio.ini:

[env]
...
extra_scripts = pre:pre_script.py
lib_extra_dirs = .pio/libdeps/shared/
shared_lib_dir = .pio/libdeps/shared/
shared_lib_deps =
  https://github.com/...
  https://github.com/...
  https://github.com/...

pre_script.py:

from SCons.Script import ARGUMENTS
import subprocess
from os import makedirs
from os.path import isdir
Import("env")

def install_shared_dependencies(env, verbose):
    # Get shared_lib_dir
    shared_lib_dir = env.GetProjectOption('shared_lib_dir')

    # Create shared_lib_dirif it does not exist
    if (not isdir(shared_lib_dir)):
        makedirs(shared_lib_dir)

    # Get lib_deps
    config = env.GetProjectConfig()
    raw_lib_deps = env.GetProjectOption('shared_lib_deps')
    lib_deps = config.parse_multi_values(raw_lib_deps)

    # Build dependency installation command
    cmd = [env.subst("$PYTHONEXE"), "-m", "platformio", "pkg", "install"]
    cmd.extend(["--storage-dir", shared_lib_dir])

    # Add verbose to command
    if int(verbose) < 1:
        cmd.append("-s")

    # Add dependencies to command
    for lib in lib_deps:
        cmd.append("-l")
        cmd.append(lib)

    # Run command
    subprocess.check_call(cmd)

# Get verbose level
VERBOSE = ARGUMENTS.get("PIOVERBOSE", 0)

# Intall dependencies listed in env shared_lib_deps to shared_lib_dir
install_shared_dependencies(env, VERBOSE)

I am happy for this issue to be closed but I still see value in having a feature in platformio to do this.

Cheers.

shennan commented 1 week ago

PlatformIO does seem to want to install multiple versions per environment. With a platformio.ini like this:

[env:Build1]
platform = atmelsam
board = adafruit_itsybitsy_m4
framework = arduino
src_filter = +<Build1>
lib_deps =
  bblanchon/ArduinoJson @ ^6.21.0

[env:Build2]
platform = atmelsam
board = adafruit_itsybitsy_m4
framework = arduino
src_filter = +<Build2>
lib_deps =
  adafruit/Adafruit NeoPixel@^1.12.3
  bblanchon/ArduinoJson @ ^6.21.0
build_flags =
  -D USE_TINYUSB  ; Enable TinyUSB stack

And directories like this:

src/
├─ Build1/
│  ├─ main.ccp
├─ Build2/
│  ├─ main.cpp

I get an error about a non-shared NeoPixel dependency in src/Build2/main.cpp:

include errors detected. Please update your includePath. Squiggles are disabled for this translation unit (/path/to.my/project/src/RecyclingSensor/main.cpp).C/C++(1696)

cannot open source file "Adafruit_NeoPixel.h"C/C++(1696)

This is after I pio lib install, and looking at the .vscode/c_cpp_properties.json I see that only includePaths for Build1 are explicitly included. As soon as I add "path/to/my/project/.pio/libdeps/Build2/Adafruit NeoPixel" to the includePaths, the error goes away.

But c_cpp_properties.json is an auto-generated file that I don't feel comfortable editing. Shouldn't installing dependencies from my platformio.ini simply add these paths automatically?