On multi environment project all libraries are included/Compiled

[Pedroalbuquerque]

Configuration

PIO 3.5 with ATOM on MAC 10.12.6

Description of problem

When defining multiple environments on a project as [env:nodemcuv2] and [env:esp32]

I need to include different libraries for each of the environment, so I use lib_deps var to define for each environment the needed libraries.

The problem I face is that if I compile a single environment using default_env var, there is not problem, the includes work fine for each one of the environment,

If I do not specify de env to use, both are built and I need to explicitly exclude some library on ESP8266 env that are included in esp32.

What I believe is happening is that the library dependency folder is common for both environements so ALL included libraries for both environment are copied to that .piolibdeps folder so I need in each environment exclude the libs included in the other environment that do not apply.

Some libraries are not compatible with both environments raising error during compilation, even if they are no #include in the code.

A second effect (I believe may come from same root cause is that in case I use compile directive that exclude some lib #include, the lib is still compiled and not being compatible with the platform, will raise error.

Steps to Reproduce

1. 2. 3.

Actual Results

with code used bellow the second effect is visible and I do not have currently code example prepared fro first symptom. So result for second is that error code is generated when env_default=moteinomega because TFT_eSPI lib is not compatible with moteinoMega

but copiles ok with env_default=nodemcuv2

Indexing .pioenvs/moteinomega/libe97/libdisplay.a Archiving .pioenvs/moteinomega/lib4e9/libAdafruit_GFX_Library.a /Users/pedroalbuquerque/documents/googledrive/projects/libraries/TFT_eSPI/TFT_eSPI.cpp:4147:2 5: fatal error: soc/spi_reg.h: No such file or directory

include "soc/spi_reg.h"

^ compilation terminated. Archiving .pioenvs/moteinomega/lib135/libAdafruit_PCD8544.a *** [.pioenvs/moteinomega/lib7bb/TFT_eSPI/TFT_eSPI.cpp.o] Error 1 Indexing .pioenvs/moteinomega/lib4e9/libAdafruit_GFX_Library.a

Indexing .pioenvs/moteinomega/lib135/libAdafruit_PCD8544.a ========================== [ERROR] Took 2.90 seconds ==========================

Expected Results

I would expect that due to the fact that macro ESP32 is not defined, everything after #if defined(ESP32) would not be compiled avoiding the error. Better said, the library referenced on an #include after that #if would not be compiled. Actually I believe the include does not take place, and the object instance using that lib is not compiled, but the lib itself is compiled anyway.

When using arduino IDE to compile both with platform MoteinoMega as well with nodemcuv2, it compiles ok with no error.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/page/projectconf.html

[platformio]
env_default=moteinomega
src_dir=./
lib_dir=~/documents/googledrive/projects/libraries
lib_extra_dirs =/Users/pedroalbuquerque/Documents/GoogleDrive/Projects/Lib_dev

[common]
build_flags = -g
    -DECHO_PORT=Serial
    -DDEBUG_PORT=Serial
    -DMQTT_MAX_PACKET_SIZE=400
    -Wcomment
lib_ignore = ESP32WebServer
    RFM69_OTA

[env:nodemcuv2]
platform = espressif8266
board = nodemcuv2
framework = arduino
build_flags = ${common.build_flags}

[env:moteinomega]
platform = atmelavr
board = moteinomega
framework = arduino
;upload_port=/dev/cu.usbserial-DN00Z51U
;upload_port=/dev/cu.usbserial-DA01I8AV
;upload_speed=115200
build_flags = ${common.build_flags}
;  -D__AVR_ATmega1284P__
lib_ignore = ${common.lib_ignore}, TFT_eSPI

Source file to reproduce issue: "Main file":

#include <libT2.h>
//#include <adafruit
#include <displayGeneric.h>

TTf f;

void setup(){
  Serial.begin(115200);

}

void loop(){

  uint8_t myint = 20;
  uint16_t myint16 = 11111;
  float myfloat = 12.3;
  struct {
      int i = -10;
      char str[10]="hello!";
  } mystruct;

  f.bugger();
  f.myf(myint);
  f.myf(myint16);
  f.myf(myfloat);
  //f.myf(myfloat);

  delay(1000);

}

Lib file that exclude another lib with compile directives

#ifndef DISPLAY_H
#define DISPLAY_H

#define DISPLAY_LCD
//#define DISPLAY_ST7735  //uncomment to select this display type/model

#include <Adafruit_GFX.h>    // Core graphics library
#if defined(DISPLAY_LCD)

    #include <Adafruit_PCD8544.h> //Required for the Nokia 5110 LCD display

    // Initiate the display instance - Software SPI (slower updates, more flexible pin options):
    // pin 24/A0 - Serial clock out (SCLK)
    // pin 25/A1 - Serial data out (DIN)
    // pin 26/A2 - Data/Command select (D/C)
    // pin 27/A3 - LCD chip select (CS)
    // pin 28/A4 - LCD reset (RST)

  #if defined(ESP32)
        // pins defined @ User_Setup.h on TFT_eSPI folder
        /*
        #include <Adafruit_ST7735.h> //Required for OLED LCD
        #define RST  26 //13 //1 //A5 //18
        #define RS   27 //12 //0 //A4 //19
        #define SDA  14 // A2 //20
        #define SCL  12 //27 // A0 //21
        #define CS   13 //26 //3 //A3 //22

        Adafruit_ST7735 display = Adafruit_ST7735(CS, RS, SDA, SCL, RST); //, RST);
        */
        #include <TFT_eSPI.h> //Required for OLED LCD
        TFT_eSPI display = TFT_eSPI();

  #else //defined(ESP8266)
    //D2 (GPIO4)    0 RST   Output from ESP to reset display
    //D1 (GPIO5)    1 CE    Output from ESP to chip select/enable display
    //D6 (GPIO12)   2 DC    Output from display data/command to ESP
    //D7 (GPIO13)   3 Din   Output from ESP SPI MOSI to display data input
    //D5 (GPIO14)   4 Clk   Output from ESP SPI clock
    //3V3   5 Vcc   3.3V from ESP to display
    //D0 (GPIO16)   6 BL    3.3V to turn backlight on, or PWM
    //G (GND)   7 Gnd   Ground

    #define RST  4 //A5 //18   // RST pin on LCD
    #define RS   10 //A4 //19   // DC pin on LCD
    #define SDA  MISO //20       // DIN pin on LCD
    #define SCL  6 //21       // CLK pin on LCD
    #define CS   9             // CE pin o LCD

    Adafruit_PCD8544 display = Adafruit_PCD8544(SCL, SDA, CS, RS, RST);
    #define PIN_LCD_LIGHT 5 //Backlight pin for NOKIA LCD (LIGHT pin on LCD)

    #endif

    // character size from Adafruit_GXF lib
    #define CHARWIDTH 6
    #define CHARHEIGHT 8
    #define SCRROTATION 0 // 0 rotation
    #define CHARSCALE 1
    #define SCRPIXELX 240
    #define SCRPIXELY 320
    #define SCRLINES 6    // 6 lines
    #define SCRCHARS 14   // 14 characters
///  #define BOXSIZE 120

#endif

Additional info

[ivankravets]

I think PlatformIO should install libraries to %projectdepsdir%/%env%/. In this case, we will not have conflict.

Possible issues:

• PIO Home depends on the one libdeps_dir
• Lib-changes in platformio.ini should lead to removing of libdeps_dir

[sarfata]

@ivankravets I think this issue was recently introduced by 3.5.3 or 3.5.4. In my opinion, breaking projects that used to compile should qualify as a bug, not an enhancement.

With platformio 3.5.2 my project with multi environments compiles fine with 3.5.2 and now with 3.5.4 it does not build anymore.

[p3p]

@ivankravets Is there a solution for this in the works? I seem to be able to work around the issue with

lib_ldf_mode      = chain+
lib_compat_mode   = strict

but it more than doubles our Travis test times, it is more of a dev issue than end user as they tend to just build one env but it still causes problems.

[ivankravets]

@p3p you can manually control LDF with lib_deps and lib_ignore options instead of lib_ldf_mode = chain+

[p3p]

Indeed I can specify everything myself or just disable the ldf as I have needed to in the past, but the issue of building incompatible libraries for envs that don't need them if they have already been installed by another env seems to be something that should be fixed?

[ivankravets]

but the issue of building incompatible libraries for platforms that don't need them if they have already been installed by another env seems to be something that should be fixed?

Yes, this should be fixed with this feature request.

Indeed I can specify everything myself or just disable the ldf

1. Disable lib_ldf_mode = chain+
2. Start building
3. If some library was included and you don't need it, exclude with lib_ignore
4. If some library was not included in build process, add it manually to lib_deps.

All these options put into specific env.

[p3p]

We already do a lot of cherry picking libraries over at Marlin we support a lot of environments, I was just going to report the bug and found this issue with a changing milestone so thought I'd make sure it was still in the works ;)

[ivankravets]

It's not an old issue :) We plan to start a work on PlatformIO Core 4.0 soon.

[p3p]

Didn't mean to imply you were sitting on your hands, sometimes issues can just fly under the radar, as you say it's not a major issue as I can work around it.

[aster94]

even if this is not completely related to this issue: note that one of the problem of the LDF is that it should include the right library based on the architecture/platform example: https://platformio.org/lib/show/870/WiFi as you can see from the platform on the right it is wrong since that library is only for Arduino WiFi Shield but it has the same name of the ESP32 version. The same happens, i guess, for the other platform

Edit: I will try to propose a way to do it: platformio should compute the priority of two libraries using the library.properties file especially the architectures field, a function like: computePriorities(library x, library y, used_platform) could be made that gives higher priority to the library which has the same architecture to the used_platform. Also built-in libraries (like in this case) should be preferred instead of external one

[ristomatti]

I ran across this while attempting to make Embedded Template Library Arduino compatible with PlatformIO. I've got to the point that espressif8266 and espressif32 platforms seem to work but supporting atmelavr (without specifically documenting it) would require setting the library to depend on ArduinoSTL. This causes build to fail on other platforms even when ArduinoSTL.h is not included.

If I define lib_compat_mode = strict in platformio.ini everything works great. So I thought maybe if in library.json I define:

  "build": {
    "libCompatMode": "strict"

... it will do the same but it doesn't. When I run pio run -e d1_mini -v I see a message indicating the unsupported library would be skipped but it is still included in the compiler parameters:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz 80KB RAM (4MB Flash)
Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF MODES: FINDER(chain) COMPATIBILITY(soft)
Collected 79 compatible libraries
Scanning dependencies...
Skip platform incompatible dependency {u'frameworks': [u'arduino'], u'version': u'>=1.1.0', u'name': u'ArduinoSTL', u'platforms': [u'atmelavr']}

In PlatformIO library index ArduinoSTL seems to also be defined only for atmelavr and atmelsam.

I also noticed the library.json documentation instructs libCompatMode should be defined as Integer but the referred documentation uses String. I tried giving it numeric values 2 and 3 but it ddn't seem to make a difference.

[ivankravets]

Please re-test with pio upgrade --dev. Updated docs => http://docs.platformio.org/en/latest/userguide/lib/index.html

pio lib --help

[mcspr]

Based on OP issue, isn't it better to make libdeps dir name based on platform+framework+(+board?) instead of env name? If multiple environments have the same lib_deps variable and the same platform,framework, libs are installed each time.

[ivankravets]

@mcspr and we will back to the same problem which we had before. For example, some envs can have different lib_deps and platform+framework+(+board?) ID will lead to a conflict issue.

The only solution is to provide isolated build environment for project configuration. This is actually what PlatformIO Core 4.0 will do.

Please note that we don't do multiple downloads of project dependencies. We cache them. Normally, the size of a library is 50-300Kb.

[mcspr]

Yes, I did miss possibility of different library versions.

However, I think it still force downloads github repos? For example

lib_deps =
    ArduinoJson@5.13.4
    https://github.com/me-no-dev/ESPAsyncTCP#7e9ed22
    https://github.com/xoseperez/NtpClient.git#0942ebc

Fixed version works fine with platformio registry, but next github repo is one picking specific commit mid versions, second one is forked library. Both will be git-cloned. Or some other method should be used, like using branch / commit archive zips?(e.g replace the first link with https://github.com/me-no-dev/ESPAsyncTCP/archive/7e9ed22ed0078097f895dfee6ebcec90046b81b9.zip)

I think I found a bug (warrants another issue to track?) Caching is evidently broken: #2559

[ivankravets]

@mcspr thank you so much for the issue https://github.com/platformio/platformio-core/issues/2559

I'll investigate!