search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.92k stars 792 forks source link

Hangs while scanning dependencies only in Docker container #4138

Open fionn-r opened 2 years ago

fionn-r commented 2 years ago

Configuration

Operating system: Host machine: Windows 10 Docker container:

FROM python:3.8.7-slim

ENV DEBIAN_FRONTEND=noninteractive
ENV PROJ_DIR /firmware

RUN apt-get update -q \
  && apt-get install --no-install-recommends -yq \
  lcov valgrind git \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install --upgrade pip setuptools
RUN python3 -m pip install platformio

# this is a really crappy way to get all the required environment installed in the container
COPY . $PROJ_DIR
RUN pio run --project-dir ${PROJ_DIR} || true
RUN pio system prune -f
RUN rm -r $PROJ_DIR

PlatformIO Version (platformio --version): Host: PlatformIO Core, version 5.2.3 Docker: PlatformIO Core, version 5.2.3

Description of problem

Compiling firmware on Docker takes a much longer time to scan the dependencies than a native Windows run. I know that the hypervisor can cause extra time but the running it with timestamps shows a 2.61s time on Windows vs 77.52 on Linux (~3000% difference).

Steps to Reproduce

If necessary I can try to create a minimal reproducible but you can use the above docker container and add it to a Platformio project and from the project directory:

docker build -t platformio-builder:latest .
docker run -v [pwd]:/firmware platformio run

Actual Results

Windows:

> filter timestamp {"$(Get-Date -Format o): $_"}; pio run -v | timestamp
...
2021-12-14T10:22:03.72: LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
2021-12-14T10:22:03.72: LDF Modes: Finder ~ chain, Compatibility ~ soft
2021-12-14T10:22:03.98: Found 87 compatible libraries
2021-12-14T10:22:03.98: Scanning dependencies...
2021-12-14T10:22:06.64: Dependency Graph
2021-12-14T10:22:06.64: |-- 
> docker run -v [pwd]:/firmware pio run -v | timestamp
...
10:24:25.59: LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
10:24:25.59: LDF Modes: Finder ~ chain, Compatibility ~ soft
10:24:33.58: Found 87 compatible libraries
10:24:33.58: Scanning dependencies...
10:25:51.10: Dependency Graph
10:25:51.11: |-- ...

Expected Results

The times are closer

If problems with PlatformIO Build System:

The content of platformio.ini: A little minimal since there's a lot in it

[platformio]
description = Firmware

[env] ;Common is done for all environments
monitor_speed = 115200
monitor_flags= --raw

lib_extra_dirs =
  ./shared-config

; ...

;==========ESP32===========
[env:esp32]
platform = espressif32
board = esp32dev
framework = arduino
build_type = debug
debug_tool = esp-prog
debug_init_break = tbreak setup

Source file to reproduce issue:

Insert here...
ivankravets commented 2 years ago

Have you resolved this issue?

fionn-r commented 2 years ago

Hi, no I haven't. I think it's some sort of issue with the LDF accessing Windows files from the container but even calling with verbose it doesn't give me any more information.

I can look into the problem a bit more but I'm not sure where to start looking

ivankravets commented 2 years ago

Do you have any updates? Could you help us to debug this issue?

fionn-r commented 2 years ago

Is there any way that I can add in extra logging so I can timestamp between

10:24:33.58: Scanning dependencies...
10:25:51.10: Dependency Graph
ivankravets commented 2 years ago

Could you reproduce this issue with the latest PlatformIO Core? 

pio upgrade --dev
fionn-r commented 2 years ago

Ok, I tried with that and I still don't get any more logging inbetween:

2022-04-04T10:37:25.2087082+02:00: Found 104 compatible libraries
2022-04-04T10:37:25.2097729+02:00: Scanning dependencies...
2022-04-04T10:38:33.1718258+02:00: Dependency Graph
❯ pio --version
PlatformIO Core, version 5.3.0b3
fionn-r commented 2 years ago

I've had a look into it and it could be to do with the way that Platformio is accessing the files when scanning the dependencies since it is accessing files local to Windows via the hypervisior.

I have seen that there's a overhead when accessing files via the hypervisior (e.g. docker, WSL) so that could have something to do with it. I guess my problem is that the LDF is scanning through a heap of files so the short term solution is to not use the LDF and manually include all the dependencies

ivankravets commented 2 years ago

You can turn off it https://docs.platformio.org/en/latest/projectconf/section_env_library.html#lib-ldf-mode

Yes, LDF scans ALL C/C++ source/header files to resolve dependencies.

fionn-r commented 2 years ago

Ok, I will try that workaround for now

Thanks for the help!

fionn-r commented 2 years ago

Can you clarify how to bypass the LDF? I have dumped the following:

...
'__PIO_LIB_BUILDERS': [ <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\ControllablePowerOutputBase'),
                          <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\PowerOutput_24V'),
                          <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\PowerOutput_5V'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit ADS1X15'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit BusIO'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit MCP4725'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit MCP4728'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit NeoPixel'),
                          <class 'piolib.PlatformIOLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Embedded Template Library'),
                          <class 'piolib.PlatformIOLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\NTPClient'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ArduinoOTA'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\AsyncUDP'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\AzureIoT'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\BLE'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\BluetoothSerial'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\DNSServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\EEPROM'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ESP32'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ESPmDNS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\FFat'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\FS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPClient'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPUpdate'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPUpdateServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\NetBIOS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Preferences'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SD'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SD_MMC'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SPI'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SPIFFS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SimpleBLE'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Ticker'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Update'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WebServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFi'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFiClientSecure'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFiProv'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Wire')],

And now I'm trying to include the dependencies manually.

I tried in n platformio.ini:

...
lib_ldf_mode = off
lib_deps =
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/AsyncUDP
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFi
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFiClientSecure
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPClient
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdate
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdateServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ArduinoOTA
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/AzureIoT
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/BLE
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/BluetoothSerial
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/DNSServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/EEPROM
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ESP32
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ESPmDNS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/FFat
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/FS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/NetBIOS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Preferences
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SD
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SD_MMC
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SPI
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SPIFFS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SimpleBLE
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Ticker
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Update
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WebServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFiProv
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Wire
  ${extra.lib_deps_base}
  etlcpp/Embedded Template Library @ ^19.3.5
  ./lib/ControllablePowerOutputs/ControllablePowerOutputBase
  ./lib/ControllablePowerOutputs/PowerOutput_24V
  ./lib/ControllablePowerOutputs/PowerOutput_5V
...

I then get issues that source files within ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdate can't find each other. I tried using lib_extra_dirs but then got a different set of errors. What is the correct way to be doing it?