search

thunderace / Esp8266-Arduino-Makefile

Makefile to build arduino code for ESP8266/ESP32x under linux (tested on debian X64)
34 stars 20 forks source link

Upload only using the command-line; doesn't find libs #13

Closed nemo9955 closed 6 years ago

nemo9955 commented 6 years ago

My use-case: I connect whatever board (wemos/nodeMCU/plain arduino) to my PC and run my-com -f and the script puts together the sketch/board/port/baud/etc needed to flash.

For about a week I'm struggling to give the right parameters to espXArduino.mk so it would locate/compile all my dependencies, either it won't find directly inside my node-lib/src files like CONSTANTS.h or Node.h (USRCDIRS= solves that) but most of the time it failes to find ArduinoJson.h (That is most probably just the first lib to appear in the dependency search). I tried lots of variables and parameters combinations, and on 2 computers (having the script, it is easilly repetable).

I run something like this :

cd
/home/me/myproj/nodes-myproj/sketches/empty
&&
make
-f
/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/Esp8266-Arduino-Makefile/espXArduino.mk
ARDUINO_VARIANT="d1_mini"
GLOBAL_USER_LIBDIR=/home/me/myproj/3rd-party/nodes-myproj/sketch-libs/*
USRCDIRS="/home/me/myproj/nodes-myproj/node-lib/src"
SERIAL_PORT=/dev/ttyUSB1
ARDUINO_HOME=/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266
size

I have a few types of libraries: git clone directly from their repos , my nodes-myproj/node-lib and the ones in the esp8266 repo. Here is my directory setup (tree from my project root directory), I want to mention that it all works with the Arduino IDE (libs linked in the ~/Arduino/libraries)

nodes-myproj
├── node-lib
│   └── src
│       ├── ActuatorImplement
│       ├── AllPeripherals.h
│       ├── BaseTypes
│       ├── CONSTANTS.h
│       ├── Node.cpp
│       ├── Node.h
│       ├── Node_mesh
│       ├── Peripherals
│       ├── RespirMesh
│       ├── SensorImplement
│       ├── SimpleList.h
│       ├── SwitchImplement
│       └── UTILS
├── README.md
└── sketches
    ├── empty
    │   ├── build.d1_mini
    │   └── empty.ino
    ├── mesh.ino
    ├── solder_sens.ino
3rd-party/nodes-myproj/sketch-libs
├── Adafruit-BMP085
├── Adafruit_NeoPixel
├── Adafruit_Sensor
├── ArduinoJson
├── Arduino-Temperature-Control-Library
├── BH1750-GY30
├── DHT-sensor
├── ESP8266Ping
├── node-lib -> /home/me/myproj/nodes-myproj/node-lib
├── OneWire
└── PainlessMesh

Here is the output when I run the command (I added some $(info ) inside espXArduino.mk ) :

[localhost] local:  cd /home/me/myproj/nodes-myproj/sketches/empty &&  make  -f /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/Esp8266-Arduino-Makefile/espXArduino.mk ARDUINO_VARIANT="d1_mini"  GLOBAL_USER_LIBDIR=/home/me/myproj/3rd-party/nodes-myproj/sketch-libs USRCDIRS="/home/me/myproj/nodes-myproj/node-lib/src" SERIAL_PORT=/dev/ttyUSB1  ARDUINO_HOME=/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266  size
-------ARDUINO_LIBS------ 
ESP8266HTTPClient ESP8266WiFi 
-------USER_LIBS------ 
 ArduinoJson 
-------USRCDIRS------ 
/home/me/myproj/nodes-myproj/node-lib/src 
-------LOCAL_SRCS------ 
 /home/me/myproj/nodes-myproj/node-lib/src/Node.cpp  /home/me/myproj/nodes-myproj/node-lib/src/Node.h /home/me/myproj/nodes-myproj/node-lib/src/SimpleList.h /home/me/myproj/nodes-myproj/node-lib/src/CONSTANTS.h /home/me/myproj/nodes-myproj/node-lib/src/AllPeripherals.h  
-------ALIBDIRS------ 
-------LIB_CXXSRC------ 
 /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266HTTPClient/src//ESP8266HTTPClient.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFiScan.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFiSTA.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFiAP.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//WiFiClientSecure.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFiMulti.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFi.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//ESP8266WiFiGeneric.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//WiFiUdp.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//WiFiServer.cpp /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src//WiFiClient.cpp 
/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266HTTPClient/src/ /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src/ 
-------LIB_CSRC------ 

-------OBJ_FILES------ 
./build.d1_mini/ESP8266HTTPClient.cpp.o ./build.d1_mini/ESP8266WiFiScan.cpp.o ./build.d1_mini/ESP8266WiFiSTA.cpp.o ./build.d1_mini/ESP8266WiFiAP.cpp.o ./build.d1_mini/WiFiClientSecure.cpp.o ./build.d1_mini/ESP8266WiFiMulti.cpp.o ./build.d1_mini/ESP8266WiFi.cpp.o ./build.d1_mini/ESP8266WiFiGeneric.cpp.o ./build.d1_mini/WiFiUdp.cpp.o ./build.d1_mini/WiFiServer.cpp.o ./build.d1_mini/WiFiClient.cpp.o ./build.d1_mini/empty.fullino.o ./build.d1_mini/Node.cpp.o 
------- end ------ 
cat empty.ino  > build.d1_mini/empty.fullino
/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ -x c++ -D_TAG_=\"2017-11-12T00:58:11+02:00\"  -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/tools/sdk/include -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/tools/sdk/lwip/include -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/tools/sdk/libc/xtensa-lx106-elf -DLWIP_OPEN_SRC -DF_CPU=80000000L -DARDUINO=10605 -DARDUINO_ESP8266_WEMOS_D1R2MINI -DESP8266 -DARDUINO_ARCH_ESP8266 -c -Os -g -mlongcalls -mtext-section-literals -fno-exceptions -fno-rtti -falign-functions=4 -std=c++11 -MMD -ffunction-sections -fdata-sections -include Arduino.h -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/cores/esp8266 -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/variants/d1_mini -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266HTTPClient/src/ -I/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266/libraries/ESP8266WiFi/src/   -I/home/me/myproj/nodes-myproj/node-lib/src build.d1_mini/empty.fullino -o build.d1_mini/empty.fullino.o
In file included from build.d1_mini/empty.fullino:1:0:
/home/me/myproj/nodes-myproj/node-lib/src/Node.h:5:25: fatal error: ArduinoJson.h: No such file or directory
 #include "ArduinoJson.h"
                         ^
compilation terminated.
/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/Esp8266-Arduino-Makefile/espXArduino.mk:335: recipe for target 'build.d1_mini/empty.fullino.o' failed
make: *** [build.d1_mini/empty.fullino.o] Error 1
rm build.d1_mini/empty.fullino
thunderace commented 6 years ago

try with this tree : 

nodes-myproj
├── node-lib
│   └── src
│       ├── ActuatorImplement
│       ├── AllPeripherals.h
│       ├── BaseTypes
│       ├── CONSTANTS.h
│       ├── Node.cpp
│       ├── Node.h
│       ├── Node_mesh
│       ├── Peripherals
│       ├── RespirMesh
│       ├── SensorImplement
│       ├── SimpleList.h
│       ├── SwitchImplement
│       └── UTILS
├── README.md
├── empty.ino
├── mesh.ino
├── solder_sens.ino
├── Makefile
├── libraries
    ├── Adafruit-BMP085
    ├── Adafruit_NeoPixel
    ├── Adafruit_Sensor
    ├── ArduinoJson
    ├── Arduino-Temperature-Control-Library
    ├── BH1750-GY30
    ├── DHT-sensor
    ├── ESP8266Ping
    ├── node-lib -> /home/me/myproj/nodes-myproj/node-lib
    ├── OneWire
    └── PainlessMesh

Makefile content : 

ARDUINO_VARIANT    = d1_mini
SERIAL_PORT = /dev/ttyUSB1
ARDUINO_HOME=/home/me/myproj/3rd-party/nodes-myproj/deploy-libs/esp8266
include /home/me/myproj/3rd-party/nodes-myproj/deploy-libs/Esp8266-Arduino-Makefile/espXArduino.mk

to build : 

cd /home/me/myproj/nodes-myproj
make

empty.ino, mesh.ino and solder_sens.ino will be build

nemo9955 commented 6 years ago

The variables in that Makefile also apear in my command, I do not think that is the problem. I printed lots of steps in the execution, and pinpointed the problem to ULIBDIRS being empty:

-------LOCAL_USER_LIBDIR------ 
/home/me/myproj/3rd-party/nodes-myproj/sketch-libs 
-------USER_LIBS------ 
ArduinoJson ESP8266Ping  
-------ULIBDIRS------

On this line:

# user libraries and sketch code
ULIBDIRS = $(sort $(dir $(wildcard \
    $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/*.c) \
...
    $(USER_LIBS:%=$(GLOBAL_USER_LIBDIR)/%/src/*/*/*.cpp))))

I understand that you are wildcard searching source files, but what does this part do $(USER_LIBS:%=$(LOCAL_USER_LIBDIR) and why doesn't it find /home/me/myproj/3rd-party/nodes-myproj/sketch-libs/ArduinoJson and /home/me/myproj/3rd-party/nodes-myproj/sketch-libs/ESP8266Ping ?

thunderace commented 6 years ago

Remove the trailing in GLOBAL_USER_LIBDIR=/home/me/myproj/3rd-party/nodes-myproj/sketch-libs/ GLOBAL_USER_LIBDIR is a directory not a list of directories.

thunderace commented 6 years ago

Try the last git version. I have added a fix in order to detect the AruinoJson library

nemo9955 commented 6 years ago

Yes, $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/*.h) \ fixes ArduinoJson. The problem is that I have still a lot of problems with dependencies, and I started to understand it is not because of my project setup. I will list some of the problems I found and after that, a proposed solution.

I added $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/src/*.h) \ for ESP8266Ping. I added 

        $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/*/*.cpp) \
    $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/*/*/*.cpp) \
    $(USER_LIBS:%=$(LOCAL_USER_LIBDIR)/%/*/*/*/*.cpp) \

for my library that does not have a src folder. I also had to rename the symlink of my lib from node-lib to Node so it would search in it when I #import<Node.h>. This problem apears also with the lib BH1750-GY30 that has the import #include <BH1750.h> . Besides #include "Node.h", I also have Node/CONSTANTS.h that I #include "CONSTANTS.h" in the .ino, I had to reorder them to 

#include "Node.h"
#include "CONSTANTS.h"

so the Node would be added to USER_LIBS and so CONSTANTS.h file's path would be available. Another problem was, inside my node-lib i import ESP8266HTTPClient.h, the script could not "find" it until I copied #include <ESP8266HTTPClient.h> into the sketches/testing_sensor.ino.

And to top it all, all the errors due to dependencies (ESP8266HTTPClient.h,CONSTANTS.h ) had nothing to do with the sketch (TARGET) I was trying to flash (ardujson_test.ino which only includes ArduinoJson ), but because ardujson_test.ino and testing_sensor.ino are siblings in the sketches folder, USER_INOSRC := $(wildcard $(addsuffix /*.ino,$(USRCDIRS))) due to some unknown reason, combines all the dependencies into the .fullino file.

Some justification on my setup:

My proposed solution:

  1. start from a list of folders (no matter if it is the root of a project or something like $(ARDUINO_HOME)/libraries ) and then recursively expand the list (maybe ignore folders like test/tests/doc/examples/etc. ) until you have a giant list of h/hpp/c/cpp files, call it ALL_DEPS, simply put, it has all the files needed to build every .ino file.
  2. take the TARGET, which should be just a file path that can be any extension (ino/c/cpp) and look for includes, add them to a list DEPENDENCIES, and add the TARGET file to a list called DEP_FILES
  3. iterate thru DEPENDENCIES, if a file name is found in ALL_DEPS, add that file path to DEP_FILES and it's includes to DEPENDENCIES
  4. repeat step 3 until the contents of DEPENDENCIES and DEP_FILES stop changing
  5. All the files paths that TARGET file needs are now in DEP_FILES, no matter the name of the root project, it's folder hierarchy or in which file there's a new include/dependeny
thunderace commented 6 years ago

I also had to rename the symlink of my lib from node-lib to Node so it would search in it when I #import. This problem apears also with the lib BH1750-GY30 that has the import #include

Yes, for now, library directory name have to be the same as the include file. I'm working on this point.

"Another problem was, inside my node-lib i import ESP8266HTTPClient.h, the script could not "find" it until I copied #include into the sketches/testing_sensor.ino."

Fixed in the dev branch. Not ready for release but you can try.

My proposed solution: start from a list of folders (no matter if it is the root of a project or something like $(ARDUINO_HOME)/libraries ) and then recursively expand the list (maybe ignore folders like test/tests/doc/examples/etc. ) until you have a giant list of h/hpp/c/cpp files, call it ALL_DEPS, simply put, it has all the files needed to build every .ino file. take the TARGET, which should be just a file path that can be any extension (ino/c/cpp) and look for includes, add them to a list DEPENDENCIES, and add the TARGET file to a list called DEP_FILES iterate thru DEPENDENCIES, if a file name is found in ALL_DEPS, add that file path to DEP_FILES and it's includes to DEPENDENCIES repeat step 3 until the contents of DEPENDENCIES and DEP_FILES stop changing All the files paths that TARGET file needs are now in DEP_FILES, no matter the name of the root project, it's folder hierarchy or in which file there's a new include/dependeny

It's how arduino_builder work. I'm working on this solution for 2 weeks but it's a litttle hard and slow.

thunderace commented 6 years ago

Done in dev branch My solution: Search for dependencies from src files (ino and c/cpp) in USER_LOCAL_LIB, USER_GLOBAL_LIB and $(ARDUINO_HOME). Do it again with libs of USER_LOCAL_LIB and USER_GLOBAL_LIB. That's all. working great with my differents projects. I will create a project with the same tree as your project and test if it's ok.