Include nothing in only specific still seems to include HttpsOTAUpdate and fail compilation

[nicklasb]

Board

ESP32

Device Description

A LilyGO TTGO-LORA32.

Hardware Configuration

Only USB.

Version

v2.0.3

IDE Name

Visual Studio Code & ESP-IDF

Edit: I am NOT using PlatformIO as that doesn't seem to work right now.

Operating System

MacOS X 12.3.1

Flash frequency

80 Mhz

PSRAM enabled

yes

Upload speed

115200

Description

Selecting no libraries in "Include only specific Arduino libraries" causes a compilation error:

arduino/libraries/Update/src/HttpsOTAUpdate.cpp:17:10: 
fatal error: esp_http_client.h: No such file or directory
#include "esp_http_client.h"
          ^~~~~~~~~~~~~~~~~~~

Enabling HttpsOTAUpdate in menuconfig causes it go start compiling again.

Sketch

Just building any mixed project should cause the problem.

Debug Message

-mlongcalls -Wno-frame-address -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -ggdb -Og -fmacro-prefix-map=~project=. -fmacro-prefix-map=~/esp/esp-idf=IDF -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu++11 -fno-exceptions -fno-rtti -D_GNU_SOURCE -DIDF_VER=\"v4.4.1\" -DESP_PLATFORM -D_POSIX_READER_WRITER_LOCKS -DARDUINO=10812 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 -DARDUINO_BOARD=\"ESP32_DEV\" -DARDUINO_VARIANT=\"esp32\" -DESP32 -MD -MT esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Update/src/HttpsOTAUpdate.cpp.obj -MF esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Update/src/HttpsOTAUpdate.cpp.obj.d -o esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Update/src/HttpsOTAUpdate.cpp.obj -c project/components/arduino/libraries/Update/src/HttpsOTAUpdate.cpp
project/components/arduino/libraries/Update/src/HttpsOTAUpdate.cpp:17:10: fatal error: esp_http_client.h: No such file or directory
 #include "esp_http_client.h"
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lbernstone]

Please include the platformio.ini you are using for help with platform.io "Just building any mixed project should cause the problem." - What does "mixed project" mean?

[nicklasb]

Hi @lbernstone ! Thank you for your reply!

I am sorry; I could have been much clearer, a bit tired:

1. I do not use Platform.IO currently due to it not being compatible with 4.4 yet. This is "pure" ESP-IDF + arduino-esp32

2. It seems to me that should happen on most such projects, regardless of other configurations, and appear somewhere in the compiler output (esp_http_client.h: No such file or directory.) amongst all the other errors. It is a typical non-tested test case when you deselect everything anywhere, in my experience. I am not sure how I create such a testing project so that it fits in the sketch area, as it is about config.

3. I mean a mixed Arduino and ESP-IDF-project, like one with arduino-esp32 as a component in an ESP-IDF-project, which this is.

[lbernstone]

Gotcha. Arduino as an ESP-IDF component. You want just arduino core. Hold the pickles, hold the lettuce.

[nicklasb]

Exactly. Basically I only use stream.h, i need it to get EspSoftwareSerial to work. For context, this is the setup: https://github.com/nicklasb/Peripheral/tree/master/components

Great reference, not from the states, so I didn't get it at first. True, I really want it my way, pretty mainstream with burgers, though. :-)

[Jason2866]

You can NOT use Arduino core 2.0.3 and IDF with Platformio. Since latest Platformio is still NOT using IDF 4.4.(1). Both have to match. You can try this setup where i updated the Platformio setup to use IDF 4.4.1

platform = https://github.com/tasmota/platform-espressif32/releases/download/v.2.0.3/platform-espressif32-v.2.0.3.zip

[nicklasb]

You can NOT use Arduino core 2.0.3 and IDF with Platformio. Since latest Platformio is still NOT using IDF 4.4.(1). Both have to match. You can try this setup where i updated the Platformio setup to use IDF 4.4.1

platform = https://github.com/tasmota/platform-espressif32/releases/download/v.2.0.3/platform-espressif32-v.2.0.3.zip

Again, I am sorry, apparently I wasn't clear with that, I am NOT using PlatformIO for this specific case (as per reasons stated above).

Either way, problem is not surely, but likely, regardless of using PlatformIO or not.

Edit: Updated the issue.

[Jason2866]

You was clear. The setup i provided above does compile arduino, idf projects with Platformio. Can be seen in the Github Actions CI here

If you have a M1 Mac, with this setup native arm toolchains will be used for compiling ;-)

[nicklasb]

You was clear. The setup i provided above does compile arduino, idf projects with Platformio. Can be seen in the Github Actions CI here

If you have a M1 Mac, with this setup native arm toolchains will be used for compiling ;-)

Ok, cool, thanks, I will try it.

This issue though, is about being able to use as little as possible of the Arduino-platform. A common need for it is for softwareserial until there is a standalalone implementation for ESP-IDF, and that basically only need stream.h and perhaps millis.

[sukesh-ak]

This issue happens even without PlatformIO. So please remove that tag.

ISSUE: I am using ESP-IDF but have few Arduino I2C sensor libraries to be integrated.

While using the menuconfig option of Include only specific Arduino libraries and disable everything other than the required ones (eg. Wire and SPI) the same compiler/linking error shows up.

../components/arduino/libraries/Update/src/HttpsOTAUpdate.cpp:17:10: fatal error: esp_http_client.h: No such file or directory

 #include "esp_http_client.h"
          ^~~~~~~~~~~~~~~~~~~

compilation terminated.

[1070/1132] Building CXX object esp-idf/arduino/CMakeFiles/__idf_arduino.dir/libraries/Update/src/Updater.cpp.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

I am doing OTA and everything else in ESP-IDF already so not interested in esp_http_client.h.

[sukesh-ak]

If I manually remove everything else and have only these in Arduino CMakeLists.txt it works fine. So the issue is somewhere related to menuconfig option workflow. Not sure where and how that is taken care of.

set(LIBRARY_SRCS
  libraries/SPI/src/SPI.cpp
  libraries/Wire/src/Wire.cpp
  )

set(BLE_SRCS
  )

set(includedirs
  variants/${IDF_TARGET}/
  cores/esp32/
  libraries/SPI/src
  libraries/Wire/src
  )

[VojtechBartoska]

@me-no-dev Can you please take o look on this? Thanks!

[me-no-dev]

I took a look. There is nothing preventing the source to compile. Checks for #if !CONFIG_ARDUINO_SELECTIVE_Update need to be added to the source and header files.

@VojtechBartoska you can add this to the roadmap for future resolution

[me-no-dev]

@sukesh-ak things are becoming so entangled with each other that I do not know for how long this option will exist in menuconfig. It was added by the community when we were using make (and esp-idf 3). With cmake we might not know the value of configuration options at all times, so limiting it in CMakeLists might not work.

[sukesh-ak]

@sukesh-ak things are becoming so entangled with each other that I do not know for how long this option will exist in menuconfig. It was added by the community when we were using make (and esp-idf 3). With cmake we might not know the value of configuration options at all times, so limiting it in CMakeLists might not work.

So do you suggest editing and removing unwanted things from CMakeLists.txt of Arduino folder being the best option? I mean without depending on the menuconfig component selection.

[nicklasb]

@sukesh-ak things are becoming so entangled with each other that I do not know for how long this option will exist in menuconfig. It was added by the community when we were using make (and esp-idf 3). With cmake we might not know the value of configuration options at all times, so limiting it in CMakeLists might not work.

Ok. Also I am a little less experienced, so for people like me the "only specific" has good value, as limiting in CMakeList seems to me like I have to know more to do. Memory usage always seem to be a constraint, and especially in cases like this when you need to use things from two worlds.

Also, it seems like a good way to keep down unnecessarily entangled and complex dependencies in the library. Or perhaps stream.h and the like could be broken out in a separate selection, if no selection isn't possible.

[sukesh-ak]

@sukesh-ak things are becoming so entangled with each other that I do not know for how long this option will exist in menuconfig. It was added by the community when we were using make (and esp-idf 3). With cmake we might not know the value of configuration options at all times, so limiting it in CMakeLists might not work.

Ok. Also I am a little less experienced, so for people like me the "only specific" has good value, as limiting in CMakeList seems to me like I have to know more to do. Memory usage always seem to be a constraint, and especially in cases like this when you need to use things from two worlds.

Also, it seems like a good way to keep down unnecessarily entangled and complex dependencies in the library. Or perhaps stream.h and the like could be broken out in a separate selection, if no selection isn't possible.

I am in the process of writing a simple sample (repo with doc) of how to integrate both worlds of ESP-IDF with Arduino library.

Hope that would help people with less experience (I spent quite sometime to figure this out myself).

[sukesh-ak]

@nicklasb

Here is the write-up with a sample https://github.com/sukesh-ak/Arduino-Library-with-IDF

Hope this helps in future.

You also have the option to trim the Arduino fat by removing things not required from Arduino CMakeLists.txt like below. Not essential though.

set(LIBRARY_SRCS
  libraries/SPI/src/SPI.cpp
  libraries/Wire/src/Wire.cpp
  )

set(BLE_SRCS
  )

set(includedirs
  variants/${IDF_TARGET}/
  cores/esp32/
  libraries/SPI/src
  libraries/Wire/src
  )

[william-ferguson-au]

So I'm a little lost.

Why when menuconfig has been explicitly configured to not include the Updates library is it being included?

[MassimilianoMontagni]

Any update about this issue?

[Jason2866]

Maybe i miss something, but it does no harm at all since the Linker will include only the needed. And as me-no-dev noted if you want to exclude as selected (for comilation task) in menuconfig you may have to add #if !CONFIG_ARDUINO_SELECTIVE_Update

[nicklasb]

Maybe i miss something, but it does no harm at all since the Linker will include only the needed. And as me-no-dev noted if you want to exclude as selected (for comilation task) in menuconfig you may have to add #if !CONFIG_ARDUINO_SELECTIVE_Update

As you might see in the original report, the error does not occur in my code, but in other parts that are included without me using them.

[Jason2866]

Yes, have you tried latest version? I can not reproduce the issue.

[nicklasb]

Yes, have you tried latest version? I can not reproduce the issue.

I have not, but will try at the earliest opportunity.

[hero]

Agree with you, the main reason is that the options in the Kconfig configuration file ("Include only specific Arduino libraries") that affect menuconfig do not work.

[hurricanefrog]

This not only happens if you select the CONFIG_ARDUINO_SELECTIVE... KConfig option. The build error also occurs if you want to trim down the build time by setting set (COMPONENTS main) in the project root CMakeLists.txt (with Arduino as a component in a ESP-IDF project), because the Arduino component doesn't specify esp_http_client as PRIV_REQUIRES.

edit: esp_https_ota is also missing from the requirements for compilation to succeed.

edit2: I just found out I can work around this by setting set (COMPONENTS esp_http_client esp_https_ota main) in the main CMakeLists.txt.

[VojtechBartoska]

Closing this as wont fix.