Document how to create a new cmake project outside the freertos source tree

jrr commented 5 years ago

Is your feature request related to a problem? Please describe.
The Getting Started Guides describe how to run the demo apps, but I can't find anything about creating a new project.

Describe the solution you would like.
Documentation describing how to create a new project outside the amazon-freertos source tree, and guidance about how to consume freertos from outside.

Additional context
With the old Make-based system, I was able to create a new ESP32 project that references amazon-freertos from outside, including this repo as an unmodified git submodule.

(This wasn't documented before, either, so it took significant effort!)

Now with the new cmake system I feel like I have to start all over.

I'm looking for something like: 1) Copy files X, Y, and Z into a new directory 2) Export an AMAZON_FREERTOS_PATH environment variable pointing the root directory of the amazon-freertos repo 3) Issue command cmake -D... to build your application image.

Is this documented anywhere? Am I missing something?

matteoscordino commented 5 years ago

Agreed, this is a big gap to user adoption in the current state of affairs. Took me quite some work, as @jrr said, to get this (extremely common) usage scenario to work with make before, and now we have to start over.

tgsong commented 5 years ago

Hi,

We haven't establish a standard way of consuming Amazon FreeRTOS outside of its source tree, and we're still evaluating several approaches for this. Generally speaking, with CMake you have 2 ways of doing this,

Add the external library (in your case, it's amazon freertos) as a subfolder in your project, and use CMake add_subdirectory command to consume it. The downside is that you have to compile external code every time.
Compile the external library and "install" its header files and build artifacts to a directory. Then in your CMake project, use the find_package command to load this dependency. This way you don't have to bring the external source code into your application, and you're just using header files and linking to static libraries. However, this needs to be supported by the external project as CMake does not magically provide this.

Normally option 2 is better than option 1 as it provides best separation of the application code and its dependencies. It's also the standard way of consuming an external library in C and C++. However, this is not implemented in Amazon FreeRTOS yet, our team spent most of the effort to support CMake build first on as many boards as possible in the last release.

Having said that, you can still use option 1 today with our CMake build support. Assuming you have the following directory structure,

- amazon-freertos
- main.c
- CMakeLists.txt

Here's a small CMake file you can try to start with,

cmake_minimum_required(VERSION 3.13)

project(my_app)

# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")
add_subdirectory(amazon-freertos)

add_executable(my_app main.c)
target_link_libraries(my_app PRIVATE AFR::mqtt)

To configure your project, run this command from your application's directory (make sure the ESP32 compiler is in environment variable PATH),

cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake

Let me know if this helps.

matteoscordino commented 5 years ago

see the discussion here for the corresponding discussion on the AWS Forum

tgsong commented 5 years ago

Hi, we have merged a pull request from Espressif which fix the rest of the problem, https://github.com/aws/amazon-freertos/pull/887

Basically you need to define IDF_PROJECT_EXECUTABLE to your application target so that IDF can be linked. And if you need to add more components to ESP-IDF, you can set IDF_EXTRA_COMPONENT_DIRS variable. Below is a small sample I copied from that PR,

cmake_minimum_required(VERSION 3.13)

project(my_app)

add_executable(my_app main.c)

# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE my_app)

get_filename_component(EXTRA_COMPONENT_DIRS
    "components/hello_world" ABSOLUTE
)

# Add some external components to the project
set(IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

target_link_libraries(my_app PRIVATE AFR::mqtt)

Let us know if this fix the problem.

jrr commented 5 years ago

This helps! I briefly fiddled with it, but need to spend some more time with it to determine if it can do everything I need.

Could you point me to examples of:

defining your own component with its own CMakeLists.txt, consumed by the root CMakeLists.txt
adding dependencies on AFR components like pkcs11 and dev_mode_key_provisioning

Lastly, is my own main.c supposed to go in a component, or be solely referenced by the root CMakeLists.txt?

matteoscordino commented 5 years ago

Thank you again @tgsong, I also tested this. The example you give works, but as @jrr pointed out, there are common cases that are not covered.

At the moment I have exactly the same questions as @jrr

defining your own component with its own CMakeLists.txt, consumed by the root CMakeLists.txt: as you can see in my CMakeLists.txt below, I have to add app/common/application_code/components/i2c_devices/i2c_bus/include to include_directories, despite the fact that that component should bring in the include directories when built into an IDF project (it uses the following idiom: set(COMPONENT_ADD_INCLUDEDIRS ". include") )
adding dependencies on AFR components like pkcs11 and dev_mode_key_provisioning same issue, the CMakeLists.txtbelow gives me the following build error: [ 89%] Building C object CMakeFiles/my_app.dir/app/common/application_code/main.c.obj /home/matteo/work/esp/my_app_201906/app/common/application_code/main.c:34:43: fatal error: aws_dev_mode_key_provisioning.h: No such file or directory

Here is my current CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)

project(my_app)

file(GLOB SOURCES "app/common/application_code/*.c")
file(GLOB HEADERS "app/common/application_code/*.h" "app/common/config_files/*.h")
add_executable(my_app ${SOURCES} ${HEADERS})

# Bring in the include directories (the i2c_devices ones would not be 
# needed if components did their job)
set(INCLUDE_DIRS ${INCLUDE_DIRS} 
    "app/common/config_files/"
    "app/common/application_code/components/i2c_devices/i2c_bus/include"
    "app/common/application_code/components/i2c_devices/sensor/hdc2010/include"
    )
include_directories(${INCLUDE_DIRS})

# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE my_app)

# define the components
get_filename_component(ESPRESSIF_CODE_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/espressif_code" ABSOLUTE)
get_filename_component(ESP_DSP_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/app/common/application_code/components/esp-dsp" ABSOLUTE)
get_filename_component(I2C_BUS_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/app/common/application_code/components/i2c_devices/i2c_bus" ABSOLUTE)
get_filename_component(HDC2010_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/app/common/application_code/components/i2c_devices/sensor/hdc2010" ABSOLUTE)

set(IDF_EXTRA_COMPONENT_DIRS    ${IDF_EXTRA_COMPONENT_DIRS} 
                            ${ESPRESSIF_CODE_COMPONENT}
                            ${ESP_DSP_COMPONENT}
                            ${I2C_BUS_COMPONENT}
                            ${HDC2010_COMPONENT}
)

# Bring in the FreeRTOS code
# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")
add_subdirectory(amazon-freertos)

message("++++++++++++++++++++++++++++++++++")
message("++++          MY APP          ++++")
message("++++++++++++++++++++++++++++++++++")

message("IDF_EXTRA_COMPONENT_DIRS: ${IDF_EXTRA_COMPONENT_DIRS}")

message("INCLUDE_DIRS: ${INCLUDE_DIRS}")

target_link_libraries(my_app PRIVATE AFR::mqtt)

tgsong commented 5 years ago

Hi,

I'm still trying to get in touch with Espressif regarding to your first question, defining your own component in your CMakeLists.txt. Generally, in CMake you do this by defining your component as a library with the CMake add_library command. Then you can link your library to your application target. Here's a small example,

# This defines your component.
add_library(my_component STATIC)

# Specify source files to your component.
target_sources(
  my_component
  PRIVATE
    src_1.c
    src_2.c
)

# Specify include directories
target_include_directories(
  my_component
  PUBLIC
    include_dir_1
    include_dir_2
)

# If your component depends on a AFR component
target_link_libraries(my_app PRIVATE AFR::pkcs11)

# Add this dependency to your application.
target_link_libraries(my_app PRIVATE my_component)

However, if your component depends on some specific ESP-IDF components, then it becomes a little bit complicated. Since in ESP-IDF, they're using the so-called component concept that they made themselves in their CMake implementation, which is different from what CMake natively offers. For AFR components, each one has a corresponding CMake target that you can use target_link_libraries to link to, like AFR::mqtt, AFR::pkcs11 and AFR::wifi. But I'm not sure about how to specify this dependencies for ESP-IDF, I'm trying to get more details from Espressif for this specific question, I'll post update once I have the answer.

Regarding to your second question, you can tell CMake to link AFR::dev_mode_key_provisioning to your application (same way I did to link AFR::mqtt to your application in the example provided before), this should bring in all public include directories from dev_mode_key_provisioning.

target_link_libraries(
  my_app
  PRIVATE AFR::dev_mode_key_provisioning
)

jrr commented 5 years ago

Starting with the simplest base case, I tried to create an empty hello world:

cmake_minimum_required(VERSION 3.13)

project(my_app)

add_executable(my_app "hello/main.c")

set(IDF_PROJECT_EXECUTABLE my_app)

set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

Building with cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake -G Ninja / ninja

It failed while building aws_demos: ../amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/main.c:44:30: fatal error: FreeRTOS_Sockets.h: No such file or directory

I need a clean slate before I can even begin to add my own code. How can I exclude the demos?

ArturBa commented 5 years ago

I have the same problem as @jrr.

../amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/main.c:44:30: fatal error: FreeRTOS_Sockets.h: No such file or directory

Also multiple problems with
```
../amazon-freertos/freertos_kernel/tasks.c:5067: undefined reference to `vApplicationIdleHook'
```
Ad. 1: I added file FreeRTOS_Sockets.h into include directory Also changed a bit project structure adding required files. You can check structure here

Ad. 2: I have copied

extern void esp_vApplicationTickHook();
void IRAM_ATTR vApplicationTickHook(){
  esp_vApplicationTickHook();
}

from demo project.

Used Cmakelist as follow:

cmake_minimum_required(VERSION 3.13)

project(my_app)

file(GLOB SOURCES "src/*.c")
file(GLOB HEADERS "config_files/*.h" "include/*.h")

add_executable(my_app ${SOURCES} ${HEADERS})

set(INCLUDE_DIRS ${INCLUDE_DIRS}
  "config_files/"
  "include/"
)
include_directories(${INCLUDE_DIRS})

# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE my_app)

get_filename_component(ESPRESSIF_CODE_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/espressif_code" ABSOLUTE)

# Add some external components to the project
set(IDF_EXTRA_COMPONENT_DIRS ${IDF_EXTRA_COMPONENT_DIRS}
  ${ESPRESSIF_CODE_COMPONENT}
)

# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

message("++++++++++++++++++++++++++++++++++")
message("++++          MY APP          ++++")
message("++++++++++++++++++++++++++++++++++")

message("IDF_EXTRA_COMPONENT_DIRS: ${IDF_EXTRA_COMPONENT_DIRS}")

message("INCLUDE_DIRS: ${INCLUDE_DIRS}")

target_link_libraries(my_app PRIVATE AFR::mqtt)

Hope that will help someone ;) PS: Also I'll refactor the structure to remove redundant files.

anurag-kar commented 5 years ago

@jrr The default target is aws_demos. Have you tried executing ninja my_app, followed by ninja flash and ninja monitor?

jrr commented 5 years ago

@anurag-kar Thanks. (🤦‍♂) In case you can't tell, I've never used cmake before 😬

I managed to get a hello world building by:

copying aws_demo/application_code/main.c's implementations of vApplicationTickHook, vApplicationIdleHook, and vApplicationDaemonTaskStartupHook.
adding amazon-freertos/demos/demo_runner/aws_demo.c.
adding a couple include dirs.

Here's my working cmakefile:

cmake_minimum_required(VERSION 3.13)

project(my_app)

file(GLOB SOURCES "hello/main.c" "amazon-freertos/demos/demo_runner/aws_demo.c")

include_directories(my_app PRIVATE "amazon-freertos/demos/include")
include_directories(my_app PRIVATE "amazon-freertos/libraries/abstractions/platform/freertos/include")

add_executable(my_app ${SOURCES} ${HEADERS})

set(IDF_PROJECT_EXECUTABLE my_app)

set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

I'm now building this with cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake -G Ninja && cd build && ninja my_app

(...)
647/647] Linking CXX executable my_app
jrr@jrrmbp ~/r/r/build (cmake-etc)> file my_app
my_app: ELF 32-bit LSB executable, Tensilica Xtensa, version 1 (SYSV), statically linked, with debug_info, not stripped

\o/

I'll try flashing this to a board tomorrow.

Edit: flashing

It worked! I flashed with ESPPORT=/dev/cu.SLAB_USBtoUART ninja flash and then connected the monitor with ./amazon-freertos/vendors/espressif/esp-idf/tools/idf.py monitor -p /dev/cu.SLAB_USBtoUART -B build.

So that's hello world; next up is porting over our actual application. I'll follow up on this thread with how that goes.

jrr commented 5 years ago

Okay, next question: how do I specify a custom sdkconfig at cmake time?

jrr commented 5 years ago

More specifically:

The conventional way for an ESP32 application to specify a few configs is with sdkconfig.defaults.
The FreeRTOS build uses this feature: https://github.com/aws/amazon-freertos/blob/d3ae47dd86c6f69557674d6a4e81535a247e1645/vendors/espressif/boards/esp32/CMakeLists.txt#L241-L247

.. without an apparent way for the end-user to supply their own. Am I missing something?

anurag-kar commented 5 years ago

@jrr You are right. It is kind of messy right now. So the only way to add your own configuration would be to change amazon-freertos/vendors/espressif/boards/esp32/aws_demos/sdkconfig.defaults.

Though, it should be straight forward to add support for allowing custom sdkconfig, i.e. by setting IDF_SDKCONFIG_DEFAULTS from the custom project cmake file.

@tgsong what do you think?

anurag-kar commented 5 years ago

@jrr @tgsong Here's a patch that should allow overriding IDF_SDKCONFIG_DEFAULTS

My project cmake file with which I tested this:

cmake_minimum_required(VERSION 3.13)

project(my_app)

add_executable(my_app main.c)

# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE my_app)

get_filename_component(EXTRA_COMPONENT_DIRS
    "components/hello_world" ABSOLUTE
)

# Add some external components to the project
set(IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

set(IDF_SDKCONFIG_DEFAULTS "${CMAKE_CURRENT_LIST_DIR}/sdkconfig.defaults")

# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

target_link_libraries(my_app PRIVATE AFR::mqtt)

tgsong commented 5 years ago

@anurag-kar Thanks for clearing up the confusion. Having a IDF_SDKCONFIG_DEFAULTS variable to allow customer to override would be great, the patch looks good to me.

I still have a question though, we used to run menuconfig target to change some settings, which stores the value in sdkconfig fie. however, since this file is generated each time by CMake now (before it was persistent), that will make menuconfig useless right? I remember I asked for this and the reason for this behaviour is to be able to switch between multiple projects (in our case demos and tests). I can think of 2 solutions here,

Do not regenerate the sdkconfig file if it's already in the build folder (similar to the previous behaviour). If we want to switch the project, just delete the build folder. We couldn't do this before because the sdkconfig was not generated in the build folder but instead under the root folder.
Detect if the user has changed the path of sdkconfig.default. For example, user may have multiple application targets and each one has its own sdkconfig.default. If user rerun CMake with different IDF_SDKCONFIG_DEFAULTS value, then regenerate the sdkconfig in the build folder.

anurag-kar commented 5 years ago

@tgsong Having solution 2 would be the most desirable. But I think for now we can revert things and have behavior as per solution 1, and add support for 2 in a follow-up PR. I'll update here when the PR is ready.

Edit : Here's the related PR : https://github.com/aws/amazon-freertos/pull/987

jrr commented 5 years ago

@anurag-kar It worked! Thanks! I'm now able to build with a custom sdkconfig.defaults with a -D to cmake like this:

-DIDF_SDKCONFIG_DEFAULTS=sdkconfig.my.defaults

Unfortunately that breaks ninja menuconfig unless I use an absolute path. Error below:

(...)
Traceback (most recent call last):
  File "/Users/jrr/repos/foo/amazon-freertos/vendors/espressif/esp-idf/tools/kconfig_new/confgen.py", line 293, in <module>
    main()
  File "/Users/jrr/repos/foo/amazon-freertos/vendors/espressif/esp-idf/tools/kconfig_new/confgen.py", line 91, in main
    raise RuntimeError("Defaults file not found: %s" % args.defaults)
RuntimeError: Defaults file not found: sdkconfig.my.defaults
FAILED: amazon-freertos/vendors/espressif/boards/esp32/esp-idf/CMakeFiles/menuconfig
(...)

Edit: In the custom sdkconfig it's now necessary to specify the partition table, e.g.:

CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="/Users/jrr/repos/foo/amazon-freertos/vendors/espressif/boards/esp32/aws_demos/partition-table.csv"
CONFIG_PARTITION_TABLE_FILENAME="/Users/jrr/repos/foo/amazon-freertos/vendors/espressif/boards/esp32/aws_demos/partition-table.csv"

anurag-kar commented 5 years ago

Unfortunately that breaks ninja menuconfig unless I use an absolute path

@jrr Sorry about that! Here's a patch for allowing relative path of defaults files. Let me know if it works for you. I'll raise the relevant PR afterwords.

In the custom sdkconfig it's now necessary to specify the partition

That shouldn't be the case unless CONFIG_PARTITION_TABLE_CUSTOM=y is set in your custom defaults file. Make sure it's not set

jrr commented 5 years ago

@anurag-kar you were right about CONFIG_PARTITION_TABLE_CUSTOM, thanks.

Re: the absolute path, your patch fixes it for me. (though it wasn't hard to workaround it with e.g. -DIDF_SDKCONFIG_DEFAULTS=`pwd`/sdkconfig.my.defaults)

jrr commented 5 years ago

Next issue: how can I control which modules are included?

With the following dependencies in my CMakeLists.txt:

target_link_libraries(my_app PRIVATE AFR::mqtt)
target_link_libraries(my_app PRIVATE 3rdparty::jsmn)
target_link_libraries(my_app PRIVATE AFR::wifi)
target_link_libraries(my_app PRIVATE AFR::platform)
target_link_libraries(my_app PRIVATE AFR::dev_mode_key_provisioning)

I get this config:

====================Configuration for Amazon FreeRTOS====================
  Version:                 201906.00 Major
  Git version:             201906.00_Major-465-gc4fa1e50b

Target microcontroller:
  vendor:                  Espressif
  board:                   ESP32-DevKitC
  description:             Development board produced by Espressif that comes in two
                           variants either with ESP-WROOM-32 or ESP32-WROVER module
  family:                  ESP32
  data ram size:           520KB
  program memory size:     4MB

Host platform:
  OS:                      Darwin-18.7.0
  Toolchain:               xtensa-esp32
  Toolchain path:          /Users/jrr/esp/xtensa-esp32-elf
  CMake generator:         Ninja

Amazon FreeRTOS modules:
  Modules to build:        ble, ble_wifi_provisioning, common, crypto, defender,
                           dev_mode_key_provisioning, freertos_plus_tcp, greengrass,
                           kernel, mqtt, ota, pkcs11, pkcs11_implementation, platform,
                           secure_sockets, serializer, shadow, tls, wifi
  Enabled by user:         ble, ble_wifi_provisioning, defender, greengrass, mqtt, ota,
                           pkcs11, pkcs11_implementation, platform, secure_sockets,
                           shadow, wifi
  Enabled by dependency:   ble_hal, common, crypto, demo_base, dev_mode_key_provisioning,
                           freertos, freertos_plus_tcp, kernel, pkcs11_mbedtls,
                           secure_sockets_freertos_plus_tcp, serializer, tls, utils
  3rdparty dependencies:   jsmn, mbedtls, pkcs11, tinycbor
  Available demos:         demo_ble, demo_ble_numeric_comparison, demo_defender,
                           demo_greengrass_connectivity, demo_mqtt, demo_ota,
                           demo_shadow, demo_tcp, demo_wifi_provisioning
  Available tests:
=========================================================================

I'd like to exclude bluetooth entirely. I'm not sure whether it's trying to link in the demos, but I don't want those either.

tgsong commented 5 years ago

@jrr at the moment BLE is enabled by dependencies in demo code and cannot be turned off, I'm going to raise a PR to fix this.

Once this is fixed, on the CMake side, you can add -DAFR_ENABLE_ALL_MODULES=0 to your cmake command, or simply set it in your CMake file, set(AFR_ENABLE_ALL_MODULES 0 CACHE INTERNAL ""), this will turn everything off by default. If you want to turn on something, add -DAFR_MODULE_<module_name>=1 to your cmake command, or set it in your CMake file set(AFR_MODULE_<module_name> 0 CACHE INTERNAL ""). Dependencies should be automatically turned on, and module_name should be the name you see from the console output, normally they match the folder name where each module lives, for example mqtt, ble, be careful it's case sensitive.

kevinresol commented 5 years ago

Looks like there are some discussion going on. Is there a summary on how to setup a clean project?

kevinresol commented 5 years ago

I am trying to setup a repo to compile a "basic setup" project i.e. the demo application code with DEMO_RUNNER_RunDemos commented out)

P.S. Those include_directories cmake directives are hacky and IMO should be removed.

Anyway, the compiler is able to find all header and source files and build them. But it can't link because or the error: multiple definition of `prvBTManagerInit'

matteoscordino commented 5 years ago

Hello @kevinresol , I have the same problem, I created issue #1131 to see whether it can be solved. I have had a reply already, but it doesn't seem to solve it.

By the way, good job in putting together that example repo, it is going to be very useful as a TL;DR for people who stumble into this thread!

chenlijun99 commented 5 years ago

I have a similar problem. How can I use only the kernel part of this repository for my custom board? What I would like to be possible is to have this repository as a Git submodule and then link only the FreeRTOS kernel target to the executable. I know I can use the FreeRTOS repository hosted on SourceForge, which comprises only FrreRTOS and FreeRTOS+plus without all the AWS&IoT stuff, but it would be nice not having to rely on some unofficial GitHub mirror.

chenlijun99 commented 5 years ago

I've temporarily solved with this Find module, but it would awesome to have something officially supported!

chegewara commented 5 years ago

I am trying to add components to vendors/espressif/boards/esp32/aws_demos/application_code/components subfolder and i am almost there. Here is my CMakeLists.txt for one component:

set(AMAZON_FREERTOS_DIR "${AFR_MODULES_DIR}")
set(AMAZON_FREERTOS_ABSTRACTIONS_DIR "${AFR_MODULES_ABSTRACTIONS_DIR}")
set(COMPONENT_ADD_INCLUDEDIRS 
    include 
    example 
    ${AMAZON_FREERTOS_ABSTRACTIONS_DIR}/ble_hal/include 
    ${AMAZON_FREERTOS_DIR}/../demos/include/
)

# Edit following two lines to set component requirements (see docs)

set(COMPONENT_REQUIRES )
set(COMPONENT_PRIV_REQUIRES )

set(COMPONENT_SRCDIRS source example)

register_component()

and here are errors i cant get rid of. This one is cause i want to move shadow to my components folder:

vendors/espressif/boards/esp32/aws_demos/config_files/iot_config.h:89:31: fatal error: iot_config_common.h: No such file or directory

demos/include/iot_config_common.h:35:45: fatal error: platform/iot_platform_types_afr.h: No such file or directory

teuteuguy commented 5 years ago

Has there been any progress on this? Still trying to keep the amazon-freertos folder as is, and using sources on the side. Can't get it to work.

Using the following folder structure:

amazon-freertos
src
    main.c
CMakeLists.txt

Note: my main.c for now, is a direct copy of amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/main.c

Using the following content in CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)

project(my_app)

add_executable(my_app src/main.c)

include_directories(my_app PRIVATE "amazon-freertos/demos/include")
include_directories(my_app PRIVATE "amazon-freertos/demos/network_manager")

# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE my_app)

# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")

add_subdirectory(amazon-freertos)

target_link_libraries(my_app PRIVATE AFR::demo_mqtt)
target_link_libraries(my_app PRIVATE AFR::demo_ble)
target_link_libraries(my_app PRIVATE AFR::demo_ble_numeric_comparison)

target_link_libraries(my_app PRIVATE AFR::wifi)
target_link_libraries(my_app PRIVATE AFR::utils)
target_link_libraries(my_app PRIVATE AFR::ble)
target_link_libraries(my_app PRIVATE AFR::ble_hal)
target_link_libraries(my_app PRIVATE AFR::platform)
target_link_libraries(my_app PRIVATE AFR::dev_mode_key_provisioning)
target_link_libraries(my_app PRIVATE AFR::pkcs11)
target_link_libraries(my_app PRIVATE AFR::mqtt)
target_link_libraries(my_app PRIVATE 3rdparty::pkcs11)

Then:

cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake

=> generates the build folder

Then trying to run: make all -j4 (as per documentation) => FAIL with:

......main.c:44:30: fatal error: FreeRTOS_Sockets.h: No such file or directory
compilation terminated.
make[2]: *** [amazon-freertos/vendors/espressif/boards/esp32/CMakeFiles/aws_demos.dir/aws_demos/application_code/main.c.obj] Error 1

sameerdasarwad commented 5 years ago

look like no one find solution with amazon freertos for creating new project i try with all method given and followed all the process but still it wont work for me if any on have with edit demos or creating new app if anyone has solution would help i am new to github and freertos want to do some experiments, thanks for the help in advance

sameerdasarwad commented 5 years ago

@chegewara i am also too close and getting same error like you , have you got any solution for this or still there my error is looks little bit different

../demos/include/iot_config_common.h:35:50: fatal error: platform/iot_platform_types_freertos.h: No such file or directory

chegewara commented 5 years ago

@sameerdasarwad In my case problem is not exactly the same as described here. My project is in vendors/espressif/boards/esp32/aws_demos with components subfolder and i was able to build it with GNU make, but now client wants me to use cmake and here problems start. I have to try to remove aws logging and use esp_logx instead, because this causing dependency problem in components.

elieDaan commented 5 years ago

Have you found a solution @chegewara ? Why is such a mess to work with AWS !?

qiutongs commented 5 years ago

I am sorry that we haven't been able to provide a working solution and we understand the frustration. We are actively working on it. Once there is a progress, we will send an update as soon as possible.

tgsong commented 5 years ago

Hi, it's been 2 months since my last comment, we're sorry that this issue is still in pending status. I do want to share some recent update here which should resolve most of the issues.

First of all, many include path issues in this thread are due to that we were unable to turn off demos build before. Since ESP-IDF only links to one executable target which will be overwritten by users, we will run into problems when still trying to compile our demos. This is addressed in this PR #1161, which just got merged into master branch today. Thanks to @kevinresol who set up this example repo, https://github.com/kevinresol/amazon-freertos-hello-world, I've tested it with the submodule updated to our latest master branch, it builds and runs properly (note that you can delete the 2 includes on line 7 and 8 in the top level CMake file, it's not necessary).

However, we do still have 1 problem left to solve, which is the configuration header files. Right now, there's no solution to overwrite our library config headers in the above example repo. Even when turning off the demos, the config headers inside the demo project will still be used. We've been working on some proposals at this moment to solve this problem in a general way for all the boards and libraries. Once this is done, it should complete the whole picture.

@free-easy FreeRTOS kernel is now officially on GitHub, see here, https://github.com/FreeRTOS/FreeRTOS-Kernel @teuteuguy @chegewara @sameerdasarwad Let me know if the latest master branch solves your problems.

Jairo-Ortega commented 5 years ago

Hello,

Finally I can run my own application on an ESP32 board. A simple hello-world with amazon-freertos folder as a subdirectory of my project.

My project structure is the recommended in this thread:

amazon-freertos
src/main.c
CMakeList.txt

After this PR (https://github.com/aws/amazon-freertos/pull/1161), I still had some issues linking the application when I executed make in the build file.

To solve that I had to make the following changes:

amazon-freertos/vendors/espressif/esp-idf/components/esp32/cpu_start.c has troubles with app_main(), like it doesn't know where is that method, so I did:
- Document line 91. So, "extern void app_main(void);" change to "//extern void app_main(void);"
- Add a reference to my main, so I include it with #include "../../../../../../src/main.c" With this I "said" to cpu_start.c use the specific (my) app_main().
In my main.c I copy the methods vApplicationGetIdleTaskMemory and vApplicationGetTimerTaskMemory from "/amazon-freertos/vendors/vendor/boards/board/aws_demos/application_code/main.c". Also, I change the relative routes of "iot_system_init.h" to "../amazon-freertos/libraries/freertos_plus/standard/utils/include/iot_system_init.h" and "iot_logging_task.h" to "../amazon-freertos/libraries/c_sdk/standard/common/include/iot_logging_task.h".

Attached you will find LinkingIssues.txt with the issues when I ran make (the output), my main.c and the CMakeList.txt.

CMakeLists.txt LinkingIssues.txt main.c.zip

tgsong commented 5 years ago

@JairoDaniel Would you mind sharing your CMakeLists.txt file? Looks like your main.c file is somehow not compiled.

tgsong commented 5 years ago

@JairoDaniel I see you have this line in your CMake file, file(GLOB SOURCES "hello/main.c"). However, you said your project structure has src/main.c. Can you double check if that main.c is under src or hello? CMake won't tell you if the SOURCES variable is empty or not.

Jairo-Ortega commented 5 years ago

@tgsong thank you for your annotation.

I don't have a folder hello (only src), this is a mistake.

Actually if you document this line works because that file doesn't exist. If I change CMake to file(GLOB SOURCES "src/main.c") whit actual configuration I will get an error. So, in the CMake I put file(GLOB SOURCES "src/main.c") and change the amazon-freertos/vendors/espressif/esp-idf/components/esp32/cpu_start.c as the original (without include my main and with the extern app_main). With that it is ok

monkeytronics commented 5 years ago

@JairoDaniel If I understand you, you have succeeded in running a hello world program for ESP32? I can't understand the CMakeLists.txt. How does the following work when the path hello/main.c doesn't exist? Is there a disconnect in my understanding of this thread somewhere?

file(GLOB SOURCES "hello/main.c")
add_executable(${CMAKE_PROJECT_NAME} ${SOURCES})

@tgsong I'm planning to have a go at this. I also want to be able to build a project outside of the AFR directory. The ideal starting point would indeed be to a project outside the AFR directory, but not just an empty hello world: a project bringing in one of the demos. Do you have any pointers or progress you can share? Is this usage scenario typical??

Jairo-Ortega commented 5 years ago

@monkeytronics in the first post of my solution I didn't notice that I had change the name of my source code (hello to src) and didn't in the cmakelist. At the end, my project works because I made some changes in the Amazon FreeRTOS code (I post those changes in my first comment).

After @tgsong told me the mistake I solved the problem in the CMakeList and with that the Amazon FreeRTOS code stay as the original.

As a resume I made this:

In my main.c I copy the methods vApplicationGetIdleTaskMemory and vApplicationGetTimerTaskMemory from "/amazon-freertos/vendors/vendor/boards/board/aws_demos/application_code/main.c". Also, I change the relative routes of "iot_system_init.h" to "../amazon-freertos/libraries/freertos_plus/standard/utils/include/iot_system_init.h" and "iot_logging_task.h" to "../amazon-freertos/libraries/c_sdk/standard/common/include/iot_logging_task.h".

Attached you will find my main.c and my CMakeList.txt

CMakeLists.txt main.c.zip

tgsong commented 5 years ago

@monkeytronics I've put up an example repo for this, https://github.com/tgsong/amazon-freertos-examples

Basically I copied the main.c file from the amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code directory, and tell it to links the MQTT demo. If you build and run it with correct set up, it should successfully completes the MQTT demo.

Note that I created an amazon-freertos-config folder at the top level, this is a temporary solution for the config file problems that I mentioned before,

However, we do still have 1 problem left to solve, which is the configuration header files. Right now, there's no solution to overwrite our library config headers in the above example repo. Even when turning off the demos, the config headers inside the demo project will still be used. We've been working on some proposals at this moment to solve this problem in a general way for all the boards and libraries. Once this is done, it should complete the whole picture.

In the CMakeLists.txt, I added include_directories(BEFORE amazon-freertos-configs) on line 21, which should let compiler pick up this folder first instead of the config folder inside amazon freertos. You can update your thing name, client credentials, and WiFi credentials in those config files.

chegewara commented 4 years ago

@tgsong Big thank you for sharing this example repository. Works great, just need to read README, because standard cmake command that is used with AFR is different than used with your example.

tgsong commented 4 years ago

Thank you all for providing valuable feedback and contribution on this topic, we've made great progress so far because of your effort. I'll work with our doc team in the meantime to update our public document.

chegewara commented 4 years ago

@tgsong I have one issue i cant solve, maybe its because of my code or missing some includes in CMakeLists.txt:

CMakeFiles/afr_demo.dir/amazon-freertos/demos/network_manager/aws_iot_network_manager.c.obj:(.literal._bleRegisterUnregisterCb+0x4): undefined reference to `BLEGAPPairingStateChangedCb'
CMakeFiles/afr_demo.dir/amazon-freertos/demos/network_manager/aws_iot_network_manager.c.obj:(.literal._wifiEnable+0x0): undefined reference to `xWiFiConnectTaskInitialize'
CMakeFiles/afr_demo.dir/amazon-freertos/demos/network_manager/aws_iot_network_manager.c.obj: In function `_wifiEnable':
/home/chegewara/upwork/leaksentinel2/build/../amazon-freertos/demos/network_manager/aws_iot_network_manager.c:926: undefined reference to `xWiFiConnectTaskInitialize'
collect2: error: ld returned 1 exit status

Missing functions are here: https://github.com/aws/amazon-freertos/blob/master/demos/include/iot_ble_numericComparison.h#L46 https://github.com/aws/amazon-freertos/blob/master/demos/include/aws_wifi_connect_task.h#L42 I have already included few AFR libraries:

target_link_libraries(afr_demo PRIVATE AFR::demo_mqtt AFR::shadow AFR::ota AFR::wifi AFR::ble AFR::ble_wifi_provisioning)

Any help is welcome.

tgsong commented 4 years ago

@chegewara The missing functions are defined in demos code, can you try also linking AFR::demo_ble_numeric_comparison and AFR::demo_wifi_provisioning? They are defined in here,

chegewara commented 4 years ago

@tgsong You are the best, thanks a lot man. That works like a charm.

elieDaan commented 4 years ago

Hello, First of all thanks a lot for all good news for adding a new project.

I've got a problem. I succeed to add custom components few weeks ago but now I need to add c-JSON to my project. And the component is impossible to add.. I also try to just add 'component/foo' from amazon-freertos-examples and it says : ../main/menu_test.h:61:17: fatal error: foo.h: No such file or directory

However I succeed to build the amazon-freertos-examples repo.

Thanks a lot for your help.

This is my CMakeLists :

cmake_minimum_required(VERSION 3.13)

project(XX)

file(GLOB SOURCES 
"/XXX/main/*.c" ABSOLUTE 
    #"amazon-freertos/demos/demo_runner/aws_demo.c" 
    #"amazon-freertos/demos/demo_runner/iot_demo_freertos.c" 
    #"amazon-freertos/demos/demo_runner/aws_demo_network_addr.c" 
    "amazon-freertos/libraries/freertos_plus/standard/utils/src/iot_system_init.c"
    )
file(GLOB HEADERS 
"/XX/main/*.h" ABSOLUTE
)
add_executable(XX ${SOURCES} ${HEADERS})
# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE bob_v6)

#Define the components
get_filename_component(ESPRESSIF_CODE_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/amazon-freertos/vendors/espressif/boards/esp32/aws_demos/application_code/espressif_code" ABSOLUTE)
get_filename_component(ESP_SPIFFS_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/spiffs" ABSOLUTE)
get_filename_component(ESP_SPIFFS_IMG_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/spiffs_image" ABSOLUTE)
get_filename_component(ESP_MKSPIFFS_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/mkspiffs" ABSOLUTE)
get_filename_component(ESP_SPIDRIVER_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/spidriver" ABSOLUTE)
get_filename_component(ESP_TFT_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/tft" ABSOLUTE)
get_filename_component(MQTT_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/mqtt_cartridge" ABSOLUTE)
get_filename_component(FOO_COMPONENT "${CMAKE_CURRENT_LIST_DIR}/components/foo" ABSOLUTE )
get_filename_component(JSON_COMPONENT "components/json" ABSOLUTE)

set(EXTRA_COMPONENT_DIRS    
                            ${FOO_COMPONENT}
                            ${ESP_TFT_COMPONENT}
                            ${ESP_SPIFFS_COMPONENT}
                            ${ESP_SPIFFS_IMG_COMPONENT}
                            ${ESP_MKSPIFFS_COMPONENT}
                            ${ESP_MKSPIFFS_COMPONENT}
                            ${ESP_SPIDRIVER_COMPONENT}
                            ${MQTT_COMPONENT}
                            ${JSON_COMPONENT}
)

# Add some external components to the project
# set(IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS}
#     ${ESPRESSIF_CODE_COMPONENT}
# )

list(APPEND IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

# Bring in the include directories 
set(INCLUDE_DIRS ${INCLUDE_DIRS} 
    "amazon-freertos/vendors/espressif/esp-idf/components/spi_flash/include"
    "amazon-freertos/demos/include/"
    "amazon-freertos/libraries/freertos_plus/standard/freertos_plus_tcp/include/"
    )
include_directories(${INCLUDE_DIRS})

include_directories(bob_v6 PRIVATE "amazon-freertos/demos/include")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/abstractions/platform/freertos/include")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/freertos_plus/standard/utils/include")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/freertos_plus/standard/utils/src")
include_directories(bob_v6 PRIVATE "amazon-freertos/demos/dev_mode_key_provisioning/include/")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/abstractions/pkcs11/include/")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/3rdparty/pkcs11/")
include_directories(bob_v6 PRIVATE "amazon-freertos/demos/network_manager/")
include_directories(bob_v6 PRIVATE "amazon-freertos/libraries/abstractions/wifi/include/")

#Set SDK Config Defaults
set(IDF_SDKCONFIG_DEFAULTS "${CMAKE_CURRENT_LIST_DIR}/sdkconfig.defaults")

# Bring in the FreeRTOS code
# AFR_BOARD tells which board we need to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")
set(EXTRA_CFLAGS --save-temps -lm)
add_subdirectory(amazon-freertos)

message("++++++++++++++++++++++++++++++++++")
message("++++          MY APP          ++++")
message("++++++++++++++++++++++++++++++++++")

message("IDF_EXTRA_COMPONENT_DIRS: ${IDF_EXTRA_COMPONENT_DIRS}")

message("INCLUDE_DIRS: ${INCLUDE_DIRS}")

message("Glob Source : ${SOURCES}")

message("Glob headers : ${HEADERS}")

target_link_libraries(XX PRIVATE AFR::demo_mqtt)
target_link_libraries(XX  PRIVATE AFR::demo_ble)
target_link_libraries(XX  PRIVATE AFR::demo_ble_numeric_comparison)

target_link_libraries(XX  PRIVATE AFR::wifi)
target_link_libraries(XX  PRIVATE AFR::utils)
target_link_libraries(XX  PRIVATE AFR::ble)
target_link_libraries(XX  PRIVATE AFR::ble_hal)
target_link_libraries(XX  PRIVATE AFR::platform)
target_link_libraries(XX  PRIVATE AFR::dev_mode_key_provisioning)
#target_link_libraries(XX  PRIVATE AFR::pkcs11)
target_link_libraries(XX  PRIVATE AFR::mqtt)
target_link_libraries(XX  PRIVATE 3rdparty::pkcs11)

tgsong commented 4 years ago

@elieDaan Is your executable actually called XX or that's just a placeholder? The name you put in add_executable on line 15 and in set on line 17 must match, basically like this,

add_executable(bob_v6 ${SOURCES} ${HEADERS})
# Tell IDF build to link against this target
set(IDF_PROJECT_EXECUTABLE bob_v6)

Also from line 59 to 67 you need to use target_include_directories if you want to set include path to your executable target. include_directories is for global settings. I also noticed that you're just referring to the include paths inside Amazon FreeRTOS, so those calls could be removed as long as you're linking the right target when calling target_link_libraries. For example, when you call target_link_libraries(bob_v6 PRIVATE AFR::wifi), amazon-freertos/libraries/abstractions/wifi/include/ should be automatically brought in.

monkeytronics commented 4 years ago

Since we are trying to get this is good shape to go to production, I've had a go at referencing the certificates directly from text files. The point is to make it easy to create certificates for many devices scripted & easily include them in the build in the NVS section of memory such that it can be separately flashed.

The ESP IDF has a bit on this: https://docs.espressif.com/projects/esp-jumpstart/en/latest/remotecontrol.html#embedding-files-in-the-firmware This tells us how to :

Save the certificates e.g. in folder: cloud_cfg/server.cert etc.
Define extern const uint8_t certificate_pem_crt_start[] asm("_binary_server_cert_start");
component.mk -> COMPONENT_EMBED_TXTFILES := folder/server.cert
CMakeList.txt -> target_add_binary_data(${COMPONENT_TARGET} "cloud_cfg/server.cert" TEXT)

Then in the code, we can just refer to the pointer certificate_pem_crt_start to get the certificate. I've tried to do this here but I honestly don't know CMake well enough to get it to work. Is this possible in Amazon FreeRTOS.

aws / amazon-freertos

Document how to create a new cmake project outside the freertos source tree #832

Edit: flashing