Document how to create a new cmake project outside the freertos source tree

[jrr]

Is your feature request related to a problem? Please describe.
The Getting Started Guides describe how to run the demo apps, but I can't find anything about creating a new project.

Describe the solution you would like.
Documentation describing how to create a new project outside the amazon-freertos source tree, and guidance about how to consume freertos from outside.

Additional context
With the old Make-based system, I was able to create a new ESP32 project that references amazon-freertos from outside, including this repo as an unmodified git submodule.

(This wasn't documented before, either, so it took significant effort!)

Now with the new cmake system I feel like I have to start all over.

I'm looking for something like: 1) Copy files X, Y, and Z into a new directory 2) Export an AMAZON_FREERTOS_PATH environment variable pointing the root directory of the amazon-freertos repo 3) Issue command cmake -D... to build your application image.

Is this documented anywhere? Am I missing something?

[chegewara]

@monkeytronics Why you are choosing this option? It is easier just to create NVS partition with certificates and thing name and flash it with 0x320000 address. You wont have to change AFR code to much, just load thing name on start and dont use vDevModeKeyProvisioning();

https://docs.espressif.com/projects/esp-idf/en/v3.1.5/api-reference/storage/nvs_partition_gen.html

[monkeytronics]

@chegewara Did you get that to work with certificates. I've just tried and the nvs_partition_gen.py script isn't working!

UPDATE:

I found I had to use the script from a more recent version of the ESP-IDF. The one included in the latest Espressif ESP-IDF worked perfectly. Bug perhaps with this version?! Don't know.

[chegewara]

@monkeytronics Yes, but i am using scripts from this repository, just a little bit changed to store DER certificates in NVS instead of PEM: https://github.com/aws-samples/aws-iot-jitp-sample-scripts/tree/master/bin

I have it for production ready product, but client changed scripts for his needs.

[monkeytronics]

@chegewara I can't find anything in there for creating a partition. Sure you got the right repo?

[chegewara]

Sorry, this is last script i added to the others:

#!/bin/bash
#
# AWSFreeRTOS partition "storage", size 0x10000, address 0x320000

THING=${1:-thing}

# python ~/esp/esp-idf/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py --input single_sample.csv --output ../sample.bin --size 0x10000 --version v1

python ~/esp/esp-idf/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py generate single_sample.csv ../sample.bin 0x10000 --version 2

with sample csv:

key,type,encoding,value
creds,namespace,,
P11_Key,file,binary,../keys/thing_name.key.der
P11_Cert,file,binary,../certs/thing_name.crt.der
info,namespace,,
nvs_key_thing_name,data,string,thing_name

# TODO edit path to der files

[elieDaan]

@tgsong It's just a placeholder ;) Replacing include_directories by target_include_directories cause me trouble for my mqtt component with the famous ''' In file included from ../amazon-freertos/vendors/espressif/boards/esp32/aws_demos/config_files/iot_config.h:76:0, from ../components/mqtt_cartridge/mqtt_cartridge.h:18, from ../components/mqtt_cartridge/mqtt_cartridge.c:11: ../amazon-freertos/demos/include/iot_config_common.h:35:50: fatal error: platform/iot_platform_types_freertos.h: No such file or directory '''

Do you think the error could came from target_include_directories/include_directories ?

[tgsong]

@elieDaan Just to make sure, so all the XX in the CMake file you posted are actually bob_v6, the name that I see you used in set(IDF_PROJECT_EXECUTABLE bob_v6)? I tried your CMake file with all the XX placeholder replaced with bob_v6 and I removed all the include_directories because they're all referring to the directory inside AFR which is not needed anyway, and turns out it builds fine for me. Can you try start from a working project first and then add more components? Without seeing the complete structure and your CMake file it's not that straight forward to figure out where it went wrong, if you are OK to upload your code to a repository I can also check for you.

[monkeytronics]

CMakeLists.txt @tgsong I am pretty sure I have the same problem. What I was hoping to do was copy the contents of the OTA demo code into a custom folder (call it ota and put it next to foo). I managed to get my core application to run fine in a few folders next to the foo folder. I copied the example in the CMake for foo. It worked.

get_filename_component( EXTRA_COMPONENT_DIRS "components/sensorA" ABSOLUTE )
list(APPEND IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

If I now try to copy and paste the contents of aws_iot_ota_update_demo.c into my new file ota.c and create a new folder structure exactly like foo. And then add:

get_filename_component( EXTRA_COMPONENT_DIRS "components/ota" ABSOLUTE )
list(APPEND IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

The folder seems to build but ninja fails when it gets to the includes in the code.

CMakeFiles/idf_component_ota.dir/src/ota.c.obj -MF amazon-freertos\esp-idf\ota\CMakeFiles\idf_component_ota.dir\src\ota.c.obj.d -o amazon-freertos/esp-idf/ota/CMakeFiles/idf_component_ota.dir/src/ota.c.obj   -c ../components/ota/src/ota.c
In file included from ../components/ota/src/ota.c:38:0:
../amazon-freertos-configs/iot_config.h:76:31: fatal error: iot_config_common.h: No such file or directory
compilation terminated.

Oddly it seems to find a different iot_config.h than the one main.c finds. But then fails inside that file anyway. If I comment out that tempararily, it can't find ../components/ota/src/ota.c:41:22: fatal error: iot_mqtt.h: No such file or directory either! I've got a bit of a mess!

Actually the one in main.c has squiggles under it and this error on hover:

#include errors detected. Please update your includePath. Squiggles are disabled for this translation unit (C:\Users\thebi\FreeRTOS\src\main.c).C/C++(1696)
cannot open source file "xtensa/tie/xt_core.h" (dependency of "iot_config.h")C/C++(1696)

Also there's a squiggle under #include "nvs_flash.h" too with the error:

cannot open source file "stdarg.h" (dependency of "nvs_flash.h")

Any help getting this to build properly so I can run the code from the ota demo in it's own task. And continue debug from there... Sorry, it's such a mess!

[tgsong]

Hi @monkeytronics I've managed to do what you want. Basically you're having 2 problems here,

1. You need to specify dependencies for your OTA components, not your application target.
2. Our OTA demo code depends on something from our demos directory, you need to add that too.

Replace your OTA components CMakeLists.txt with this,

# source files of this components.
set(COMPONENT_SRCS src/ota.c)

# add this components, this will define a CMake library target.
register_component()

# the important part, specify dependencies for this component, COMPONENT_TARGET is a
# variable defined by ESP-IDF when you call register_component().
target_link_libraries(
    ${COMPONENT_TARGET}
    PRIVATE
        # Obviously this components depends on OTA.
        AFR::ota
        # MQTT is a private dependency for OTA, so it's not automatically included here,
        # because MQTT is not exposed in any OTA public header. However, the OTA demo
        # needs MQTT to initialize the connection, so you need it here.
        AFR::mqtt
        # This demo depends on some stuff from our demos directory like common config,
        # network manager, etc.
        AFR::demo_base
)

However, I don't recommend you directly extract our current demo code into a component without any modification for now because the demo dependencies are kinda messy and subject to change. For example, we're planning to take network manager out of demos directory, and refactor our configuration files, so in the future you shouldn't need the AFR::demo_base dependency as this is never intended to be public.

[monkeytronics]

@tgsong. Thanks. So sounds like this is definitely not the way. Which is good because as you say the demos are hard baked and subject to change. So the real question is how to embed the ota as per the documentation. https://docs.aws.amazon.com/freertos/latest/userguide/integrate-ota-agent.html gives an example but its incomplete. It does state for brevity stuff is left out. Speaking as an intermediate level coder, I feel that brevity is less important than completeness. It would be great if the example could be expanded including all the build requirements.

The real problem to my mind is that every 'how do I do ' question is answered by : see the demo. But taking the example of ota, as you say that code doesn't copy and paste. In fact there are complex dependencies and its intrinsically not the right way to use the afr library. Also, it uses completely different mqtt client code. Citing one thing as an example of another when they are different is very confusing. I appreciate you guys are working so hard to provide an incredible and almost universal library. And I am in awe of this and can hardly imagine how difficult it is. Please accept my suggestions as humble user feedback of a lesser mortal: not a criticism in any way.

[chegewara]

@monkeytronics I have project in which i changed few things in OTA and shadow examples and moved them to new files, which are in esp32 main folder. Recently i was able to move whole project from release-1.5 and standard folder system to this one from @tgsong repository (AFR as component). I had to make some changes in CMakeLists but it works. I think this is most important part CMakeLists.txt file and should works for you too:

target_link_libraries(afr_demo PRIVATE AFR::demo_mqtt AFR::shadow AFR::ota AFR::wifi AFR::ble AFR::ble_wifi_provisioning AFR::demo_ble_numeric_comparison AFR::demo_wifi_provisioning)

[tgsong]

@monkeytronics Thank you very much for you feedback, we really appreciate it. As you pointed out, we still have a lot to improve on usability. As of now, our demo code is tightly integrated into our project, it demos how our code works, but it fails to demo how our user could consume our library. And we haven't yet provided enough document to show how a user could set up a project and consume our library as a dependency. The team is aware of this problem, and your feedback is really important to us as it will help us to prioritize it.

As for that specific OTA demo document, you're right, it is out-dated and it uses our old MQTT library. It is already on our radar, and will be updated with the upcoming releases.

[monkeytronics]

Thanks guys. @chegewara Well Done getting it to work!

So the correct approach for using AFR is to create it as a component, and never reference anything from the demo code. I.e. we never need to define CONFIG_OTA_UPDATE_DEMO_ENABLED or any of the other demos! Is this correct?

Can I check, when you said the OTA demo document is outdated, do you mean that the fragment in this link is wrong : https://docs.aws.amazon.com/freertos/latest/userguide/integrate-ota-agent.html, and the OTA demo is correct? In which case, how do we handle things like vMqttDemoDeleteNetworkConnection() etc which are dependent on the demo. We end up in a a mess where we have to enable CONFIG_OTA_UPDATE_DEMO_ENABLED in order to run OTA? This must be wrong!

To help us keep moving forwards right now, @tgsong would it be possible to supply us with a correct working example and the necessary build requirements. Based on the size of the example in the link above, hopefully, it's actually not too much code.

Are the examples for using MQTT, BLE etc up to date. Because they are next up.

[tgsong]

@monkeytronics Yes, the correct way is to only consume the library code, and never reference demos. However, you can copy our demo code and turn it into your own code. Having said that, we recognize it is not easy for our customers to do this way.

For OTA demo, yes, the code pieces in our public document is outdated. And the code in our repository is correct. There are some legacy reasons for things like vMqttDemoDeleteNetworkConnection(). Our OTA demo code was made before the network manager was created, and we're planning to migrate it to use the network manager, so that we can remove all the CONFIG_OTA_UPDATE_DEMO_ENABLED used in the OTA demo. That macro is intended to conditionally switch which demo function to run in our demo project. As a user when making your own project, you are correct, it should never be needed.

We're working on updating the OTA component in our library right now, and I'll provide update once the above problems are fixed. For MQTT and BLE, they're update to date, I haven't checked how easy it is to extract their demo out though.

[JoshuaRack]

Thanks or all the great work here. It would be good to get something into the docs asap, even if its very provisional and prefaced with warnings about future changes etc. I followed the current getting started guide and was pleased to quickly get the demo up. Then came much confusion as I tried to figure out how to move over from my current esp-idf build... tgsong sums it up perfectly above with "we haven't yet provided enough document to show how a user could set up a project and consume our library as a dependency". For now I've used up my time budget to evaluate aws iot on esp32, hopefully by next time things will be different.

[tgsong]

We've updated our ESP32 getting started document with the information in this thread, you can find it here, https://docs.aws.amazon.com/freertos/latest/userguide/getting_started_espressif.html#getting_started_espressif_cmake_project

I'm closing this issue now, thank you for all the valuable feedback, you're welcome to open new issues if you have further questions.

[chegewara]

@anurag-kar Thanks for clearing up the confusion. Having a IDF_SDKCONFIG_DEFAULTS variable to allow customer to override would be great, the patch looks good to me.

I still have a question though, we used to run menuconfig target to change some settings, which stores the value in sdkconfig fie. however, since this file is generated each time by CMake now (before it was persistent), that will make menuconfig useless right? I remember I asked for this and the reason for this behaviour is to be able to switch between multiple projects (in our case demos and tests). I can think of 2 solutions here,

1. Do not regenerate the sdkconfig file if it's already in the build folder (similar to the previous behaviour). If we want to switch the project, just delete the build folder. We couldn't do this before because the sdkconfig was not generated in the build folder but instead under the root folder.
2. Detect if the user has changed the path of sdkconfig.default. For example, user may have multiple application targets and each one has its own sdkconfig.default. If user rerun CMake with different IDF_SDKCONFIG_DEFAULTS value, then regenerate the sdkconfig in the build folder.

Hi, i know the issue is closed for some time, but i think this change in PR #987 is not working properly. I am using AFR as component in esp-idf and i would like to use sdkconfig.defaults: 1) using to build: cmake -S . -B build -DIDF_SDKCONFIG_DEFAULTS=sdkconfig.my.defaults -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake -GNinja 2) setting from sdkconfig.my.defaults are used and settings are set in sdkconfig, 3) this is what is not working: i cant open menuconfig because:

Loading defaults file sdkconfig.my.defaults...
Traceback (most recent call last):
  File "/home/chegewara/upwork/leaksentinel2/amazon-freertos/vendors/espressif/esp-idf/tools/kconfig_new/confgen.py", line 338, in <module>
    main()
  File "/home/chegewara/upwork/leaksentinel2/amazon-freertos/vendors/espressif/esp-idf/tools/kconfig_new/confgen.py", line 99, in main
    raise RuntimeError("Defaults file not found: %s" % name)
RuntimeError: Defaults file not found: sdkconfig.my.defaults
FAILED: amazon-freertos/CMakeFiles/menuconfig 

Can it be confirmed? I am using AFR commit: 306c3bca9300af788ba2be82d52063da162ff294 Thanks

EDIT i found that using -DIDF_SDKCONFIG_DEFAULTS= with absolute path will works or

cmake -S . -B build -DIDF_SDKCONFIG_DEFAULTS=$PWD/sdkconfig.my.defaults -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/tools/cmake/toolchains/xtensa-esp32.cmake -GNinja