Using Custom user_settings.h with wolfSSL in HomeKit after Version Update

Version

5.7.2

Description

Hi everyone and @gojimmypi,

I'm working on an ESP-IDF HomeKit project and using wolfSSL as a managed component to simplify version updates. I've added wolfSSL as a component by including it in components/homekit/idf_component.yml, which makes it easier to update just by adjusting the version number.

Screenshot 2024-09-11 at 18 21 24

In the components/homekit/user_settings.h file, I’ve set the necessary configurations to ensure smooth compilation.

Screenshot 2024-09-11 at 18 21 47

However, my challenge is maintaining these custom settings across version updates. After updating to a new version of wolfSSL (e.g., from 5.7.2 to 5.7.3), I'd like my user_settings.h file to be used by default, without requiring additional manual steps.

Screenshot 2024-09-11 at 18 22 03

Here’s a summary of what I've done so far:

wolfSSL is added as a managed component in components/homekit/idf_component.yml Custom configurations are set in components/homekit/user_settings.h wolfSSL is automatically downloaded as a managed component during the build process.

Question:

What adjustments or changes can I make to ensure my custom user_settings.h file is used automatically after updating the wolfSSL version in idf_component.yml? In other words, how can I ensure the project works "out of the box" with custom configurations after a version update?

Any suggestions or insights on how to approach this would be greatly appreciated!

Thanks in advance.

Best regards, Achim Pieters

Hi @AchimPieters

Thank you again for all your help in getting wolfSSL working well with your awesome ESP32 Apple HomeKit examples. Not only is that a cool project, but your use-case has been helpful in the design of the wolfSSL Managed Component.

What adjustments or changes can I make to ensure my custom user_settings.h file is used automatically ?

That's an excellent question! The answer is that to use the Managed Components, a custom user_settings.h is not possible at this time. The user_settings.h is currently part of the component; If the component manager sees that any part of the component has changed, then the component needs to be converted to a non-managed component. This of course defeats the purpose and value of having a Managed Component in the first place.

In order to customize wolfSSL for a given project, the current plan is to use the Espressif method of having all settings defined via the idy.py menuconfig. This means having sdkconfig.defaults available with an appropriate Kconfig file as described in the Project Configuration Documentation.

I suppose in theory, it might be possible to have a cleverly-crafted CMakeLists.txt that points to a wolfSSL user_settings.h in a user-defined directory outside of the component. I could see that as being both beneficial, but also potentially problematic. I'll need to experiment a bit with that. The trick would be having a user settings both inside and outside of the component. Perhaps some sort of "supplementary user settings file"? I don't know.

For now, please take a look at my current reference template app. Introduced in https://github.com/wolfSSL/wolfssl/pull/7866 I have many new settings available in the respective Kconfig file that will allow use of menuconfig and the respective sdkconfig.default files.

Please let me know what settings you need that are not included in the template app ASAP.

I'm expecting a wolfSSL release in the relatively near future (and a Managed Component update shortly thereafter). I'd like to add any relevant settings that the Apple HomeKit support might need. It would be great to have those included in the next release.

Like your examples, I have many different ESP32 examples, not only here in the wolfSSL repo, but also:

I would also like to have a common user_settings.h, CMakeLists.txt, and Kconfig in the components/wolfssl directory used by all of the examples.

how can I ensure the project works "out of the box" with custom configurations after a version update?

I think the best method is with the sdkconfig.defaults and respective support in the user_settings.h file as well as in settings.h (see also https://github.com/wolfSSL/wolfssl/pull/7953 and the related changes to settings.h).

Note that the template user_settings.h already has a specific Apple HomeKit Section ready for any defaults across all HomeKit apps.

I'm interested in your feedback and suggestions.

Cheers

Hi @gojimmypi

Thank you for your detailed response. I’ve attached my current user_settings.h file.

user_settings.h.txt

After going through your message, I agree that using a sdkconfig.defaults OR Kconfig file could indeed be the solution here. Where sdkconfig.defaults has my preference.

This way, you can set up the necessary configurations per project, outside of the wolfSSL managed component. This would allow for seamless updates to the latest version of wolfSSL without having to manually adjust everything.

One idea could be to include a sample sdkconfig.defaults file as a template, which could be copied for each project and modified as needed. This would make it easier for users to customize settings while maintaining the benefits of the managed component.

Looking forward to hearing your thoughts on this approach!

Cheers

Hi @gojimmypi,

Have you made any progress on the above?

Hi @AchimPieters , no, not yet. Sorry for the delay.

I've been working on improving the esp-tls with wolfSSL in #7936. See also the related settings.h changes there.

If you could take a look at my template and/or esp_http_client_example and let me know which settings are missing, that would be very helpful.

I'm also taking any other suggestions, and even pull requests.

Otherwise I'll try to get to that as soon as I can. Thanks for your patience. I'd definitely like to have this update in the next release of wolfSSL that I suspect will be in the coming weeks.

About WolfSSL (WS) in ESP idf and managed components

Let me try to make an analogy with a pizza restaurant.

There are pizzas on the menu and pizzas where you select your own toppings. A crossover also exists with a menu pizza with extra toppings.

Now, WS as a managed component is menu pizza. We can only get it as a fixed item and with the template initiative, WS creates the menu. If we want to mess with the toppings, we have to invade the kitchen.

If we use WS as a git component, we can build our pizza and choose all the toppings. This gives us maximum flexibility. And the git component system gives us the possibility to adjust WS or upstream components any way we want in our own fork.

The key problem I have as an inventor is the inflexibility of tinkering with the toppings in managed mode. If there is a new version of WS, it will overwrite my settings, so I will avoid that , which is bad also. And once I have created my perfect recipe, it will take a lot of effort to get it on the menu of the restaurant.

The other side is that the offering for copy-cats will be easier to use.

In my opinion, there are no copy-cats in the target group of WS. There are inventors that integrate WS and then there are copy-cats that use these inventions. Like the HomeKit stack Achim ported and where I have been able to contribute. Those contributions worked because the component system, but would be terrible in the managed component style. This is because the managend components are no longer git aware of their own, but become part of the main project. That last bit makes source tracking difficult and bloats the project repository.

Now, to make it worse, the HomeKit stack needs to provide the settings for WS, not the main project that uses HomeKit as a component.

So, what to do?

I would want the settings of WS to be taken from a location set in the component that includes it in the dependency file. And maybe also, a layered approach where every higher layer can add a few details to the settings.

How to do that, I do not know yet, but I think the current approach is not appealing to inventors...

And one more thing: it took me an hour to make sure that what was in the managed component was actually derived 1:1 from the official WS repository. With security in mind, we do not want a backdoor to be installed by Espressif without detection.

@gojimmypi, I hope this helps in your quest to make your great product even better!

My 2 cents, M10

Hi @M10tech and thank you for your feedback!

Yes, you have some completely valid and accurate points.

managed components are no longer git aware of their own, but become part of the main project. That last bit makes source tracking difficult and bloats the project repository.

This in particular is unfortunately very true. The Managed Components definitely have there place: making it easy to add wolfSSL library to a project and get started easily. The unfortunate thing, as you pointed out, is that the code is completely disconnected from GitHub source control. In fact, it is even painful to make the simplest changes, as the component is then no longer "managed".

As for bloat, well, the installed files could be added to .gitignore, but otherwise I agree that the files add to project size. It does, however, completely contain everything in a simple, easy-to-use example.

we do not want a backdoor to be installed by Espressif without detection

This seems unlikely, but then once "published" you are correct that there's currently no way to prove the code is the same as-published. When the publishing is completed, it is all in a tgz package from my (the publisher's) perspective when uploaded. I suspect there's a way to have a hash to prove the downloaded library is the exact same as published. I'll need to do a bit more homework on this. This is an excellent suggestion, thanks for pointing it out.

This same problem exists of course, when wolfSSL is copied to the HomeKit repo. Actually, this is probably even worse from a security perspective.

In a commercial situation, we can provide guidance to ensure code integrity and reliability.

it took me an hour to make sure that what was in the managed component was actually derived 1:1 from the official WS repository

I am open to suggestions how to make this easier, in particular if I don't find a hash feature as described above for the Managed Component. I could perhaps create my own in-directory hashes at publish time and include the hash fingerprint in an official wolfSSL source.

So, what to do?

Well, like many things: it depends. I would probably not use a Managed Component in a project actively in development. As you noted, the disconnect from GitHub and inability to contribute upstream is a pretty big disadvantage.

In the case of the Apple HomeKit project, @AchimPieters really wants to make things as simple for new users as possible. I get it: that's important. I'm trying to discourage from having an entire local copy of wolfSSL in the project. (mainly for all the same issues of source integrity, and contributing to upstream). If you think the Managed Component adds bloat, OMG, a copy of the entire repo is even worse! lol

Using a Managed Component seemed to be a good way to accomplish "using wolfSSL but not having an entire copy".

There's also of course the option of having a submodule, the way the Espressif esp_wolfssl does. There's a place for that. I'm not particularly fond of the idea for ESP32 projects. It also adds a bit of complexity for the new user.

Me, well I use my development branch of wolfSSL when working on any project. A very small number of files are needed in the local project. See the template example.

All my projects are in a workspace directory:

workspace/wolfssl
workspace/myproject

The CMakeLists.txt can find wolfSSL in a parent directory, or have a specific location by setting WOLFSSL_ROOT. This is great for development, but admittedly a bit more complex for things like the Apple HomeKit.

For example, in the case of the HomeKit LED, one could add a local wolfSSL component, and simply remove the wolfSSL from EXTRA_COMPONENT_DIRS from the project's MakeLists.txt.

I would want the settings of WS to be taken from a location set in the component that includes it in the dependency file.

I believe having your own clone of wolfSSL and using the WOLFSSL_ROOT would accomplish this. Although the "dependency file" part would not work for the Managed Components. It would need to be a regular, non-managed component. I believe this is the only option for how the Espressif Managed Components work.

I know this all needs to be documented better. In fact, just this week I've been working on this.

I hope this helps in your quest to make your great product even better!

Thank you! I sincerely appreciate your feedback. You have a well informed and accurate assessment of the pros and cons of the Managed Components.

Do you have an interest in wolfSSL beyond the public HomeKit examples? Please reach out to support@wolfssl.com if there are things you'd like to discuss for a private project.

I'm certainly open to any additional ideas and feedback you have. Thanks again for taking the time to describe your concerns.

Hi @gojimmypi ,

As for bloat, well, the installed files could be added to .gitignore,

excellent idea but hard to get in place, right?

Just for brainstorming, without thought of feasibility or usefulness:

make a branch or a separate repo fork with submodule of wolfssl in the wolf GitHub account
put enough .gitignore to kill the bloat
upload that branch in your .tgz including the .git stuff
the git engine will detect any type of mismatches and the commit-id will be your fingerprint
with a way to point to settings outside of that managed component we might be in business.

In answer of your other replies: When I develop, I want the develop environment to be the same as the final delivery. Setting a WOLFSSL_ROOT does not achieve that. I use LCM4ESP32 from HomeACcessoryKid and really like that flow. It has a beta mode for testing stuff before it hits the final users. But like this I know that every loose end of the integration is covered.

Hi @gojimmypi ,

Is there any update on a possible solution for this issue? I know Hackaday Supercon is this weekend, but you mentioned wanting to have an update ready for it.

If not, I’d love to hear how it went!

Hi @AchimPieters - many Espressif updates where included in the latest wolfSSL 5.7.4 Release, however I did not have time to determine if all the of the desired Apple HomeKit settings can be configured with the Kconfig (menuconfig) settings, nor have I had a chance to publish that release of wolfSSL as a Managed Component.

The good thing is that your HomeKit examples are all functional, but yes: I do think it would be better to have a Managed Component rather than an entire copy of wolfSSL in your repo.

@M10tech sorry for the delay in responding. There's a somewhat related topic Program can't find WolfSSL headers in ESP-IDF on the wolfSSL forum that may be of interest. You have some good ideas; let me know what I can do to help.

I want the develop environment to be the same as the final delivery. Setting a WOLFSSL_ROOT does not achieve that.

I'm curious why? I would think that specifically having a static version of wolfSSL (say, a clone checked out on a particular release tag), and shared between projects (or develop / release builds) would accomplish that goal. I must be missing something, so perhaps you could give an example? Thank you.

wolfSSL / wolfssl