Improve WiFi-Provisioning

[DerPicknicker]

Hi @CarlosDerSeher ,

in the other Issue we discussed the Wifi-provisoning. I think two options would increase the usability a lot.

The current Situation doesn't allow changing Wifi-SSIDs / Passwords without flashing the board again.

Option 1: Using the current implementation and start the Provisioning again after 3-5 failed connection attempts.

Option 2: Use this Library: https://github.com/jnthas/Improv-WiFi-Library and support the Improv-Protocol. Big benefit is that you don't need the Espressif app and you can flash it with your browser and setup the wifi as well in your browser. I did a small research and I can't find a pure C-Version of the Library. I guess a lot of Code should be working also in pure C. Only the Vectors might be Problem if I am informed correctly.

[CarlosDerSeher]

I'll have a look at it as soon as my development machine is up and running again. I installed some new drivers yesterday and it won't boot anymore since then :(

[CarlosDerSeher]

Also I saw you can also use serial connection with Improv lib to provision wifi

[DerPicknicker]

Yes that's the main usecase. The webflasher supports that. So you can flash it and directly set up the WiFi.

[CarlosDerSeher]

Ok I'll check it out. Something new to learn here :) I've not done much mixing of C and C++ code but seems it is possible to write a C wrapper for C++ functions. I'll see how much effort this will take and report here.

[DerPicknicker]

I think the benefit is that we can build the image with the GitHub action (I will try to build them after syncing is merged). And the webinstaller will automatically have the latest versions online.

[CarlosDerSeher]

Let's see what @anabolyc thinks about the espressif implementation and if he wants to use that before we put any effort into this. It can also do serial provisioning

[anabolyc]

I think the benefit is that we can build the image with the GitHub action (I will try to build them after syncing is merged). And the webinstaller will automatically have the latest versions online.

I tried to do that in the other project with artifacts taken from the release, which in turn were generated by GitHub action using the release tag. The problem is that esp-web-tools would only allow firmware to be pulled from the same domain. So I had to place binaries in the repository itself. It is possible to commit them into the repo using GitHub action, it is a bit clumsy but should work.

In any case, until WiFi provisioning is in place, it would not work anyway. I'll try to look into that before EOW.

[anabolyc]

@CarlosDerSeher @DerPicknicker I just tried example implementation of WiFi Provisioning. Book example have only BLE and SoftAP methods implemented, and they both require installing an app on your phone (an extra device) and not particularly elegant.

The Console option seems to be functionally equivalent to improv-wifi but it is not supported by ESP Web Tools. I asked folks if they are planning to add it, but I would not expect this to happen soon.

So I still think improv-wifi is the best option, as long as it is possible to wrap it up into esp-idf component.

[anabolyc]

@CarlosDerSeher here is the ESPHome implementation demo. Just to give you an idea how easy this is to flash and configure wifi in the same UI

[CarlosDerSeher]

@anabolyc they use BLE as a transport though don't they?

[anabolyc]

@anabolyc they use BLE as a transport though don't they?

If you're asking about ESPHome, they use Serial-console provisioning. Espressif's implementation has 3 options, BLE is one of them. Bear in mind, it is not available for all ESP32 versions.

[CarlosDerSeher]

I've been looking at this example and it seems to me it is for BLE provisioning. Do you have a link where serial provisioning using improv is shown?

[CarlosDerSeher]

Ok, found some more documentation. Still, an example would be great.

[DerPicknicker]

Ok, found some more documentation. Still, an example would be great.

What is the problem by using the library which I linked? Seems fairly easy to port to pure c. Only the integration into the snapcast code is not that straightforward because we must change the current logic.. checking from time to time if WiFi provisioning data is available or if connection failed multiple times

[CarlosDerSeher]

Hahaha ok my brain completely forgot about the arduino implementation :)

[DerPicknicker]

@CarlosDerSeher out of sync => hardsync 😉

The only thing I can't say is how much effort it is to mix c++ code in IDF or port it to pure c. Which way is the best. That depends on the code structure. You have definitely more insights ;-)

[CarlosDerSeher]

@DerPicknicker @anabolyc ok, so I am trying implement improv Wifi but I am unsure how to test if the responses are right... Any pointers / scripts / web sites which enable me to send improv packets or at least generate improv packets which I can send over the serial port?

[DerPicknicker]

@CarlosDerSeher .. Great to hear.

I guess here if you connect you should be able to provision the device.

[CarlosDerSeher]

Ok that's a start, but I guess there is no possibility to debug with this approach... I have no option to see what's going on on the serial port. If there is no way to generate defined packets so I can step manually through the provisioning process I guess my only option is to attach another (two) usb2serial RX line to my dev board... Any other suggestions how to proceed smarter here?

How is the checksum calculated here?

[DerPicknicker]

You can use putty to connect to the port and send them manually or you could check the browser developer settings if there is some output what is going on. I guess there should be some logging.

Alternatively print out each packet which is sent to your dev-board like a loopback device.

[DerPicknicker]

See this taken from here (282):

 void ImprovSerialComponent::send_response_(std::vector<uint8_t> &response) {
   std::vector<uint8_t> data = {'I', 'M', 'P', 'R', 'O', 'V'};
   data.resize(9);
   data[6] = IMPROV_SERIAL_VERSION;
   data[7] = TYPE_RPC_RESPONSE;
   data[8] = response.size();
   data.insert(data.end(), response.begin(), response.end());

   uint8_t checksum = 0x00;
   for (uint8_t d : data)
     checksum += d;
   data.push_back(checksum);

   this->write_data_(data);
 }

[CarlosDerSeher]

Yeah, already saw this right after my post. thanks though :)

[CarlosDerSeher]

@anabolyc @DerPicknicker just a little teaser, I got improv service working and I am able to get credentials through https://web.esphome.io/ into my esp32. What's missing now is integration into wifi_interface component. Hopefully I'll be able to push the branch by the end of the week :)

[DerPicknicker]

Really Cool! This would be great after that I will do my best to create an GitHub action to build automatically binary's for the web-installer

[anabolyc]

@CarlosDerSeher that's really cool. Can't wait to try it out. @DerPicknicker I will try to help there, as I want to integrate it into my web installer

[CarlosDerSeher]

@DerPicknicker @anabolyc ok implementation is still very error prone as most edge cases aren't tested checked for. For example I don't know what will happen if wrong credentials are supplied. Also there are some dirty delays in the code which should be done better. IP retrieval isn't very good inside wifi_provisioning component. some more work is needed here. But it works mostly and you can try it out and maybe leave some comments on how to improve the implementation.

[DerPicknicker]

Hey, that sounds great.

I checked your main.c file .. From my perspective we should think about a different approach.

Some time ago I used the FreeRtos really heavily. I think we should create a task for WiFi provisioning and other tasks.

Then you could pin the task to a specific core. Inside the task I would create an timerInterrupt which triggers every 5second to check if new serial data is available. Then stop WiFi and get the new SSID / password try to connect.

That task could also handle the WiFi-Reset feature. That could run independently from the SNAPCAST task so that shouldn't be interrupted by the other task if you pin it to a specific core.

What do you think about my thoughts @CarlosDerSeher ?

[CarlosDerSeher]

I am not exactly sure what you mean. Provisioning is already a task of it's own. It is only started if the device isn't provisioned. Uart data is received using events, a queue which blocks until the idf driver unblocks, I guess from an interrupt. No need to check every 5s.

[DerPicknicker]

Okay let's try to rephrase it :

I meant that I would regularly check for provisioning data because I changed my WiFi now, I can just plug the esp32 into my pc and send new WiFi data. Now you can only do it if the device has no WiFi data... That's why I would create a new task that check regularly for new data.

I would create a task that checks for it every x seconds or checks if new serial data is available doesn't know if there is an api for a serial interrupt.

You start 2 tasks in the main OTA and HTTP, nothing more?

And here I would start another tasks that is just checking for new provisioning data.

[CarlosDerSeher]

Most tasks are started in some init function.

If we disable logging we could leave provisioning active all the time but I am not sure if this is desirable. Currently logging is only disabled when provisioning is active. So press reset button a couple of times and you will be able to provision your device again.

[DerPicknicker]

If I pressed the button 3x will also start the Espressif provisioning and the improv WiFi? Or only the last one?

The only problem I see is if a board doesn't have the button you won't be able to restart the provisioning. That's why I thought about to check from time to time.

[CarlosDerSeher]

You got a point. Some time ago I found a component which enables logging to a network port. Maybe ghis will be a possible route to go. So disable logging to serial but enable network logs?!

[DerPicknicker]

@CarlosDerSeher ...

See here https://github.com/VedantParanjape/esp-wifi-logger

Or alternatively display the logs on the HTML-Page? Safe them to the NVS and place a button which downloads the text-file.

[CarlosDerSeher]

I'd prefer to not wear out the flash by writing all those logs to it. I think it will be better put to RAM and presented on a udp port or similar. I'll see how much RAM I am willing to sacrifice for that :)

[DerPicknicker]

Good point. Do we really need logs? I think not really . If the code is stable enough I would just disable them completely.

[CarlosDerSeher]

@DerPicknicker ok I pushed some changes to the branch because I found this whole discussion about logging obsolete. I tried provisioning and logs always on, and what can I say... it just works together. The improv header "IMPROV" ensures all other data sent is discarded by the web config interface.

[DerPicknicker]

@CarlosDerSeher .. That sounds great. When do you think it will be merged? After syncing is merged?

[CarlosDerSeher]

Yes we can merge this together or merge beforehand to sync branch doesn't matter

[anabolyc]

Hey @CarlosDerSeher @DerPicknicker I can confirm that wifi-provisioning works via web-installer. Great work! I will try integrating it into the pipeline and let you know how it goes

[CarlosDerSeher]

@anabolyc I think the implementation might change at some point though as I don't like how I've done it currently. For now I guess it is good enough.

[DerPicknicker]

@anabolyc ... I can support you. If you want. Contact me on discord. I would love to see if we can build binary's for each supported board (just copying the SDKCONFIGS)...

[anabolyc]

I mostly have it working, just one thing concerns me. Do I need to configure PSRAM or is it assumed on every board? Upd: found it, will enable and retest everything ;)

[DerPicknicker]

@anabolyc .. Could you share how you did it? Because it would be Cool if we can do that for all supported boards (lyrat, Lyrat Mini, etc.)

[anabolyc]

@CarlosDerSeher @DerPicknicker

Here is the pipeline

The idea is that whenever code is committed to the feature branch, the pipeline fires up and builds firmware for defined boards sdkconfigs (I added three Esparagus boards for now). Built images are committed to the same branch and manifest files are updated.

Then I need to pull the latest changes into my feature branch for testing locally and it can be merged to master, where the pipeline is not enabled.

Earlier I added a web installer that defines UI for the web installer app. Built artifacts and manifests are published to docs folder, which is automatically published to GitHub pages.

Comitting binaries approach is not very neat, but esp-web-tools require firmware files to be served from the same domain (allowed cross-domain sources to be precise), so I cannot pull binaries from build artifacts or release files, they have to be published together with GitHub pages. An alternative approach is to host binaries outside GitHub, but I don't want to maintain and pay for external infra.

Example run here

[anabolyc]

@CarlosDerSeher I think I mentioned before, that I can share the Esparagus or Louder-ESP board with you if you plan to support and use it. I appreciate your work here and that's the least I can do to support you. Obviously, @DerPicknicker already has one :)

[DerPicknicker]

Thank you for the detailed explanation - maybe @CarlosDerSeher can also share his thoughts. I think the Snapcast-Repo should include the Webinstaller for all boards as it's the place where all changes are happening.

So maybe we can add a GitHub page to this repo where we can get the builded binary's. That includes all Louder-Board and also the Lyrat boards.

That's only an strategic discussion;-)

@anabolyc I use your board quite often for testing it's great.

[CarlosDerSeher]

@anabolyc thanks I appreciate your offer. As I already have my own line level solution the Louder-ESP would be of interest to me. Maybe it is able to replace my justboom amp hat rpi player. I doubt it though as it probably hasn't enough juice for outside playback in my garden. But for sure there will be a spot for it :)

You are selling these players commercially in Europe? How do you handle CE?

[CarlosDerSeher]

Comitting binaries approach is not very neat, but esp-web-tools require firmware files to be served from the same domain (allowed cross-domain sources to be precise), so I cannot pull binaries from build artifacts or release files, they have to be published together with GitHub pages.

I don't really get this part. Normally I would think the best way is to use github releases to host pre-built binaries

[anabolyc]

I don't really get this part. Normally I would think the best way is to use github releases to host pre-built binaries

I started with that approach. Problem is releases are served from github.com domain. Github pages (and thus installer) from github.io domain. JS code cannot fetch files cross-domain unless CORS headers are present, and they are not in fact.

[anabolyc]

@anabolyc thanks I appreciate your offer. As I already have my own line level solution the Louder-ESP would be of interest to me. Maybe it is able to replace my justboom amp hat rpi player. I doubt it though as it probably hasn't enough juice for outside playback in my garden. But for sure there will be a spot for it :)

Pls drop me a line to andriy@sonocotta.com, I'll need your address for shipping. Just to be sure, Louder-ESP is a bare board, while Louder Esparagus is a cased solution, but both have external wifi antenna.

You are selling these players commercially in Europe? How do you handle CE?

I worked with Crowd Supply on CE/UKCA compliance, they have good lawyers :)