Flashing with user provided firmware

[DanielBaulig]

Was it ever considered to allow the user to provide a firmware binary instead of the application providing a manifest with fixed binaries? Similar to how the user can provide a custom firmware on https://web.esphome.io

That would make the esp-web-tools button more useful for more generic use-cases in addition to installing specific firmware binaries.

Context: I am building a web interface for interacting and controlling generic ESPHome devices using the web_server API. I'd love to add the ability to flash firmware and provision WIFI to that interface, too. Obviously I could just use esp-tool and improv-wifi-sdk directly, but why reinvent the whole wheel, just because one spoke I'd like to have is missing?

Is this something that the project was willing to take a pull request on?

[balloob]

That's not something we would consider.

I would consider a contribution that allows to specify manifest contents directly on the button. That way you could generate a manifest on the fly that references a selected firmware.

[DanielBaulig]

Thanks for the response. That's obviously not, what I was hoping for :-/

Let me think about your proposal a little. To make sure I understood correctly what your are suggesting: when you say a generated manifest could reference a selected firmware, then how should I think about this? is the following flow roughly what you imagined in this case?

1. User selects a firmware using an application provided <input type="file" /> dialog.
2. An object URL for the selected file is generated using URL.createObjectURL
3. A manifest is generated using the firmwares generated object URL in it's builds element's path.
4. The generated manifest is assigned to <esp-web-install-button />
5. The <esp-web-install-button /> is triggered offering the firmware for installation.

The main difference here -- at least from a user flow perspective -- would be that the firmware file would have to be selected prior to initiating the <esp-web-install-button /> flow, rather than as part of it.

Did I get this correctly?

[balloob]

Yeah, that was along the lines that I was thinking.

[DanielBaulig]

I spent some time thinking about this some more. Sadly I'm not convinced this will allow me to fully implement what I am hoping to do. The user story outlined above would work and be reasonable. However, I would also like to provide the user the ability to provision Wi-Fi on any MCU already supporting Improv Serial. And I think this solution would still not reasonably allow this. Let me explain.

If I was going to provide Wi-Fi provisioning through ESP Web Tools using the same flow as above, then the user would have to select a firmware binary file in the first step. Given the user story of provisioning Wi-Fi they may not have a firmware binary and they probably have no intention to actually flash it anyway.

In theory at least one could provide a dummy firmware binary when the user opts the provision Wi-Fi flow from the application (e.g. URL.createObjectURL(new Blob([]))). However, then we'd run into problems when the user chooses the "Install firmware' option in the ESP Web Tools dialog. It would now proceed to flash an empty or some other dummy firmware onto the MCU, which is almost certainly not what the user intended.

Given that this user story is critical for my use-case I'm not sure if the dynamic manifest solution will work for me. Do you have any more ideas or thoughts on this?

I'd also like to better understand what your specific objections are to expanding the ESP Web Tools install button by an option to provide a user supplied firmware binary. Maybe I can think of a way to solve for your specific objections while still supporting the user stories I am imagining.

I'll also outline in some more detail how I was imagining the flow to work within ESP Web Tools, both from the users perspective, as well as from a technical perspective. Just to make sure there are no misunderstandings.

User story:

• User connects an unflashed MCU via USB
• User clicks ESP Web Tools install button
• "Connecting" dialog shows
• "Device dashboard" dialog shows with options "Install firmware" and "Logs & Console"
• User clicks "Install firmware", which triggers a file selection dialog.
• User selects a firmware binary to flash that implements Improv Serial
• "Confirm Installation" dialog shows
• User selects "Install"
• "Installing" dialog shows and flashes the user provided binary file to the connected MCU
• "Installation complete" dialog shows
• User selects "Next"
• "Configure Wi-Fi" dialog shows
• User inputs their Wi-Fi credentials and clicks "Connect"
• "Device connected" dialog shows, offering the user to "Visit Device" or "Skip"
• User selects "Skip"
• "Device dashboard" dialog shows, this time with the options "Install firmware", "Visit Device", "Change Wi-Fi" and "Logs & Console".
• User clicks the "x" to close the Device dashboard

From a technical perspective it could work as follows:

1. The user provided firmware binary is an option that needs to be explicitly opted into by the application and is exclusive to the manifest provided binaries (i.e. you can't have both). A dedicated attribute on the button (e.g. usefiledialog="true") might be a good way for the application to express this opt-in.
2. All the current fields in the manifest become superfluous for this flow and no manifest needs to be provided by the application with the ESP Web Tools install button in this mode.
3. Internally, just a couple of changes seem to be necessary: 3.1. With the file dialog opt-in enabled, a dummy manifest is generated internally. It sets reasonable defaults, like "name" to "firmware" and "version" to an empty string. Since ESP Web Tools won't be able to determine if firmware without Improv Serial support is already installed on the device or if a supplied firmware binary is the same or updated version of the one running on the device it is probably advisable to set "new_install_prompt_erase" to true. The "builds" array remains empty. No other fields are set. 3.2. With the field dialog opt-in enabled, checkFirmwareSame is not called. ESP-Web-Tools would assume the firmware to always not be the same. 3.3 During flashing, if the build array is empty, instead a of sending a request to a part URL to generate a blob entry in the file array, a <input type="file" /> dialog is generated and a Promise for it's result added to the file array instead. If a file gets selected by the user, the Promise is resolved. If the dialog gets closed without a file selected the Promise gets rejected.

It seems like these three changes are the only ones necessary to support this flow. Note that this specific implementation is just a suggestion. I care more about the user story than the technical details. A couple of additional points about the technical details that are worth mentioning:

• I've chosen the technical implementation requiring the generally least amount of changes in the code base rather than the one that is the most explicit. E.g. instead of generating a dummy Manifest, it might make sense to be more explicit about this alternative flow within the code.
• In theory it's possible to have the application provide the dummy manifest with an empty builds list instead of an explicit opt-in, keeping the API surface smaller. It seemed to me that on the API level this opt-in should be more explicit though.
• If the file dialog gets cancelled and no file selected, the resulting error screen will talk about a failed firmware download. This is probably confusing for the user. Also see my next point.
• It is probably advisable to trigger the file dialog more directly from the click on "Install firmware" instead of from within the flash process and provide the resulting Blob/File to the flashing process instead. While this will require some more changes in more locations, it would avoid the above failed download problem and would allow one to simply not start the flashing process at all if the user cancels out of the file selection process.

Thanks! I appreciate the thought and time you are putting into this.

[DanielBaulig]

Any more or additional thoughts on this?

It's something I need for my project and it appears it might be useful for other projects, too. I'd love to provide an upstream patch that enables this workflow cleanly rather than having to fork, cobble together some wonky workaround or rebuild large parts of esp-web-tools-install button functionality from scratch.

But if none of my arguments have been convincing or helpful, let me know and I'll move on.

[balloob]

I don't think we should adopt this in ESP Web Tools. It's really made for ready-made firmware. Once you want to allow selecting different firmwares, it becomes a different focus. Just like we did for ESPHome Web, I suggest that you create your own application using esptool-js.

[DanielBaulig]

That's fair. Thank you!

[Jason2866]

@DanielBaulig I made a fork of esp-web-tools for project Tasmota. One reason was to provide flashing of user provided firmware (needs to be one file "factory image") You find the fork here Running in action https://tasmota.github.io/install/

[DanielBaulig]

@Jason2866 Thank you! I'll check it out.

I have started building out my own implementation based on Improv Serial SDK and esptool for now. Being able to roll a completely custom UI does come with some useful benefits.