OTA PIC firmware upgrade

[benjaminvdb]

I have an OTGW (PIC16F1847) with a WeMos D1 module from Nodo-Shop. The PIC has firmware 6.2 and the WeMos D1 firmware 0.9.5 (the latest).

Earlier I flashed the PIC firmware using a USB cable, but this only seemed to work if the WeMos was removed from the OTGW board. Now that the OTGW is in a housing, I must disassemble it before I can flash the PIC using a USB cable. It would therefore be great if I could upgrade the PIC firmware OTA instead.

If I go to the "PIC firmware" page of the webUI, I can get the latest version (6.3) by clicking the download button. When I download this file from FSExplorer and compare the md5 checksum with the latest OTGW firmware, I see they are identical. However, the "Device information" page of the webUI still indicates 6.2; this can also be seen via telnet during boot.

How can I flash the latest PIC firmware OTA?

[Barracuda09]

Yes, this is something I would like to know as well. I am unable to flash the new PIC firmware 6.3 as well. I used otmonitor to try to flash the PIC, but it fails to put the PIC in reset mode and says it needs a manually reset, but both do not work.

Were could be the problem? or is it a user error?

[benjaminvdb]

@Barracuda09 If you're trying to perform an OTA flash through otmonitor failure is expected as indicated by the documentation:

Warning: Never flash your OTGW PIC firmware through wifi using OTmonitor application, you can brick your OTGW PIC. Instead use the buildin PIC firmware upgrade feature (based on code by Schelte Bron)

I'm trying to perform an OTA flash through this repositories firmware using the webUI, not otmonitor.

[Barracuda09]

Yes, but otmonitor uses OTWG-firmware as well, to perform the PIC firmware flash. But could be a problem bricking if network interruptions occur.

But web interface did not work for me either, that is why I tied the otmonitor path.

[Barracuda09]

I think, it may have to do with the WeMos D1 running out of memory trying to allocate mem here: https://github.com/rvdbreemen/OTGW-firmware/blob/627e3b918562c506e4e54b5dcedb0e0463aa55e1/OTGWSerial.cpp#L460-L468

[hvxl]

Upgrading of PIC firmware 6.X fails because the PIC16F1847 uses a slightly different bootloader algorithm than the PIC16F88 due to differences in the hardware. This change has not yet been implemented in the OTGW-firmware for the ESP8266.

Upgrading with OTmonitor fails because the OTGW-firmware collects serial data for a second or until a newline. Bootloader packages end with ETX, not LF. And the bootloader waits up to 1 second for a firmware upgrade to start. So, the OTGW-firmware delays the initial package just long enough for the bootloader to time out.

[tomm1ed]

Upgrading of PIC firmware 6.X fails because the PIC16F1847 uses a slightly different bootloader algorithm than the PIC16F88 due to differences in the hardware. This change has not yet been implemented in the OTGW-firmware for the ESP8266.

Upgrading with OTmonitor fails because the OTGW-firmware collects serial data for a second or until a newline. Bootloader packages end with ETX, not LF. And the bootloader waits up to 1 second for a firmware upgrade to start. So, the OTGW-firmware delays the initial package just long enough for the bootloader to time out.

So in conclusion OTA flashing is not supported on PIC16F1847? My OTGW is completely 'build in' so requiring to get it out to flash the PIC is a royal PITA. @rvdbreemen do you have any plans on implementing this in the OTGW-Firmware?

Also is there any documentation how to flash through the web interface (I was so far unable to find that ;) )

[Barracuda09]

@tomm1ed

OTA for PIC is possible but not correctly implemented in this firmware. How ever this firmware works https://github.com/hvxl/otgwmcu, but lacks the features that are in this firmware.

[tomm1ed]

https://github.com/hvxl/otgwmcu

Weird I see @rvdbreemen as contributor on that firmware. Can we expect an update to OTGW-Firmware to allow upgrades?

(and also; where is the documentation how an upgrade would go ;) )

[Barracuda09]

Yes, it is possible to get it in OTWG-Firmware, but it takes time to do so.

The upgrade process is not that difficult here is how to do it in otgwmcu:

[rvdbreemen]

Weird I see @rvdbreemen as contributor on that firmware. Can we expect an update to OTGW-Firmware to allow upgrades?

Yes, that's the plan. Actually @hvxl has given me the code months ago. Life happened. And there is a way around that my firmware does not support the 6.x firmware yet.

Planning to fix this though. I can't stay behind, need to push several small fixes to the firmware. Plus there is something wrong with the flashing of the current binary.

[benjaminvdb]

@rvdbreemen If you open an issue for the issues that you're aware of then perhaps someone else can take a look at it if you like.

[rvdbreemen]

@benjaminvdb I would love to have some collaborators. Are you by any change a programmer that can help me out here?

The dev tree with the upgrade code is here: https://github.com/rvdbreemen/OTGW-firmware/tree/dev-prep-6x-upgrades

[tomm1ed]

Weird I see @rvdbreemen as contributor on that firmware. Can we expect an update to OTGW-Firmware to allow upgrades?

Yes, that's the plan. Actually @hvxl has given me the code months ago. Life happened. And there is a way around that my firmware does not support the 6.x firmware yet.

Planning to fix this though. I can't stay behind, need to push several small fixes to the firmware. Plus there is something wrong with the flashing of the current binary.

Ah wonderfull! Looking forward to any updates you can supply. I know how life can be ;)

[0crap]

Subscribed.

My recently obtained OTGW also has a PIC16F1847 running firmware 6.2 I'll wait trying v6.3 until this issue is fixed.

[benjaminvdb]

@benjaminvdb I would love to have some collaborators. Are you by any change a programmer that can help me out here?

The dev tree with the upgrade code is here: https://github.com/rvdbreemen/OTGW-firmware/tree/dev-prep-6x-upgrades

@rvdbreemen I'm a programmer, but programming microcontrollers in C++ isn't what I do for a living 😅

I meant to say that if development happens transparently, it increases the probability of contributions from the community. Even if I had the capabilities to help out, I wouldn't know where to start, but maybe that's just me.

Do you have a link to Schelte's upgrade code? I see there's one commit in the dev branch that refers to him, but I'm not sure what the current status is. Is his work fully integrated but not working correctly with the rest of the codebase, are parts still missing, etc.?

[rvdbreemen]

@benjaminvdb the dev-prep-6x-upgrades was the branch I was working on. Good to hear you are a programmer too. I do develop everything in the open and accept Pull Requests. This summer I was working on integrating it, you probably looked at that branch.

I love to see some development on the firmware, so you are more then welcome to check out my 6x upgrade branch. And no, I did not get it to work. Due to personal reasons, I stopped this summer, other stuff had more priority. I am taking my Christmas vacation and hope to spend some time on my project again.

So if you feel you want to help me out, please do.

[benjaminvdb]

@rvdbreemen What I meant to say with regards to transparency is perhaps more an issue of organization. In my mind - but correct me if I'm wrong - there is working code capable of upgrading the PIC in Schelte's codebase. However, I don't think I have access to it; I found only firmware binaries on his website. If I have his exact code then I can try to integrate it.

For me, a partial integration is not an ideal starting point. I found it hard to understand the aim of the development branch and had the impression it was trying to do multiple things. I guess that's fine if commits are squashed and traceable to individual issues, but I don't think that's currently the case. I'd suggest to create individual issues for bugs, features, etc. and keep relevant information in there so it's easier for others to contribute. I understand it requires some effort, but it could alleviate the workload in the long run. You've created something wonderful and useful here and I'm sure others will want to contribute to keep it alive and kicking!

In short, could you share the relevant upgrade code from Schelte's codebase? I promise to then take a stab at it!

I wish you a wonderful Christmas 🎅

[hvxl]

The library is part of my public otgwmcu project: https://github.com/hvxl/otgwmcu/tree/master/libraries/OTGWSerial

[benjaminvdb]

@hvxl That's awesome. Thanks! :pray:

[Barracuda09]

I have looked to this branch https://github.com/rvdbreemen/OTGW-firmware/tree/dev-prep-6x-upgrades as well.

My problem was that the malloc failed to allocate memory. See comment: https://github.com/rvdbreemen/OTGW-firmware/issues/148#issuecomment-1339828542

There is a lot of 'debug' strings that takes up a lot of memory.

So it would maybe wise to clean up this first? or some other method to put it into flash memory.

Edit: I have had it at some point that the memory (Free mem) was available but then still nothing happend (No progress bar as in otgwmcu firmware)

Edit...Edit: Yes that I had moved as much strings as possible to have more free memory

[Barracuda09]

I have to dive into this some more, but what is better was my problem. This OTWG-firmware or otwgmcu. This firmware support more like FSExplorer, MQTT and commands etc. (only not yet PIC firmware update)

I personally only use MQTT the rest is nice to have. So I stopped at this point to see were my time would be wise to invest.

Edit: I do like this web interface of otwgmcu see

[0crap]

MQTT is probably the most commonly used feature, in combination with Home automation software. I would vote for the MQTT stuff and the possibility to keep the PIC up to date. If that works, add missing stuff in small iterations?

But I'm not qualified to answer the question, maybe this approach breaks everything. ;-)

[Barracuda09]

The main reason I like to update the PIC firmware OTA was to modify the PIC firmware.

[rvdbreemen]

Thanks for all the input on what you would like to see working. I would simply try to fix the OTA upgrading first. The UI was designed for use on a smartphone in my case. I get why you like other UI, it looks a lot like the OTmonitor software too.

[rvdbreemen]

So over the last few weeks I have been trying to finally get back into getting this fixed. First I had to update a lot of library stuff, but that's done now. So I am back to about where I stopped before the summer vacation, trying to make 6.x upgrades work using @hvxl excellent serial class.

In this thread there have been several hints on what might be the issue. The problem is actually that the new OTGWSerial.cpp library claims 8k of memory (malloc that @Barracuda09 pointed out), this is needed for the 6.x firmware. This however is not possible, it fails to allocate memory and thus no upgrade ever happens.

So now I have to see if I can reduce the memory footprint, without loosing functionality. Hmmm, any suggestions?

I see @0crap and @Barracuda09 already figured this out a while ago. Now dropping some functionality might be an option, but where do I start. I have to think about it. I agree that MQTT and Home Assistant integration are the core of my firmware, it always was about that. The REST API is used to communicate with the Web pages, and can be used to control the device.

Like I said, I need to think about this, upgrading to a ESP32 compatible device might be a way to go, with WeMos that is possible ;-)

[tjfsteele]

If you can stream the firmware from the filesystem instead of keeping it in memory..?

Tim

On Sun, 8 Jan 2023, 21:03 Robert van den Breemen, @.***> wrote:

So over the last few weeks I have been trying to finally get back into getting this fixed. First I had to update a lot of library stuff, but that's done now. So I am back to about where I stopped before the summer vacation, trying to make 6.x upgrades work using @hvxl https://github.com/hvxl excellent serial class.

In this thread there have been several hints on what might be the issue. The problem is actually that the new OTGWSerial.cpp library claims 8k of memory (malloc that @Barracuda09 https://github.com/Barracuda09 pointed out), this is needed for the 6.x firmware. This however is not possible, it fails to allocate memory and thus no upgrade ever happens.

So now I have to see if I can reduce the memory footprint, without loosing functionality. Hmmm, any suggestions?

I see @0crap https://github.com/0crap and @Barracuda09 https://github.com/Barracuda09 already figured this out a while ago. Now dropping some functionality might be an option, but where do I start. I have to think about it. I agree that MQTT and Home Assistant integration are the core of my firmware, it always was about that. The REST API is used to communicate with the Web pages, and can be used to control the device.

Like I said, I need to think about this, upgrading to a ESP32 compatible device might be a way to go, with WeMos that is possible ;-)

— Reply to this email directly, view it on GitHub https://github.com/rvdbreemen/OTGW-firmware/issues/148#issuecomment-1374926567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN3CV4AXJGVAXHA5FVOYXHLWRMTQPANCNFSM6AAAAAASLO7RE4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[benjaminvdb]

In this thread there have been several hints on what might be the issue. The problem is actually that the new OTGWSerial.cpp library claims 8k of memory (malloc that @Barracuda09 pointed out), this is needed for the 6.x firmware. This however is not possible, it fails to allocate memory and thus no upgrade ever happens.

@hvxl Is 8k of memory actually required for the current 6.x firmware or is this to prepare the code for larger firmware sizes in the future? If not, what is the amount of memory currently required? I've tried to figure this out from the binary file produced by hex2bin but I was unable to.

[hvxl]

Not the full 8k is required. But the code is not placed in one contiguous area by the linker. The PIC goto and call instructions can only address a 2k block. Therefor the linker must place each code package in its entirety inside a single block. With the latest firmware, the linker has placed code in address ranges 0x000000 - 0x00077b, 0x000800 - 0x000a43, and 0x001800 - 0x001ea5. The self-programming code is in 0x001f00 - 0x001fe9. So allocating only the required amount of memory is going to be a bit complicated.

[rvdbreemen]

What if we write out the hex translation to a binary file. And from there we stream it to the PIC over serial. That way the impact on the code is small.

PS. In the mean time I will go and move strings to flash memory to save space...

[Barracuda09]

Hi @hvxl

What is actually the purpose of reading in the total file?

• To decode the hex char to binary char
• To separate the data- with prog-mem out of the file?

[rvdbreemen]

If I remember correctly the hex file format is a simple looking but feature rich way of writing a binary memory dump into a ascii fileformat which includes things like checksums per line.

The linker generates this as an optimized file format that can be programmed by programmers. As most communicate over serial links it also needs to be robust and compact.

Reading the file seems to do:

1. Create a linear memory so it can be programmed into the PIC from the ESP

2. Validate the file on problems while transferring several hops.

3. Read relevant information from the file, like version information on the firmware itself.

In short it makes sense. Now we just need a way to not have to allocate so much memory 😃

[rvdbreemen]

Link to file format explainer: www.elproducts.com/understanding-hex-files.html

And more details here: http://ww1.microchip.com/downloads/en/DeviceDoc/33014K.pdf

[Barracuda09]

Hi @rvdbreemen

Yes, I am familiar with this hex file format and programmers. But for some reason I think it could be done with less memory allocation.

But I have for testing here moved a lot of strings (or removed) to flash memory to clear (heap) memory. Edit: But that seem not to work out as well, this could well be a lack of the web interface progress bar

[hvxl]

There are several reasons for reading the complete file. In addition to what is already mentioned:

• Determine for which PIC the firmware is intended.
• The PICs must be programmed in blocks of 32 words. The hex file does not necessarily provide the code aligned to these blocks, or even in sequential order. Programming a block in parts would erase what was programmed before, leading to corruption of the firmware.
• Figure out the metrics for the progress indication.

So, reading the complete code into memory was the easiest solution. Doing it in two passes is very likely possible. But it will be much more complicated. Especially if a code block is revisited later in the hex file. It may be necessary to disallow that. But it still has to be detected.

[rvdbreemen]

But I have for testing here moved a lot of strings (or removed) to flash memory to clear (heap) memory.

I did the same thing last night. To no resort. I got 16kB of free memory.

But that seem not to work out as well, this could well be a lack of the web interface progress bar

It just failed with me. You can check the debug logs for our of memory error.

[akula3]

Hi, I bricked the P16F1847 by using OTmonitor via OTA. I know, it is my fault :( A replacement already ordered at nodoshop. Nevertheless, is there a possibility to recover the bricked PIC? Thank you for your valuable suggestions!

[tjfsteele]

You may be able to recover it using otmonitor via usb

On Tue, 10 Jan 2023, 20:22 akula3, @.***> wrote:

Hi, I bricked the P16F1847 by using OTmonitor via OTA. I know, it is my fault :( A replacement already ordered at nodoshop. Nevertheless, is there a possibility to recover the bricked PIC? Thank you for your valuable suggestions!

— Reply to this email directly, view it on GitHub https://github.com/rvdbreemen/OTGW-firmware/issues/148#issuecomment-1377802013, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN3CV4AND4FAPINFLEDGDYDWRXAI5ANCNFSM6AAAAAASLO7RE4 . You are receiving this because you commented.Message ID: @.***>

[Barracuda09]

Hi @akula3

Yes, you have a change to use: https://github.com/hvxl/otgwmcu to "unbrick" your PIC. Just flash this to your NodeMCU and update your PIC from there. I had success with flashing the PIC (It was not bricked). It seems to work quite well and I believe it will also work when the main part is flashed wrong, at least if the self programming code is not erased

[rvdbreemen]

@Barracuda09 how much memory did you win by dumping removing strings and stuff? Just curious how far you got with this.

[Barracuda09]

@rvdbreemen, I do not know how many exactly. But at least enough to allocate the memory for OTGWUpgradeData.

I placed most of the text/strings between F() macro and removed(empty) defines such as #define MQTTDebugTln(...)

Edit: Executable segment sizes: ICACHE : 32768 - flash instruction cache IROM : 514300 - code in flash (default or ICACHE_FLASH_ATTR) IRAM : 29501 / 32768 - code in IRAM (IRAM_ATTR, ISRs...) DATA : 1672 ) - initialized variables (global, static) in RAM/HEAP RODATA : 15584 ) / 81920 - constants (global, static) in RAM/HEAP BSS : 34168 ) - zeroed variables (global, static) in RAM/HEAP Sketch uses 561057 bytes (53%) of program storage space. Maximum is 1044464 bytes. Global variables use 51424 bytes (62%) of dynamic memory, leaving 30496 bytes for local variables. Maximum is 81920 bytes.

[akula3]

@Barracuda09, thank you for pointing me into the right direction! Now the PIC is healthy again, eventough I did it in slightly different way you suggested. I downloaded fw.bin and fs.bin from http://www.otgw.tclcode.com/download.html#otgwmcu and uploaded into D1 mini, then connected to OTmonitor via USB and uploaded there the newest PIC firmware 6.4 Thanks once more!

[Barracuda09]

Hi @akula3 credits go out to @tjfsteele as well for the USB suggestion.

[rvdbreemen]

The v0.10.0 firmware has been released and now flashes both pic16f1848 (6.x firmware) as pic16f88 (5.x firmware). I think we can close this issue now ;-)

[Barracuda09]

Hi @rvdbreemen

This seem to work to update. But how can I update the PIC with a self build hex file? or am I missing something?

[benjaminvdb]

The v0.10.0 firmware has been released and now flashes both pic16f1848 (6.x firmware) as pic16f88 (5.x firmware). I think we can close this issue now ;-)

This is incredible work, @rvdbreemen. It works flawlessly! 🙌

[rvdbreemen]

@Barracuda09 currently the filebrowser does not go into directories. That means you cannot manually upload your custom hex file. The only way is to create a custom FS using the Arduino plugin yourself that includes your custom hexfile.

[hvxl]

Maybe a simpler fix/work-around than making the file browser handle subdirectories, could be to search for hex files in the pic-specific directory and the root directory.

@Barracuda09, the latest firmware does allow upgrading the PIC with OTmonitor. So that's another option. Of course it's a bit more risky if your WiFi is not rock solid.

[Barracuda09]

Hi @rvdbreemen & @hvxl

Thanks for you feedback and suggestions. I will try this out.

Thanks for your support!!