Blockwise usage of tiny-vcdiff - question

[Laczen]

Hi @jue89,

Thank you for providing the tiny-vcdiff library. I have seen your riot presentation on vcdiff updates, and I am thinking about creating a tiny-vcdiff based bootloader.

The idea is to create a bootloader with one slot for the firmware image and a (flash) area to store the vcdiff patch and a (flash) area for storing only the firmware flash erase blocks that are affected by the patch (the restore area).

The bootloader would start by looking for the patch and if available in full start the upgrade.

The upgrade would work window (flash erase block size) by window and (high level): a. Copy the influenced firmware window (flash sector) to the restore, b. Apply the patch on the copied window and write to the firmware, c. Boot the firmware d. If the firmware is OK it removes the patch and clears the restore area,

This process will be slightly modified to make sure that each step of the upgrade can be interrupted and continued after restart.

Do you think it is a good idea to create such a bootloader ? Most of the upgrade process is done by processing the vcdiff patch in blocks (of flash erase block size), does tiny-vcdiff support such a blockwise processing ? In some cases it might be advantageous to process the blocks without really doing something (e.g. to find out how many blocks are influenced), is this possible with tiny-vcdiff ?

Thanks in advance for any reply

[jue89]

Do you think it is a good idea to create such a bootloader ?

I'd suggest to put as little code as possible in the bootloader (its hard to update them over-the-air!) and let the application code help to prepare stuff for the bootloader. So I would let the application do the VCDIFF patching thing and prepare the new image for the bootloader.

Most of the upgrade process is done by processing the vcdiff patch in blocks (of flash erase block size), does tiny-vcdiff support such a blockwise processing ?

It's allowed to supply the patches byte by byte. You just have to make sure that all bytes of the patch are fed into the library in the right order.

In some cases it might be advantageous to process the blocks without really doing something (e.g. to find out how many blocks are influenced), is this possible with tiny-vcdiff ?

vcdiff does not allow patching in-place. It always reads from VCDIFF source and writes to VCDIFF target. It will never look into the target to decide whether things have to be written or not.

[Laczen]

@jue89, thank you for your prompt reply.

I generally agree with the idea to put as little code as possible in the bootloader. But I think there would be a major benefit in required space for images when the bootloader does the VCDIFF processing (of course this would require VCDIFF to be sufficiently mature).

I will toy around a bit with your library to see if I can make it work with predefined block size equal to the flash sector size. I think it is possible to use a ram buffer as the target (with a context containing the location) and then do use this to write to the correct flash location.

[Laczen]

@jue89, I have done some testing and it seems that the buffer size is not always respected. When a file is processed with a buffer size of 1024 I see reads and writes of the buffer size until the first add is done. After this add the read and write happens with buffer size but this is no longer aligned with the buffer size.

WIN VCD_SOURCE [0x0+16868] => [0x0+16872]
  COPY from SEGMENT [0x0+1024]
 => [0x0+1024]
  COPY from SEGMENT [0x400+1024]
 => [0x400+1024]
  COPY from SEGMENT [0x800+1024]
 => [0x800+1024]
  COPY from SEGMENT [0xc00+232]
 => [0xc00+232]
  ADD => [0xce8+5]
/*************************/ Missing filling to block end!
  COPY from SEGMENT [0xced+1024]
 => [0xced+1024]
  COPY from SEGMENT [0x10ed+1024]
 => [0x10ed+1024]

[jue89]

Yeah, I think that's how it works. vcdiff doesn't care about alignment. And its hard to do so, because un-aligned inserts result into un-aligned patches. Or am I misunderstanding your problem?

[Laczen]

Yeah, I think that's how it works. vcdiff doesn't care about alignment. And its hard to do so, because un-aligned inserts result into un-aligned patches. Or am I misunderstanding your problem?

Your understanding is perfectly correct. Doesn't this make the target_driver.erase useless ?

[jue89]

No, it isn't. An example how to handle those unaligned erases: https://github.com/jue89/RIOT/blob/feature/pkg-tiny-vcdiff/pkg/tinyvcdiff/contrib/tinyvcdiff_mtd/mtd.c#L11-L34

[Laczen]

No, it isn't. An example how to handle those unaligned erases: https://github.com/jue89/RIOT/blob/feature/pkg-tiny-vcdiff/pkg/tinyvcdiff/contrib/tinyvcdiff_mtd/mtd.c#L11-L34

Nicely done, I agree this is a good method.