Create a generic, portable, open source hex-manip facility

[ckormanyos]

We need a generic, portable, open source hex-manip facility for the manipulation/creation of binary and hex files.

[ajneu]

Just out of interest... What are you thinking of here? What should the hex-manip facility do? This might be something, I'd perhaps like to help/contribute to... (well, provided that it doesn't require "hard" ultra-specialist knowhow, ... and provided I find time...)

[ckormanyos]

The hex-manip program needs to read and inter-convert hex files to and from standard formats such as I-HEX86, S-Rec, raw binary. Hereby, a major function is filling unused code areas with a defined value. Another heavily-used feature is the creation of, for example, a CRC or similar checksum placed at a fixed location in program code and used for verification of program-code-integrity at run-time (e.g. in the idle task).

When we make a HEX mask for, let's say, the beagle bone, we begin with a I-HEX86 extract with objdump or objcopy. We might use a hex-manip program to fill this raw HEX-dump to a given size such as 32kB with a defined fill pattern. Then we need to convert the filled I-HEX86 to raw binary and finally hang a skinny boot header composed of two 32-bit words (size of the image and memory location) at the beginning of the file (such that it boots the bare-metal app with the standard boot-loader). This file is called "MLO" for the case of beagle bone, but other systems have other needs for hex-manip.

It's not really hard to write a hex-manip program. Basically, one needs to specify an input language for buffer manipulation, address-shifting, calculating and placing checksums, etc. Finally, a hex-manip program needs to be worked into the build.

[ckormanyos]

The only actual running use-case for this in ../../real-time-cpp is creating the binary image MLO for target am335x. If a simple tool can be found/created for this purpose alone, then it is sufficient for closing this ticket.

[Ahelion]

Hello, There is a python library: (https://github.com/yweweler/pysrec) it only knows srec files, but it can be used from command line to do basic stuff. If the build would output a srec, then we can modify and do stuff to it.

[ckormanyos]

There is a python library: (https://github.com/yweweler/pysrec)

Thank you @Ahelion. That seems like an interesting option. I will investigate for several purposes and get back to you regarding this.

If the build would output a srec ...

See #174

[Ahelion]

@ckormanyos there is another python tool(more advanced) that does well lots of stuff that tools like vector Hexview does: https://hexrec.readthedocs.io/en/latest/cli.html The problem with both is that they are dependend on a local python instalation.

[ckormanyos]

there is another python tool(more advanced) that does well lots of stuff that tools like vector Hexview does

Looks kool 👍

The problem with both is that they are dependend on a local python instalation.

Indeed. I'm not after that, although that is no real problem on CI pipelines or to require from clients these days. Nonetheless, I simply just don't want to deal with an installed Python requirement.

For this repository, I have only two targets requiring additional hex/binary file manipulation. These are the targets BeagleBone, which requires binary file MLO with 8 byte header and RPi-Zero which requires kernel.img simple, naked raw binary to load at pre-programmed boot address.

These two requirements are so trivially simple to fulfill I might simply end up writing a program that handles these two items plus possibly supplies fill-byte/fill-pattern option. Idea is to write a handful of header only C++ for this, build it on the spot in CI and go playing. I will probably end up doing that over Christmas time or early next year.

[ckormanyos]

I will probably end up doing that over Christmas time or early next year.

Or... if a volunteer offers help for this?

[Ahelion]

@ckormanyos got your point. We can port this utility ot windows : http://srecord.sourceforge.net/srecord-1.64.pdf it can be compiled via cygwin. We can talk offline about it ( Mircea from work here btw)

[ckormanyos]

Hi Mircea. Agreed. There are a few good options.

[ckormanyos]

port this utility

Hi @Ahelion. Thx. As a first step I could see if the above-mentioned package builds on msys2/mingw64.

At the moment, I am actually building a whole farm of GCC cross compilers on msys2/mingw64. So I'm kind of in the swing of this. If that plays out, it should be relatively straightforward to move on to a portable build system.

At the moment, that package is still too large for what real-time-cpp needs, but it looks interesting...

[Ahelion]

@ckormanyos found it precompiled allready: https://sourceforge.net/projects/srecord/ It is versio 1.63, it has a manual and examples. First thing that I searched was how to add a CRC to a binary file:

[Ahelion]

@ckormanyos any input related to the tool srecord?

[ckormanyos]

any input related to the tool srecord?

That particular work is too large for what real-time-cpp needs.

[ckormanyos]

Note to self: Take a look at this. Could this project serve as a backend for read/write and improt/export of various hex-file formats?

[ckormanyos]

Note to self: On the completely simple (other) end of the spectrum of tool complexity is to simply use xxd where needed. The only actual issue needing a solution in this repo at the moment is the BBB target (am335x), as mentioned in #204.

Since this project requires a two-dword header for the binary image, this could relatively easily be done with xxd, which is kind of portable.

[Ahelion]

@ckormanyos As a user I need simplicity, using a tool like Srecord could help me focus on my tasks. As you see above, it is easy as pie to make a crc over a region and insert it.

[ckormanyos]

easy as pie to make a crc over a region and insert it

Would you be in a position to explicitly show such a use case within the context of real-time-cpp via PR? Maybe tyke a resulting SREC file, align it to 64k - 4 bytes. Acc a CRC? Something like that...

[ckormanyos]

Please also be aware that the only real requirement here is not a CRC, rather the 8 byte header for micro-SD card of the kernel image of the BBB. This consists of a byte count plus load address. I don't know if the afore-mentioned program SREC can do that... And it needs to be binary.

I guess, I am reducing this issue to the sole requirement of this repo right now. CRCs are not needed.

[Ahelion]

Hello, Ok, so that is the requirement, so a header is needed, from first glance I do not know if the tool can do it. I was expecting something like plain data manipulation, move things arround, fill, crc, split, etc. Please, write like a requirement. What the input hex file is and what should the header exactly contain. I will read the manual and check. From the top of my head, would this be done with the gcc linker script? I mean, just have a 8 byte location where to insert the load address, and for the byte count something like: __flash_end-__flash_start. I do not know if this is possible, just thinking in text right now.

[ckormanyos]

Hi Mircea (@Ahelion), I guess my text has been somewhat all over the place and I am, in fact, collecting my thoughts myself. This has led this issue astray and introduced some confusion, for which I am sorry.

But my thoughts are actually consolidating. At the present time, I think I will handle the very few needs for such a tool that are present in this repository in a different way.

I'll try to use standard simple tools like the linker itself (ld, which can pack buffers) combined with objdump, possibly combined with xxd (and possibly also with sed or gawk if needed) to handle the very few hex manipulations required.

This is a big step, but I am actually going to close this issue. I might open up a new issue at a later time. But I'll do a preliminary close here.

[ckormanyos]

See also #204