Add encode/decode helpers?

[natevw]

I haven't spent too much time trying to track down their exact provenance but there's a set of helpers for serializing/deserialization integers into/from bytes:

The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec(), le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec() functions encode and decode integers to/from octet stream on any alignment in big/little endian format.

e.g. helpers like uint64_t le64dec(const void * p) and void be16enc(void * p, uint16_t x) that convert between native numbers and raw buffers.

Found these via https://github.com/ARM-software/arm-trusted-firmware/blob/master/include/lib/libc/endian.h#L83 and they appear in a variety of search results e.g. NetBSD manpages and OpenBSM source code and helper libraries like https://libusual.github.io/endian_8h.html.

Frankly, they make more sense to me since ideally one wouldn't use a "mixed up native integer" like htonl/be32 return, but rather one has/wants the underlying bytes somewhere.

[michaeljclark]

I see they take pointers. The compiler can usually optimize out access via pointers because the functions are static inline, but I prefer value-based interfaces. We also end up with alignment issues because not all architectures can perform unaligned loads. That isn't a problem with value-based interfaces. I have some data serialization similar to what you mention in these two repositories:

• https://github.com/michaeljclark/vf128
• https://github.com/michaeljclark/crefl

However, I use a macro so that endianness is converted as a value versus through a pointer. Nevertheless the APIs themselves have some interfaces with pointer-based serialization to read and write data. I would like them to be value-based but returning tuples and destructuring return values is not something we can do eaily in C. I have a branch locally where I have been benchmarking value-based versus pointer-based interfaces. It is not simple because MSVC and Clang/GCC optimize them very differently. The value-based interfaces tend to perform better on Linux or macOS due to the SysV ABI, but MS ABI can't for example return a structure containing two values in two registers, instead multiple return values must go through memory.

Thanks very much for the feedback btw! This repo is totally unofficial as you are probably aware.

[natevw]

I see they take pointers. The compiler can usually optimize out access via pointers because the functions are static inline, but I prefer value-based interfaces. We also end up with alignment issues because not all architectures can perform unaligned loads.

Not to totally brush aside performance concerns but I would still stand by my request for pointer-based interfaces.

I'd agree that the lower-level value conversions are themselves a big missing piece in portable C and "C/C++". Having value-to-eulav and conditional value-to-${value or eulav depending on host architecture} conversion macros set up around the compiler-specific intrinsics would be a great thing to get standardized.

But in practice I see the main use for endian helpers in converting from bytes to values, and from values to bytes. And if all I have is a backwards value in a variable, I'm still left with the homework problem of getting that into an outgoing buffer without accidentally un-backwardsing it or hitting some other language pitfall in the process.

The encode and decode interfaces offered by the "endian.h" headers floating around solve exactly that problem [and the problem going the opposite direction] in exactly the right context. That is, whenever someone asks "how do I get this N-bit number as bytes…?" the question they get back is "…in which endian?". Having standard helpers shortcuts that and the answer becomes "use be{N}enc(bytes, number) if you want big-endian bytes, or le{N}enc(bytes, number) if you want little-endian bytes".

Then the implementation can encapsulate the best way to handle that on a particular architecture, whether that's:

// HT: https://www.anintegratedworld.com/how-be32enc-and-le32enc-works-in-c/
static inline void be32enc(void *pp, uint32_t x) {
  uint8_t *p = (uint8_t *)pp;
  p[3] = x & 0xff;
  p[2] = (x >> 8) & 0xff;
  p[1] = (x >> 16) & 0xff;
  p[0] = (x >> 24) & 0xff;
}

or

static inline void be32enc(void* pp, uint32_t num) {
  uint32_t beNum = be32(num);
  memcpy(pp, &beNum, sizeof(beNum));
}

or

#if _BYTE_ORDER == _LITTLE_ENDIAN
  #define be32enc(pp,num)  (*(uint32_t*)pp = __builtin_bswap32(num))
#else
  #define be32enc(pp,num)  (*(uint32_t*)pp = (num))
#endif

or whatever is a correct and proper and efficient solution on a given platform.

[michaeljclark]

yes, it would be possible to do this but I do not know if it is a good idea.

something composed with respect to the primitive conversion functions might be cleaner and should produce decent code via layering. the macros have a Microsoft mode that composes explicitly with bswap while LLVM detects swaps via shifts or unions:

static uint32_t be32_load(uint32_t *addr) { return be32(*addr); }
static uint32_t be32_store(uint32_t *addr, uint32_t val) { return *addr = be32(val); }

it is my opinion that the current htonl and ntohl style interfaces are a tiny bit of an ignorange layer in the sense that if you read the docs, there are apparently two functions for converting to and from but they are just aliases for the same function, and the revised interfaces are simpler than using bswap because they are host endian sensitive. the point here is that we already standardize bswap. we have the BSD host to network swap interfaces. we have proposed these revised simple interfaces. what exactly is the benefit of a wrapper around a load or store in light of those?

I think it may be more useful to add memcpy variants with two host endian sensitive copies and an explicit swap:

• memcpy_le{w}(void *dest, const void *src, size_t n);
• memcpy_be{w}(void *dest, const void *src, size_t n);
• memcpy_bswap{w}(void *dest, const void *src, size_t n);

constraints with respect to alignment and non-width size intervals would need to be nailed down:

• question whether to allow n that is not a multiple of the width. this has some limited use-cases.
• in the simplest case one would assume source and target are aligned to the width.
• in the complex unaligned case one would have an offset modulus the width for source and target.

crypto libraries often implement an endian converting memcpy, but leave memory-to-register to the compiler. there is a rationale for load and store interfaces to handle unaligned loads and stores but the cleanest way might be to compose via unaligned load and store functions. perhaps <stdalign.h> could add wrappers for unaligned load and store?

the simplified interfaces are adopted from practice in libraries like sodium and supercop, and add to those already present on the Unices, but most importantly, standardize existing best practices. I admit I have seen the load and store wrappers but once you see how they are composed, they become trivial and just add noise to what could otherwise be clean code.