GreyCat
commented
6 years ago You're completely correct, C port has been in heavy discussion since almost the very beginning of the project, yet nobody ever created an issue about it (and that's bad, because it's hard to collect all these discussions in one place).
There are/were several major issues with C target, though. It become a somewhat lengthy review of what's been discussed over the years, but I believe I've remembered most of the points and tried to order them from most serious to least serious.
Completely different workflow in mind
It turns out that most people who need C support in KS have completely different workflow that what KS provides now. Right now, KS does a very simple thing: it gets a binary format serialization spec and generates API around it. It usually does zero transformations, except for very simple and technical ones (i.e. endianness and that kind of stuff) — whatever's in the format, it all is reflected exactly as is in the memory. C people usually strive for performance and memory efficiency and would prefer to not save stuff that can be used right away and then just thrown out.
A very simple example:
seq:
- id: len_foo
type: u2
- id: foo
size: len_foo
type: strThis is usually ok for many modern languages, but a lot of people who wanted C target automatically suggest that:
len_foomust not be stored in the structures that KS generates in memory at all — it must be used once during the parsing and then just thrown away- Given that we're talking about "string" data type, why not convert it into "pure C string", as most C stdlib functions expect it to be — i.e. no length information, just a zero byte termination. Of course, this also implies that (1) we'll need to allocate one more byte than
len_foofor that zero byte, (2) we need to actually put that zero byte into a string (although it's clearly not existing in the stream), (3) we're silently agreeing on dealing with zero-terminated strings, i.e.foocould never contain a zero inside it.
A more complex (and real-life) example is a typical parsing of any network packet, for example, an udp_datagram. Typical current vision of what KS might create is something like this:
typedef struct udp_datagram_t {
uint16_t src_port;
uint16_t dst_port;
uint16_t length;
uint16_t checksum;
char* body;
} udp_datagram_t;
udp_datagram_t* read_udp_datagram(kaitai_stream* io) {
udp_datagram_t* r = (udp_datagram_t*) malloc(sizeof(udp_datagram_t));
r->src_port = read_u2be(io);
r->dst_port = read_u2be(io);
r->length = read_u2be(io);
r->checksum = read_u2be(io);
r->body = read_byte_eos(io);
return r;
}It turns out that many users would be comfortable with completely different mechanism than "read function just fills in some structures in memory and returns a pointer to them":
- Some would want certain callbacks to be called every time an attribute is "parsed", and do not need it to be stored in single memory structure at all, i.e.:
void read_udp_datagram(kaitai_stream* io, udp_diagram_callbacks* callbacks) {
uint16_t src_port = read_u2be(io);
if (io->status != OK) {
udp_diagram_callbacks->on_error(io->status);
return;
}
udp_diagram_callbacks->on_read_src_port(src_port);
// ...
}-
Some suggested more complex pubsub-like models, so there's some intermediate machinery where user applies to "subscribe" to only certain events like "this part of structure is finally completely read".
-
Some users suggested that such low-level packet parsing usually happens on incomplete/fragmented packets/structures. Typically, in such a situation KS would just stop reading and throw an exception. In C, however, they would prefer to be able to continuously resupply additional stream buffer contents into a single "reader" procedure, which would keep track of what have been already "parsed" on previous iterations (and not invoke relevant callbacks twice), and actually even to be able to resume parsing from certain points.
Not an "everything is an expression" language
Simply put, almost everything we had before supported "every KS expression translates into target language expression" idiom. That is, if you need to do string concatenation, i.e.
seq:
- id: a
type: strz
- id: b
type: strz
instances:
c:
value: a + b... you do that a + b in one single-line expression everywhere. Even C++ allowed us to get away with a + b using std::string. In C, however, it traditionally boils down to many lines and temporary variables:
// Real-life code would be even more complex, probably with more checks, etc.
size_t len_a = strlen(a);
size_t len_b = strlen(b);
char *tmp = (char *) malloc(len_a + len_b + 1);
memcpy(tmp, a, len_a);
memcpy(tmp + len_a, b, len_b);
tmp[len_a + len_b] = 0;This issue, however, was more or less solved with advent of #146.
Complex memory management
What's not solved however, is that such arbitrary allocations of temporary variables sometimes result in more complex memory management and need for additional manual cleanup. In the example above, tmp would likely be used directly as c value, and thus there's no need to store it additionally. However, if multiple operations will occur, we'll either need to store these intermediate values, or use some clever logic to either reusing these temporary buffers (and/or avoid extra copying), or clean them up right after they're no longer needed (i.e. earlier than in object's destructor).
Actually, even "allocate everything on the heap" is not universally agreed upon in many C apps. So, typical parsing of user-defined type like that:
udp_datagram_t* r = (udp_datagram_t*) malloc(sizeof(udp_datagram_t));might be suggested to be replaced with passing a ready-made pointer to structure to fill into that read_* functions and creation of that udp_datagram_t on a stack of the caller instead.
No single standard library
For KS, we need some basic stuff like:
- Byte arrays, which could report length of the contents that they store. There are no standard structure like that in C:
typedef byte_array {
int len;
void* data;
} byte_array;- Strings (again, knowing its length and, ideally, encoding-aware). If we'll stick to traditional
char*strings, then we're getting hit with "no zero bytes inside" requirement, which might hurt some formats. - True element arrays which (1) know its size, (2) allow growth.
There are tons of "enhanced standard" libraries that do that, but there's no universal agreement on that. Probably roughly 80% of C applications roll something homebrew like that inside them. Out of "standard" implementations, there is glib, klib, libmowgli, libulz, tons of lesser known libraries, there's a huge assortment of string-related libs, array-related libs, etc. Out of them, probably glib is most well-known and well-maintained, but even a suggestion to use that frequently encounters a huge resistance in many C developers.
Another possible way (albeit not way too well-received by many developers) is to roll our own (yet another) implementation of all that stuff, and deal with ks_string*, ks_bytes*, ks_array*, etc, instead of char*, whatever_t[], etc.
No simple solution here, and whatever we would choose probably won't be accepted by many C developers. Probably if we'll implement support for top 3 (or top 5) popular libs that will cover at least some popular options.
Exception support
As we all know, C does not have any standard exception support, and typical KS-generated code relies on them a lot, i.e.:
r->src_port = read_u2be(io);
r->dst_port = read_u2be(io);
r->length = read_u2be(io);
// ...On every step, read_u2be might encounter end of stream (or IO error) and it won't be able to suceed parsing yet another 2 bytes. Typical solution for that in C is using return codes and passing value-to-fill by reference, i.e.:
int err;
err = read_u2be(io, &(r->src_port));
if (err != 0)
return err;
err = read_u2be(io, &(r->dst_port));
if (err != 0)
return err;
// ...Since Go support introduction (#146), that became possible, although probably it still be a pain-in-the-ass to use in C :(
Another quick "solution" for C is to use signals/abortions to handle these erroneous situations. In fact, it would even be ok in many use cases like embedded stuff, because things are not usually supposed to blow up there and if they do, then everything is lost already, there's no graceful exists, user interactions, "Send error report to the vendor" dialogs, etc.
Stream abstraction
Relatively minor and solveable issue, but still an issue: what would be a concept of "KS stream" be in C? Two popular options:
FILE*— usually it's not buffered, so many sequential "read_u2be" would translate into literal "read 2 bytes" syscalls, which is terribly inefficient. Besides, one can't read from in-memory array usingFILE*char*— just use in-memory array and screw everything else. On-disk file parsing can be done usingmmap, but this is (1) very platform-dependent, (2) pretty inefficient for lots of smaller files. And a question about handling IO errors (or at least end-of-streams) still remain, so we'll need a wrapper for that to store mapped length.
Probably C runtime would need to implement all these options and allow for end-user to choose. Nothing too scary, but still an issue to be solved.
Zorgatone
KOLANICH
arekbulski
smaximov
jonahharris
DarkShadow44
generalmimon
Hi, I would like to know if you would consider (or have any plans already) to port the project for use with "plain" C (other than C++ and C#). I would use it, and not all the systems (even embedded maybe?) support C++ and/or C#. Having a C version would enable portability on any system and even more languages with C bindings