[REQ] Generation of plain, dependency-free C code for microcontrollers

[MelanieT]

I'm using the openapi-generator extensively, but have so far been held back by the inability to generate dependency-free C-code that an be loaded onto an ESP-32 or similar, large microcontroller. Every server generator pulls in so many dependencies that the microcontroller's memory would filled up by the dependencies' code alone, if it were even possible to compile these dependencies for the target CPU at all. In order to open up the IoT to openapi, a simple C server stub would be a good solution.

It would, to be universal, probably need to work like this:

int default_api_process(const char method, const char url, const char inputBuffer, int inputSize, char outputBuffer, int *outputSize);

where the return value would be the HTTP status to send, the outputBuffer would be filled with the fully formatted (with all headers) html document, the inputBuffer represents the request body and the URL is the complete URL to process.

That would be useful on ESP8266, ESP-32 and other wifi-enabled IoT CPUs.

Using C++ would be possible as well, however, these small CPUs have no heap and therefore no operator new.

[zhemant]

Hello Melanie, We currently have C client Generator, on the client side we use only libcurl to send the request and there is no other library dependency. You can check the sample here: https://github.com/OpenAPITools/openapi-generator/tree/master/samples/client/petstore/c. To create server stub for IoT CPUs, I would need to know a few things as pre-requisite. I need to know if there is any inbuilt library for these where which act as the server. e.g libmicrohttpd is used in normal c codes to create server stubs. I read that #include this particular device has support for the server using this import which has server and client support which is base and on top of that we can try to use the code generated by openapi-generator

[MelanieT]

Hi,

thanks for getting back to me so quickly.

The idea with IoT is that there are constantly new, evolving designs, and that one therefore would NOT want to create a fully functioning server, but rather leave the actual TCP connection handling to user-provided chip specific boilerplate.

So there should not be any socket accept logic, any reply sending in such a server stub, but much rather, as I have prototyped above, that single entrypoint the user herself can feed the received data to. If anyone tried to create a full server, that would cause a number of issues:

• New or odd chips would have to be each individually supported
• There would be no option for the user code to do any pre- or post checks like authentication, which, on the IoT, may not be a standard HTTP auth mode
• User code could not divert URLs that don't belong to the APIs URL space to other functions without the server stub pre-massaging that data.

All these points lead tot he conclusion that a server that can be used in the IoT world would be like this:

char reply_buffer[1024];

mychip_handle_http_request(const request_struct *req) // Hypothetical chip API
{
    int bufsiz = sizeof(reply_buffer);

    int retcode = default_api_process(req->method, req->full_url, req->body, req->body_length, reply_buffer, &bufsiz);
    mychip_send_http_reply(retcode, null, reply_buffer, bufsiz);
}

The default_api_process would be the generated method, one for each api defined. It would be up to the user program to divvy up the URL space and decide which one needs to be called for a specific URL.

The main purpose of the server stub here is to remove the tedium of parameter packing and unpacking, the huge amount of boilerplate code that requires and the often happening copypasta errors. It is not to actually run the server, because in most IoT chips, the firmware does a pretty good job of it.

Try to compare ESP8266, ESP-23 and any Atmel Mega with a WizFi or similar HTTP-to-serial converter! You can easily see that running the server itself must remain outside the scope of the server stub in order for it to be useful.

[zhemant]

It is possible to create default_api_process API without any library dependency. We can use similar structure of c client which has cjson and list library integrated with minimalistic codes which makes c client light and library independent (except libcurl).

About reqest data, this will contain only the required data which was sent as body or it will contain complete request including header and body ?

[MelanieT]

In IoT, it usually has the whole request. Parsing is left to the user as in most cases no parsing is even needed, so the overhead isn't incurred by default. In most cases, there is a thin layer implementing the verb-url-version reception, then the whole thing is handed off to the user. The GET or POST line is not usually part of this data. SImilarly, the hypothetical HTTP reply function shown has a null in the place a real one may have space for extra headers. Here again, it is assumed that the retcode-message text, e.g. HTTP 200 OK, is generated by the chip's libs and that then the headers block is dumped out, followed by a blank line, then the body block. If the stub needs to return response headers, it would have to give something like a const char const , similar to argv[], for greatest flexibility.

[zhemant]

For the API, what I would suggest is we make is as below:

mychip_handle_http_request(const struct request *req) // Hypothetical chip API
{
   struct response *resp;
   if(!default_api_process(req, &resp))
          printf("error");
   mychip_send_http_reply(resp);
}

Here, req and resp struct will have all necessary field which should be present for request and response body.

I would ask you if possible could you write a sample code for the server which only accepts a request and sends a reply with string and 200 OK for microprocessor you are using, with a structure similar to the above structure. I can find a way to generalise it so that it can be used with any microprocessor.

[MelanieT]

That would mean allocating a struct (as opposed to using formal params) and then translate from that chip's struct to an abstract struct, then on return translate back. Cycles are precious on microcontrollers and that overhead isn't desirable. Also, there is no heap. no malloc, no free, no new, no delete. All allocations on many MCUs are static, so this needs to work with statically allocated data and formal params only. This is why the raw buffer approach is what is needed in practice. Your code above assumes that the MCU will have a http library that will parse out a request fully. Many don't, though. The ESP-32 has a pretty usable one, ESP8266 has a only binary buffers (because all it provides are sockets) and no Atmel/Microchip controller (except ARM baed ones) have any HTTP support at all (they use a serial line!). What you get is verb, url, http version (sometimes) and the request raw bytes. What you send is the response code and the raw response bytes. Many MCUs don't parse anything and HTTP parsing is not for the faint of heart to do right. What precisely are your objections to

int default_api_process(const char *method, const char *url, const char *inputBuffer, int inputSize, char *outputBuffer, int *outputSize);

?

[MelanieT]

Just speculation - it it maybe that the generated code would need to include a HTTP parser? If so, that is precisely the intention, because, as I said, many MCUs don't have one.

[zhemant]

I have no objection to what you have proposed. I had some different idea in mind. But I am getting a better picture now. I am not experienced with MCU's I am good with C. so learning about MCU's and how coding is working in there.

https://pastebin.com/3JJThk8V this is a sample code I took from google. This is for ESP8266 I suppose from the import what I could see. This code has 3 imports. Are these 3 header imports present in most of the MCU's? If it is possible to have them then what I see from the code it is manageable to generate code using openapi-generator but I think it will be supporting basic functionality for start not complex structure. Basic means simple GET on data/status of pins or PUT/POST to update the status of pins.

[MelanieT]

If you see Serial.Begin in any code, or any mention of "arduino", please stop reading and disregard the source and it's information. The "arduino" framework is a framework that was created to allow children to use the easy-to-use but totally feature-free arduino IDE to create programs for blinking LEDs and such. The very basic premise of arduino is to not use the hardware capabilities of an MCU but instead emulate everything in software. That is so that every pin can be used for every function, nonwithstanding the fact that "bit banging" everything eats so many cycles that no serious use can be made of the MCU anymore. No IoT professional (I am one) would be seen dead using it!

Again, the code for MCUs that would fill a very real need would include a HTTP parser, a HTTP response generator and JSON decode/encode in the generated code, not depending on anything at all the MCU may or may not supply.

At it's very base level, the MCU is a bit of silicon, it has no built-in libraries at all! Anything else is added on top, and wildly different implementations of various things exist.

Just to point out the most glaringly different one, HTTP via Hayes AT protocol!

So, please, try not to think "framework" and don't try to use any ready made software. The code that is generated should be runnable on the bare metal, server operation, reception, transmission and really anything apart from taking a raw body and outputting a raw body, isn't in scope for an IoT server stub.

Trying to leverage anything a given firmware may provide, or a given framework like (puke) "arduino" provides, would limit more options than it opens. No serious professional IoT app uses the bitbanging heap-of-manure called arduino, it was designed for teaching primary school children, not for real world applications!

So the generated code would need to bring it's own http parsing, building and json decode/encode. It's the price to pay for being truly usable on the IoT platforms vs. being a "nice try".

[wing328]

From what I read, I think we better off creating a new C client generator for IoT platforms instead of modifying the existing one to support all use cases.

[zhemant]

@MelanieT could you please direct me to any example library which i can refer to how codes are written on the devices you are using so we stay on correct path. Also openapi-generator is responsible for a structure of code, we will need http server/client and json support from other library which are used in IoT environment.

@wing328 Yes for sure we cannot modify current C client Generator. It works with pointers and memory allocations from what I understood of IoT devices it is not possible.

That would mean allocating a struct (as opposed to using formal params) and then translate from that chip's struct to an abstract struct, then on return translate back. Cycles are precious on microcontrollers and that overhead isn't desirable. Also, there is no heap. no malloc, no free, no new, no delete

[MelanieT]

I can only reiterate again, there are no libraries in most cases. MCU devices are too small to carry that kind of overhead, code is usually copypasted together, with functions not needed deleted. There are no frameworks, no other work to build on and no common method to access the internet and / or serve.

That is why that server stub should not even try to interface with the MCU, but create standalone code. It would be up to the user to hook it up to the internet services his particular MCU or board provides. There is too much variety to make this generic.

[zhemant]

Can you share some opensource project link for MCU's you are working with which handle JSON and HTTP?

[MelanieT]

@zhemant There is a difference between HTTP and the implementation of the server socket. The server socket stuff is MCU specific, you can't have a generic one as they depend not only on the MCU but also on the board it is on. HTTP as such, as in parsing headers and bodies, will need to be included in the client. A starting point for ESP8266 is here:

http://www.no-bullshit.net/httpd.c http://www.no-bullshit.net/httpd.h

Now, this code is very, very specific to one particular chip. You would not want to generate any of the espconn?* stuff, for instance.

[MelanieT]

@zhemant Regarding parsing JSON, there are a number of useful snippets around. I can look further, but i've always built JSON by hand. Of course, parsing it, you'd want to be a bit more robust. Please do remember some of these chips have as little as 32 Kbytes program memory and as little as 256 bytes of RAM! I think it's realistic to consider devices from 32KB flash and 1kb ram upwards. That means there is a tradeoff between robustness of parsing and memory required to do it. It is understood that smaller processors will not be able to process long messages. Limitations are the IoT's middle name.