[Feature Request] sendPost function

[TomTheGeek]

I would like to write power monitoring stats to an Influx time series database. InfluxDB only supports HTTP Post data writing - https://archive.docs.influxdata.com/influxdb/v1.2/guides/writing_data/

Please add an HTTP Post function that allows setting headers. Or point me in the direction of the source code that would need to be modified.

[openshwprojects]

Hello, we already have "SendGet" command, maybe we could create a "SendPost" as well? What kind of headers do you need? Can you show full sample POST request you need to make?

[TomTheGeek]

A "SendPost" would be great!

Here is some arduino code as example.

URL

http://10.0.0.1:8086/write?db=myInfluxDB

Header

http.addHeader("Content-Type", "application/octet-stream");

Post content

temperature,clientid=0A:00:27:00:00:06 value=56.27\nbrightness,clientid=0A:00:27:00:00:06 value=458\nhumidity,clientid=0A:00:27:00:00:06 value=45.2

[openshwprojects]

I can try to look into it if you want. Or maybe you can start looking into it first and I will help when it's needed? https://github.com/openshwprojects/OpenBK7231T_App/blob/5ab820c8190df549f201712e6093ed5d9d914430/src/httpclient/http_client.c#L1165 Honestly... we have HTTPCLIENT_GET... there seems also to be HTTPCLIENT_POST already... https://github.com/openshwprojects/OpenBK7231T_App/blob/5ab820c8190df549f201712e6093ed5d9d914430/src/httpclient/http_client.c#L290

[TomTheGeek]

It does seem like most of the code is already there. Maybe something like this? I haven't tried compiling this yet.

int HTTPClient_Async_SendPost(const char *url_in, int http_port, char content_type, char post_content){
    httprequest_t *request;
    httpclient_t *client;
    httpclient_data_t *client_data;
    char *url;

    // OBK UPDATE: use our own strdup which expands constants
    // So $CH5 gets changed to channel value integer, etc...
    url = CMD_ExpandingStrdup(url_in);  
    if(url==0) {
        ADDLOG_ERROR(LOG_FEATURE_HTTP_CLIENT, "HTTPClient_Async_SendPost for %s, failed to alloc URL memory\r\n");
        return 1;
    }

    request = (httprequest_t *) malloc(sizeof(httprequest_t));
    if(url==0) {
        ADDLOG_ERROR(LOG_FEATURE_HTTP_CLIENT, "HTTPClient_Async_SendPost for %s, failed to alloc request memory\r\n");
        return 1;
    }

       ADDLOG_INFO(LOG_FEATURE_HTTP_CLIENT, "HTTPClient_Async_SendPost for %s, sizeof(httprequest_t) == %i!\r\n",
        url_in,sizeof(httprequest_t));

    memset(request, 0, sizeof(*request));
    request->flags |= HTTPREQUEST_FLAG_FREE_SELFONDONE;
    request->flags |= HTTPREQUEST_FLAG_FREE_URLONDONE;
    client = &request->client;
    client_data = &request->client_data;

    client_data->response_buf = 3;  //Sets a buffer to store the result. HTTP return codes are 3 bytes long
    client_data->response_buf_len = 3;  //Sets the buffer size.
    HTTPClient_SetCustomHeader(client, "");  //Sets the custom header if needed.
    client_data->post_buf = post_content;  //Sets the user data to be posted.
    client_data->post_buf_len = sizeof(client_data->post_buf);  //Sets the post data length.
    client_data->post_content_type = content_type;  //Sets the content type.
    request->data_callback = 0;
    request->port = http_port;
    request->url = url;
    request->method = HTTPCLIENT_POST;
    request->timeout = 10000;
    HTTPClient_Async_SendGeneric(request);

    return 0;
}

[openshwprojects]

You can compile it online, when you create a fork. We have a Github build system. Otherwise I'll check this tomorrow and try to fix if there are any problems.

[openshwprojects]

Hm, "char" instead of "char*", you also don't strdup...

[TomTheGeek]

There will be lots of errors. Maybe should have called it pseudo-code.

I haven't programmed in C for very long time.

[openshwprojects]

You set responseBuffer to integer 3? client_data->response_buf = 3; //Sets a buffer to store the result. HTTP return codes are 3 bytes long It should be a buffer... but I am working on it ,wait I may need your help with testing, depending on how well it goes

[openshwprojects]

It receives soemthing, I need to find some dummy POST API online now so I can test.. with HTTP, ofcousre.. or maybe I will try to setup something quickly in node

[TomTheGeek]

Like I said it's been a long time... Please undo any stupid mistakes you find! I was thinking "204" is the standard response, which is 3 bytes...

[openshwprojects]

It receivd something:

[TomTheGeek]

Good news!

[openshwprojects]

backlog setChannel 10 13; setChannel 11 87; SendPOST http://localhost:3000/ 3000 "application/json" "{ \"a\":$CH10, \"b\":$CH11 }"

[openshwprojects]

btw, can you tell us more about your database setup? If you have a moment to spare, please consider creating a short tutorial about your database logging stuff/approach, when you get it working, and post it on our forums: https://www.elektroda.com/rtvforum/forum517.html we can send you a free gift for a tutorial, if you're interested: https://www.elektroda.com/rtvforum/topic3950844.html

I will do few more tests and commit POST code

[TomTheGeek]

That looks great, nice work. Looking forward to the commit.

I'll post some examples when I get the power logging setup. It's not very complicated, once OBK can post data over http InfluxDB does the rest.

[openshwprojects]

@TomTheGeek I pushed the changes commit, i didnt do much testing yet, please test and also watch heap, I hope there are no memory leaks, I will play around it after I get some sleep.

[TomTheGeek]

Excellent, will test ASAP

[TomTheGeek]

Got around to testing this just now, not sure if it's working yet.

Test command: SendPost http://192.168.0.101/write?db=energy 8086 application/octet-stream voltage,clientid=A0:92:08:CB:41:E1 value=$voltage\ncurrent,clientid=A0:92:08:CB:41:E1 value=$current\npower,clientid=A0:92:08:CB:41:E1 value=$power

log output

Info:CMD: CMD_SendPOST received with args http://192.168.0.101/write?db=energy 8086 application/octet-stream voltage,clientid=A0:92:08:CB:41:E1 value=$voltage\ncurrent,clientid=A0:92:08:CB:41:E1 value=$current\npower,clientid=A0:92:08:CB:41:E1 value=$power
Info:HTTP_CLIENT:HTTPClient_Async_SendPost for http://192.168.0.101/write?db=energy, sizeof(httprequest_t) == 124!
Info:CMD:[WebApp Cmd 'SendPost http://192.168.0.101/write?db=energy 8086 application/octet-stream voltage,clientid=A0:92:08:CB:41:E1 value=$voltage\ncurrent,clientid=A0:92:08:CB:41:E1 value=$current\npower,clientid=A0:92:08:CB:41:E1 value=$power' Result] OK
Info:HTTP_CLIENT:Parse url http://192.168.0.101/write?db=energy
Info:HTTP_CLIENT:host: '192.168.0.101', port: 8086
Info:HTTP_CLIENT:HAL_TCP_Establish: created socket 5

Error:HTTP_CLIENT:success to establish tcp, fd=5
Error:HTTP_CLIENT:connection is closed
Error:HTTP_CLIENT:httpclient - no response buff, skipped 145
Error:HTTP_CLIENT:shutdown error -1

Some errors and I'm not seeing any new measurements in the database.

There are some spaces in the content, not sure how to escape those.

[openshwprojects]

Tokenizer splits arguments by spaces. You need to take string into " . If you want to use " as " in data, then escape with \"

[TomTheGeek]

So I've had some success with this now after testing the data with Curl first. Turns out InfluxDB doesn't like the "\n" being used for newlines.

But power monitoring is working with these commands:

addRepeatingEventID 300 -1 101 SendPost "http://10.0.0.1/write?db=energy" 8086 "application/octet-stream" "voltage,clientid=A0:92:08:CB:41:E1 value=$voltage"
addRepeatingEventID 300 -1 102 SendPost "http://10.0.0.1/write?db=energy" 8086 "application/octet-stream" "current,clientid=A0:92:08:CB:41:E1 value=$current"
addRepeatingEventID 300 -1 103 SendPost "http://10.0.0.1/write?db=energy" 8086 "application/octet-stream" "power,clientid=A0:92:08:CB:41:E1 value=$power"

I could reduce it down to just one Repeating Event if I can figure out how to get the line breaks to be encoded properly.

Had it crash after about 15min just now had to manually power cycle it. Will keep testing but great progress regardless.

[TomTheGeek]

Confirmed it's crashing now after a few minutes. I will try adding a delay between the repeating events so they're not all happening at once.

[openshwprojects]

In worst case, we can just write a simple driver in C that will do the same, sending power/voltage/current seems like a very common use case.

What's the crash reason, what's the reboot reason shown, does it runs out of memory?

[TomTheGeek]

How do I check the crash dump? reboot reason is shown as "0 - Pwr"

Seems to happen even with just one repeating event. Always crashes within an hour on a 5 min repeating event.

[TomTheGeek]

The memory leak, or whatever is broken, is still broken for SendPost. I added some more calls to free() but the WiFi stack still fails after sending about 10 commands. The button on the device is still responsive when this happens.

Any suggestions on how to debug this? I've already got the code forked and have been able to compile binaries for testing.