Failure encoding utf-8 data in json POST

[myndzi]

I wrote a script to bulk upload a bunch of files to an API. One of the files was named yellow-café-au-lait-bowls-vintage_3.tif. The accented 'e' caused the API to return an error parsing the JSON.

The following addition to a helper function resolved the problem:

        if (method === 'get') {
            return bhttp[method](`${config.apiBase}${path}${query}`, opts);
        } else if (method === 'post') {
            return bhttp[method](`${config.apiBase}${path}`, params, opts);
        } else if (method === 'rawpost') {
            opts.method = 'POST';
            opts.encodeJSON = false;
            opts.inputBuffer = Buffer.from(JSON.stringify(params));
            opts.headers['Content-Type'] = 'application/json';
            return bhttp.request(`${config.apiBase}${path}`, opts);
        }

As far as I can tell, something about the post helper method being passed an object with utf-8 text in the value is causing problems. The API is https so I can't easily sniff the data; if this is enough info to go on then great. If you need more info, let me know what could help.

[myndzi]

Further investigation shows it to be a problem calculating the length of the content. I think this has also caused problems with some other files I was trying to upload, in the other direction (they don't appear to conclude their upload). Edit: The file-upload bits are using streams, so not sure what that's about and may or may not be related to bhttp; I also can't seem to upload them from the browser webface so I'd ignore this bit for now.

The failing request is missing the "} from the end of the JSON data, while the successful request is not. I assume this is because bhttp (correctly) only writes the number of bytes that it says it will write, but (incorrectly) miscalculates the bytes to be the length of the UTF-8 string in code points or something like that. Could probably be fixed by first converting to a buffer and checking the length of the buffer.

[myndzi]

Fault looks to be here: https://github.com/joepie91/node-bhttp/blob/master/lib/bhttp.js#L261 and here: https://github.com/joepie91/node-bhttp/blob/master/lib/bhttp.js#L268