Fix #37 - Encode non utf-8 decodeable binary payloads to base64 when exporting as har

httptoolkit / httptoolkit-ui

The UI of HTTP Toolkit

https://httptoolkit.com

GNU Affero General Public License v3.0

292 stars 108 forks source link

Fix #37 - Encode non utf-8 decodeable binary payloads to base64 when exporting as har #38

Closed aluasm closed 2 years ago

aluasm commented 2 years ago

Fixes #37

CLAassistant commented 2 years ago

All committers have signed the CLA.

pimterry commented 2 years ago

I've done a quick test of existing approaches to this, by running the following in browser consoles:

fetch('https://httpbin.org/post', { method: 'POST', body: new Blob([new Uint8Array([0xa4, 0x26, 0x37, 0x13, 0x35, 0xe7, 0x50, 0x27])]) })

That sends your data from #37 as a POST body.

HTTP Toolkit exports this as �&7\u00135�P'. The raw bytes there are efbf bd26 375c 7530 3031 3335 efbf bd50 27.
Both Firefox & Chrome meanwhile export it as ¤&7\u00135çP', i.e. c2a4 2637 5c75 3030 3133 35c3 a750 27. Still incorrect, but a bit closer!

Neither of them include it in any other more useful form.

No helpful answers there really. Might be worth matching the format of the others, but it's not the same as the raw data regardless. If there are any other tools that do export in some useful format we could match, that'd be very interesting, but it looks unlikely I think.

aluasm commented 2 years ago

No helpful answers there really. Might be worth matching the format of the others, but it's not the same as the raw data regardless. If there are any other tools that do export in some useful format we could match, that'd be very interesting, but it looks unlikely I think.

FF/Chrome output seems to be ISO-8859-1 and not UTF-8. I wouldn't worry too much about matching that, mainly because it's way easier to detect incorrect encoding on the latter.

If there are any other tools that do export in some useful format we could match, that'd be very interesting, but it looks unlikely I think.

Not much options around, maybe "Web-Debugging Proxy" solutions such as Fiddler/Charles/HTTPDebugger provide HAR export functionality that we could test, however, I highly doubt anything of interest would come up.

aluasm commented 2 years ago

4c7f0f9f44074db80ae19f572b88446f50c3c39e should address most of what was discussed.

Still room for debate on how _postDataDiscarded and comment fields can be interpreted internally.

aluasm commented 2 years ago

@pimterry any updates on this?

pimterry commented 2 years ago

Thanks for the PR @aluasm, now merged. That'll be deployed in just a few minutes, and then your HTTP Toolkit will update in the background.

Just in case you're not aware, HTTP Toolkit Pro is free for all open-source contributors. If you can let me know your email address, I'll set you up with a free Pro account. You can either share it here or email me directly at tim @ httptoolkit.tech if you'd prefer.