[Suggestion] Features

[lionelhorn]

Hello,

I've been using HttpToolkit for a bit. I was using fiddler and mitmproxy in the past.

Here are a few feature suggestions

1. Column customization in list view (like response payload size)
2. Delete selected request/responses according to filters (or with CTRL+click) instead of all of them.
3. Don't capture some request/responses according to filters.
4. Import/export filters (to better use 3.)
5. Transform response contents (like in JsonEditorOnline or with jq / jsonpath)
6. Replay request (with modification) (like in Fiddler)
7. Ask for confirmation to load the content on big responses payload (as loading the json viewer on big files hangs for a bit for instance)

@pimterry

[pimterry]

Hi @lionelhorn! Thanks for the feedback, that's very useful. I think a few of these are already tracked elsewhere:

• Column customization in list view https://github.com/httptoolkit/httptoolkit/issues/222

• Replay request (with modification) https://github.com/httptoolkit/httptoolkit/issues/48

• Transform response contents

  For simple transormations (replace values by key, clear values, add new values to array) this is already possible within transform rules.

  For more complex cases, I think https://github.com/httptoolkit/httptoolkit/issues/37 should cover this better - rather than build a complex and still limited UI, you should be able to quickly write a script, which would allow you to use any arbitrary logic you like. Most of the plumbing for this is already in place as its built into Mockttp, it's just a question of exposing it in the UI. Would that work for you?

• Delete selected request/responses according to filters (or with CTRL+click) instead of all of them. https://github.com/httptoolkit/httptoolkit/issues/76 plus a way to select all currently filtered requests should provide this.

• Ask for confirmation to load the content on big responses payload (as loading the json viewer on big files hangs for a bit for instance)

  That's an interesting idea! I haven't considered this before. I'm surprised this comes up much though...

  I did have somebody else mention this recently too, but on my machine (an new-ish i5 but nothing special) viewing a 10MB JSON file takes about a second, with the UI remaining fully responsive at the same time so it doesn't block you, or a 100MB file takes a little over 5 seconds (admittedly less responsively) which doesn't seem too unreasonable. My expectation is that even 10MB is an unusually large JSON file to see in typical HTTP traffic. Can you explain more about the workflow where this would be useful? Are you often searching through big streams of many huge JSON files that you don't want to see? Or is it that you're having issues with much smaller JSON content? If the latter then that's a bug I'd like to just get fixed entirely.

  As a quick workaround on this in the meantime by the way - the body rendering is all done lazily, so if you just collapse the body card before selecting the request then it won't load the content and everything should be instant.

• Don't capture some request/responses according to filters. Can you explain more about the use case for this? I'm not sure I understand why you'd want to do that.

• Import/export filters Exports are actually already filtered - if you enter a filter and then export, only the matching data will be exported. Can you explain more about why you'd want to filter imports though?

For each of those existing issues, you can add a :+1: there to vote for that, or subscribe to the issue via GitHub for notifications once that's available.

[lionelhorn]

Thanks for the quick reply. I voted on the relevant issues.

• Ask for confirmation to load the content on big responses payload (as loading the json viewer on big files hangs for a bit for instance) Can you explain more about the workflow where this would be useful? Are you often searching through big streams of many huge JSON files that you don't want to see?

I will need to find the payload again but I think it took longer than a few seconds on a i7-10750H CPU.

Don't capture some request/responses according to filters. Can you explain more about the use case for this? I'm not sure I understand why you'd want to do that.

Sometimes the android os does not enable me to filter the app I want to intercept the traffic from using the httptoolkit helper app because of a bug in lineage/cyanogen. So I get loads of unrelated requests. And I know in advance I'm not interested in all. (Like google play/telemetry requests) aren't relevant to what I want to observe.

Import/export filters Exports are actually already filtered - if you enter a filter and then export, only the matching data will be exported. Can you explain more about why you'd want to filter imports though?

Good to know. Being able to mark specific requests by CTRL+click without having to build filter would be nice. And even a small text box to be able to add comments. Like what the request does.

[pimterry]

I will need to find the payload again but I think it took longer than a few seconds on a i7-10750H CPU.

Ok, that would definitely be interesting. If you see this issue again, please open a bug with details over here.

Sometimes the android os does not enable me to filter the app I want to intercept the traffic from using the httptoolkit helper app because of a bug in lineage/cyanogen.

Hmm, interesting. I'll think about this... There is a related issue on this at https://github.com/httptoolkit/httptoolkit/issues/78. That's focused more on compliance etc but I think the same logic applies. The challenge is the UX here, as it's quite complicated, and there's the potential for many layers of filtering to become confusing.

Really it just sounds like Lineage needs to fix the bug in their VPN implementation :smile:. On correctly working Android devices, the per-app filtering in the Android app should significantly reduce the background traffic noise for you. There's also an ongoing project to do targeted per-app interception via Frida that's coming over the next year too: https://httptoolkit.com/blog/frida-mobile-interception-funding/.

In the meantime, one thing you can do with Pro via the advanced settings is to configure TLS passthrough, where HTTPS & TLS to certain domains skip the proxy entirely. Those will still appear in the log, but just one line per connection (instead of per-request) and the traffic is forwarded without interception. That might be useful for your case, if there's a small set of domains you know you want to ignore.

Being able to mark specific requests by CTRL+click without having to build filter would be nice.

Definitely! I think that should be covered by the multi-selection UI - once that's in place you'll be able to select arbitrary requests and do things like export/delete/mock/pin on the whole group.

And even a small text box to be able to add comments. Like what the request does.

There's a feature tracking this here: https://github.com/httptoolkit/httptoolkit/issues/241

---

I think that covers all the points here in issues elsewhere, and I'd rather manage each of these in separate issues, so I'm going to close this for now. Do feel free to reply on any of those other specific issues if you want to discuss them further though, and thanks again for the in-depth feedback!