Send http requests/responses to executabels in an easy to parse format instead of text

[rew1nter]

Its hard to parse a raw http request or response manually as text when you have to remake the request. Piper could make parsing easy by sending requests/responses in some parsable format like YAML, JSON or XML.

[dnet]

Which programming language has a shortage of HTTP parsers yet supports YAML/JSON/XML? Also, these three are text-based, while HTTP can contain arbitrary binary content, so at least these formats would require Base64 or similar encoding, like Burp does when you save a request to an XML file.

What do you think about CBOR or ProtoBuf? These would be more compact and faster to parse/serialize, although people might be less familiar with them. ProtoBuf would be handy since it's already used by the project, so less bloat would be added for users who wouldn't use this feature.

If you submit a PR, I'd be glad to review it and accept if it fits the project.

[scarletturtle]

I write golang, which doesn't have any third party http text parser at all and a partial stdib support.

It's not about parsing speed. It's about making it easy for the executable to work with the piped http data. Yaml/json/har would have ready to use fields, which would be easy in case of rebuilding in order to resend the request and for modification before doing so. For this purpose we see tools like logger++ export and nuclei using such formats.

I'm not familiar with protobuf, but if it has easy to parse fields like yaml/JSON/har/XML, I'll be okay with it and get a hold of.

Lastly while I'd like to, I'm not familiar with java to make a pr 😄

[scarletturtle]

I just realized I'm commenting from another account. But it's me XD

[dnet]

I'm not familiar with java to make a pr

Then good news: it's not Java, it's Kotlin ;)

[scarletturtle]

Neither familiar with that XD

[dnet]

I write golang, which doesn't have any third party http text parser at all and a partial stdib support.

How about this? https://www.reddit.com/r/golang/comments/b5dtbh/comment/ejcrdbs/

[rew1nter]

This is the stdlib I talked about. It only works with http1

[dnet]

It only works with http1

Could you elaborate? What do you mean exactly by http1? Even when you enable HTTP headers, Piper will send textual representation of the HTTP transaction, even if Burp used binary HTTP/2 on the wire.

[scarletturtle]

It'll stop parsing the text if it sees http2 as proto

[scarletturtle]

I know the workaround you're thinking of. I thought about it too

[dnet]

Since that's only a string reference (as everything else is still text as in HTTP/1.1), I wouldn't care much about replacing it with a simple string replacing method/function. Also, if the targeted webapp/API works without it, you can disable HTTP/2 both for the client and the server "side" of Burp.

[scarletturtle]

The workaround I thought of. This is something I've scheduled for tomorrow to do.

[xssssrf]

1.Unable to know protocol scheme from text (http or https) 2.Unable to know remote address from text (host != remote address)

[dnet]

@sn00pyd0g3 both points are valid, although these could be solved in alternative ways, e.g. environment variables. Would that solve your particular problem(s) or do you also need what others mentioned above w.r.t parsing HTTP?

[xssssrf]

@dnet I think, it is possible to display the full url in http packet text. e.g.

GET https://httpbin.org/uuid HTTP/1.1
Host: httpbin.org
Connection: close

[dnet]

It is possible, however, it will alter the request itself, which requires

• parsing the request yet again before passing it to the script
• serializing it after prepending the real host to the path
• doing the same in reverse, since the server won't like such a format

This kind of in-band signalling doesn't look like something that's worth it and fits this project. But I can be persuaded if someone shows me how this would be better than using an out-of-band signalling like environment variables.

[xssssrf]

It is feasible to use environment variables. Another way is to use extra HTTP headers to attach information, such as PIPER-REMOTE-ADDR: https://example.com:443/.This has little additional impact and is easy to implement.

[dnet]

That's a better idea, thanks! I guess both should be opt-in, so how about an option like this:

• do not pass host/port information (default)
• pass host/port in faux HTTP header
• pass host/port in environment variable

[xssssrf]

Yes, don't forget protocol scheme. Thank you for making this awesome extension!

[dnet]

It seems that threading this information

• from the point where this metadata is still available
• to the point where it can be passed into the extension

is harder than I thought (before diving into the code), so for now this has a lower priority. When I'll have more time, I'll look into it, in the meantime, if someone else wants to take the lead on this, PRs are welcome. :)