search

gingeleski / headbro

Headless browser rendering service for HTTP responses.
GNU General Public License v3.0
4 stars 0 forks source link

Implement use of a full request string as an input vector #25

Closed gingeleski closed 5 years ago

gingeleski commented 5 years ago

See #24 for details - that was a research task, this covers the actual implementation.

gingeleski commented 5 years ago

The more I look into how to set up this request interceptor, the more it looks like we might have to hit BrowserMob directly instead of going through the Python library.

Not sure all the hooks to tweak an interceptor as we need it are in the Python library right now.

gingeleski commented 5 years ago

"Javascript request filters have access to the variables..."

So as an example you might...

POST http://localhost:3000/proxy/9091/interceptor/request

With headers:

Content-Type: text/plain

And body:

request.getMethod().removeHeaders("User-Agent"); request.getMethod().addHeader("User-Agent", "Bananabot/1.0");
gingeleski commented 5 years ago

Just needs testing now.

gingeleski commented 5 years ago

And a documentation update before closing!

gingeleski commented 5 years ago

Example 1 from the comments of #24 as (what I think is) a JSON-safe string...

{"request_string":"GET /i/js_inst?c_name=ui_metrics HTTP/1.1\nHost: twitter.com\nConnection: close\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36\nAccept: */*\nReferer: https://twitter.com/\nAccept-Encoding: gzip, deflate\nAccept-Language: en-US,en;q=0.9\nCookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCCwEHXBnAToMY3NyZl9p%250AZCIlZTVlYjQ4YjBiMzNlZjkxYTNjNDk1MjJiOWMyNzc1NDg6B2lkIiUzZjBh%250AZWMzODQyNzA2YzNiNTk4ZTBmNzEyOWU5YjI3Yw%253D%253D--130d99ec83024c78af96cab8e27c93f2fa9ad533; personalization_id=\"v1_uMFXwPN0Vl1s72/E3SuAMQ==\"; guest_id=v1%3A154377420906496841; ct0=56b8e8eee00322f4b805c60d8bf2bad3\nDNT: 1"}
gingeleski commented 5 years ago

Interceptor doesn't appear to be working yet. Sent a POST of the payload from the previous comment, then this spat out in console -

DEBUG: url = http://twitter.com/i/js_inst?c_name=ui_metrics
DEBUG: method = GET
DEBUG: headers = {'Host': 'twitter.com', 'Connection': 'close', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) App
leWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36', 'Accept': '*/*', 'Referer': 'https://twitter.com/',
 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.9', 'Cookie': '_twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0a
W9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCCwEHXBnAToMY3NyZl9p%250AZCIlZTVlYjQ4YjBiMzN
lZjkxYTNjNDk1MjJiOWMyNzc1NDg6B2lkIiUzZjBh%250AZWMzODQyNzA2YzNiNTk4ZTBmNzEyOWU5YjI3Yw%253D%253D--130d99ec83024c78af96cab8e27c
93f2fa9ad533; personalization_id="v1_uMFXwPN0Vl1s72/E3SuAMQ=="; guest_id=v1%3A154377420906496841; ct0=56b8e8eee00322f4b805c6
0d8bf2bad3', 'DNT': '1'}
DEBUG: generated canary rEUnZkYu
127.0.0.1 - - [04/Dec/2018 20:47:36] "POST /render/string HTTP/1.1" 200 -

With this coming back as the response -

{"status_code": 0, "headers": [], "alerts": [], "confirms": [], "prompts": [], "errors": [{"level": "SEVERE", "message": "http://areunzkyu.com/ - Failed to load resource: the server responded with a status of 502 (Bad Gateway)", "source": "network", "timestamp": 1543974455616}], "messages": [], "body": "<html xmlns=\"http://www.w3.org/1999/xhtml\"><head></head><body>Bad Gateway: http://areunzkyu.com/</body></html>"}

Problem seems to be interceptor not doing its job.

gingeleski commented 5 years ago

Going to document this then merge it in and file this under a new bug report.

This isn't a breaking change, per se.

gingeleski commented 5 years ago

For future reference I've been making updates to swagger.json using https://openapi.design