[question] ability to modify HTTP headers inflight?

[warren-bank]

1. using this app, is there any way to configure rules that match by hostname to conditionally add/edit/remove either outbound request or inbound response HTTP headers?
2. if not, are you aware of another similar (non-root, VPN simulator) app that does have that capability?

for example:

1. rule:
   • hostname pattern: ^.*$
   • direction: response
   • mode: remove
   • header name: content-security-policy
   • header value: null

[emanuele-f]

Hello, this could be implemented via a mitmproxy plugin. Internally, PCAPdroid-mitm runs mitmproxy, so this could be intergrated into PCAPdroid-mitm, by providing a UI for such feature. I've recently done a similar integration to allow users to inject Javascript code, check out https://emanuele-f.github.io/PCAPdroid/advanced_features#46-js-injector and https://github.com/emanuele-f/PCAPdroid-mitm/blob/master/app/src/main/python/js_injector.py#L61, which in essence is a mitmproxy plugin.

Another option, quicker to test without modifing PCAPdroid-mitm, is to enable the socks5 proxy mode of PCAPdroid, redirecting the traffic to your mitmproxy instance running in SOCKS5 listener mode. This way you can just load your plugin on the mitmproxy instance via cli options, e.g. mitmdump --mode socks5 -p 8050 --flow-detail 0 -s your_plugin.py.

What is your use-case for this? It could be a nice feature to have in PCAPdroid

[warren-bank]

First off.. your app is amazing! I'm embarrassed to admit that I don't use it more often. But when I need to monitor network traffic, or add a quick firewall to block a particular app.. it's my go-to, and it works perfectly!

That said.. I've never dug any deeper.. into its implementation, or even being aware that it has any plugin architecture.. sufficed to say, that I'll be reading the code today.. as I'm very interested in all of the other potential use-cases.. the sky is the limit.

It's very amusing that your example of how to modify network traffic inflight.. involves injecting javascript into html responses.. and uses a Tampermonkey-esque userscript syntax to do it.

The reason I say so, is that the use-case that brings me here is pretty much exactly that. I have a very rough-around-the-edges Android app that uses a customized WebView to:

• intercept and save Tampermonkey-esque userscripts to a DB
• each time the URL location in the top window changes
  • compare the URL against all userscripts
  • conditionally evaluate javascript in the security context of that window

It works great, but with two important limitations.. that are both the result of WebView having a very limited API surface:

1. iframes
   • no API to monitor their loading lifecycle
   • no API to evaluate JS in the context of their child window
2. HTTP headers
   • no API to modify either request or response values
     • most importantly, content-security-policy in the response can block the evaluation of userscript JS

The suggested workaround is:

• prevent the WebView from ever making any network requests
  • intercept
  • proxy
  • make any desired changes inflight
  • hand off the response

I could do that, but I went googling for a more generic approach.. I figured a debugging proxy that functions as a VPN (such as your own).. might be a good option.

Now that I know there's a plugin system, I've got some reading to do.. I'm sure I could think of several other interesting use-cases.. but for the moment, a generic way to conditionally modify HTTP headers on a per-app basis would be amazing!

It's funny.. this topic (of HTTP header modification) feels like it's been plaguing me for years..

1. wrote an addon for Firefox (legacy)
   • Firefox (legacy) went away
   • Pale Moon is a fork
     • it still works
     • I doubt that anyone still uses it
2. wrote a webextension for Chromium and Firefox (modern)
   • it still works
   • I still use it

In any case, I'll dig into your repo and poke around.

Thanks again for writing such a truly great app, and especially for releasing its code with a GPL.

[warren-bank]

I was completely unaware of PCAPdroid-mitm.

Now that I've installed it, and played around.. I'm (once again) amazed by how good your app is.. or rather, your apps are.

---

here is what I did to run my own mitmproxy script:

1. read the docs :smiley:

2. wrote a very simple mitmproxy script to add both a request and response header:

     # saved on phone to file path:
     #   /storage/emulated/0/PCAPdroid-mitm/mitmproxy-scripts/http-add-header.py
   
     class AddHeader:
         def __init__(self):
             self.req_num = 0
             self.res_num = 0
   
         def request(self, flow):
             self.req_num = self.req_num + 1
             flow.request.headers["count"] = str(self.req_num)
   
         def response(self, flow):
             self.res_num = self.res_num + 1
             flow.response.headers["count"] = str(self.res_num)
   
     addons = [AddHeader()]

3. used apktool to modify a PCAPdroid-mitm APK downloaded from its releases
   • needed to edit: AndroidManifest.xml
     to add: <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
4. install the modified APK to phone
5. go into Android Settings > Apps > PCAPdroid mitm > Permissions to manually grant this permission
6. run PCAPdroid
   • go into app Settings:
     • Additional mitmproxy options = -s /storage/emulated/0/PCAPdroid-mitm/mitmproxy-scripts/http-add-header.py
   • choose a Target app:
     • I used WebMonkey to test
     • it has a simple WebView and prompts with the option to proceed when there is an issue with an HTTPS certificate
   • start capture
7. run WebMonkey
   • enter URL: https://httpbin.org/get

amazingly, it works perfectly..

• the server replies with a copy of the request headers it was sent,
  and they do include the header added by the mitmproxy script

• the PCAPdroid log shows that both the request and the response headers have been successfully added by the mitmproxy script

• ---

partly a suggestion, and partly a request..

1. add the READ_EXTERNAL_STORAGE to PCAPdroid-mitm
2. use the directory /storage/emulated/0/PCAPdroid-mitm as the root for various types of configuration files
   • /storage/emulated/0/PCAPdroid-mitm/mitmproxy-scripts is where I stored my test mitmproxy script
   • /storage/emulated/0/PCAPdroid-mitm/js-userscripts could be used as a directory for users to place their Tampermonkey-esque userscripts
3. when PCAPdroid-mitm starts, it could..
   • request runtime permission, if not yet granted
   • scan these directories to load all scripts
     • every file in mitmproxy-scripts is added to the command line for mitmdump with the -s option
     • every file in js-userscripts is used by the embedded js_injector.py mitmproxy script

I think this convention of scanning particular directories.. would make PCAPdroid-mitm highly customizable by the more "advanced" users. It essentially becomes plug-and-play.

[warren-bank]

as an after-thought..

there would be a benefit to moving the js_injector.py mitmproxy script.. from being embedded in the PCAPdroid-mitm app.. to this mitmproxy-scripts external directory:

1. it only runs when the permission to read from external storage has been granted,
   which would prevent the script from accidentally crashing the PCAPdroid-mitm app..
   by trying to read from the js-userscripts directory when it doesn't have permission to do so
2. it allows users the ability to tweak the script to their own liking,
   or delete it entirely

[emanuele-f]

Thank you very much for your feedback and detailed description of the steps for your integration. In essence we are trying to solve the same problem, which is the lack of proper Tampermonkey/plugins support in Android web browser apps, with a specific focus on webcast features.

My specific use case is to extract a video URL from a web page on a phone and play it on a firestick device, which lacks Chromecast. To accomplish this I'm using the following technologies:

• Brave browser, which has a powerful and customizeable adblocker and nice privacy features. In particular, I find very convenient the ability to load custom adblock filter lists via URL
• Js Injector via PCAPdroid mitm to extract the video URL from the website, add a cast button which triggers a share intent via navigator.share
• Use HTTP Shortcuts to send an HTTP request with the video URL
• On the firestick, use Just Player Receiver to receive the HTTP request and start Just Player to play the video

I'm quite happy with this, as the experience is very similar to the one of chromecast. In the past I've also tried your Android-WebCast app to extract the video URL but I remember I had some issues with it, missing adblock or problems with Android TV if I remember correctly. In my opinion, this approach to use a customized browser app has pros and cons. Some big pros are that you can extend the Javascript engine with native functions, e.g. start intents and access persistent storage, and of course provide a customized user experience (e.g. for the cast functions). But the cons are that the user is forced to use this app instead of his favorite browser, you need to maintain such an app and probably provide adblock and other features a user may need. I like this approach of using PCAPdroid-mitm as I can leave it running in the background and forget about it, only decrypt the specific websites which I'm interesting into, and use my favorite browser.

I've open https://github.com/emanuele-f/PCAPdroid-mitm/issues/8 and https://github.com/emanuele-f/PCAPdroid-mitm/issues/9 to track this request. I agree that this kind of extensibility would be very appealing for power users, so this is something I plan to implement soon. One limitation to keep in mind is in the integration of python/native libraries, which may require building them via chaquopy.

For the Js Injector addon, since it's integrated into the PCAPdroid-mitm UI, I think it's better to ship it with the mitm app. I will add the ability to enable/disable such addons from the mitm app. The ability to read user scripts from the local storage could be optional, so I'm not concerned about a possible not-granted READ_EXTERNAL_STORAGE permission, I will handle it.

[warren-bank]

You're right about the Ad Blocker that I've built into both WebCast and WebMonkey being inadequate.. they both include a small static list of hosts to block.

The implementation was intended to happen in two stages:

1. hard code the data and get the Ad Blocker plumbing in place and working
2. allow the user to update the data from a URL

But, I suppose I got lazy and never got around to the 2nd stage.

Personally, I usually like to run an Ad Blocker at a system-wide level.. though, in practice, I usually only do so on my rooted phones using AdAway.. since it does its job via the hosts file.. rather than needing to run its own virtual VPN, as is the case on non-rooted devices. Since my (inexpensive Chinese) TV box came rooted, it's not really something that I notice. When I run AdAway on a device with root, I always use v4.3.6.. which (I believe) was the last version that did not include any non-root VPN functionality.. as it's super light-weight and works perfectly for my needs.

If you're looking for (yet another) challenge.. a full-fledged system-wide ad blocker could be written as a mitmproxy script.. and included with PCAPdroid-mitm as an internal addon, shipped with the app. Some user configuration could be integrated into the UI for the app, such as:

• list of URLs for the chosen blocklists
• update frequency
• a button to trigger a manual update

Similar such mitmproxy scripts already exist, and could be used as the starting point for customization.. for example:

• by: epitron
  with: dependencies
  • adblockparser
  • re2

[warren-bank]

as a friendly comment.. regarding your methodology for transferring a video stream URL from a webpage to a video player app..

• I've found that most.. nearly all.. video hosts secure their stream by the Referer HTTP request header
• Just-Player-Receiver doesn't accept and forward this value.. it probably should
  • I'm not aware whether or not Just Player supports accepting this Referer value via Intent
  • personally, I use my own video player.. which I know does (..and then some)

[warren-bank]

PS: I've added a new repo that contains the mitmproxy scripts that I've written and successfully tested with PCAPdroid mitm. It'll be a growing collection. My version of PCAPdroid mitm currently has permission to both read from and write to external storage. You'd be welcome to improve upon.. enhance with UI.. and embed any script that you find useful.

[emanuele-f]

a full-fledged system-wide ad blocker could be written as a mitmproxy script

Yeah, compared to the system-wide adblock, this would be more powerfull, thanks to its ability to match URLs in decrypted traffic. But decryption without root is limited to a very small set of apps, mostly browsers. Anyway, this is not something I'm willing to invest time on for now

Similar such mitmproxy scripts already exist, and could be used as the starting point for customization

Interesting links, thanks for sharing

I've found that most.. nearly all.. video hosts secure their stream by the Referer HTTP request header

I see, this is also something I've also experienced in the past for some websites. I think many websites today rely on third-party CDNs, so they don't use this referrrer-based. Anyway, an extension of the Just-Player-Receiver and Just Player should solve this.

I've added a new repo that contains the mitmproxy scripts that I've written and successfully tested with PCAPdroid mitm

Thanks for sharing this. I will add the external storage permission and removal of CSP headers soon

Also if you would like to contribute to PCAPdroid/mitm your help is welcome. The mitm addon has quite a bit of build dependenices, but it should build just fine

[warren-bank]

I'm loving this :smiley:

I just updated my detect_media_files script..

it basically does the same thing as the WebCast app.. except it uses content-type response headers to detect media, which the app cannot do; instead, the app uses the file extension of the requested URL to guess at its mime type.. and isn't nearly as reliable.

so now, this detect_media_files script:

• detects audio and video media requested by any app that uses the mitm proxy
• provides an HTML list of all detected media by navigating (in a web browser app that uses the mitm proxy) to: this URL
  • it uses XHR to obtain the raw JSON data from: this URL
  • it reverses the sort order to show newest media first
  • it builds a DOM that:
  • display the media info
  • provides links to forward the media to an external website that belongs to my webcast-reloaded Chrome extension,
    and provides several utilities to use the media with other tools,
    or simply watch it in-browser
  • it can work in WebMonkey in conjunction with a special userscript,
    which makes the "Watch" button integrate with Android;
    it opens an implicit Intent..
  • to watch the media in any available media player app
  • to open in the WebCast app's internal video player,
    which can push the media to a Chromecast device
  • to open in the WebCast app's ExoAirPlayer sender Activity,
    which can push the media to an instance of ExoAirPlayer
  • to open in the WebCast app's HLS Proxy sender Activity,
    which can push the media to an instance of HLS Proxy
  • etc.. you get the idea :smiley:

[warren-bank]

I played around with that adblock script (mentioned earlier).. and got it working pretty nicely. It uses re instead of re2, which is probably a big performance hit.. but it still works really well.

[warren-bank]

I also played around with the general concept for your javascript injector script, and made a more generic script that is configured a little differently, but works in a very similar way.

[warren-bank]

I know you said that you liked the idea of having a directory for users to add their own mitmproxy scripts.. and opened PCAPdroid-mitm issue 8 to track its development.. but rather than globbing a list of all the .py files in a directory, and adding each to the mitmdump cli command with a -s option.. it occurred to me that each script defines an array of addons.. so a single script could be used to bootstrap an entire directory of other scripts.. by initializing each script individually, and copying the values from each script's addons list into a single aggregate array.

I wrote this run_all script.. and it works great. Now, in PCAPdroid.. when I configure:

• Additional mitmproxy options = -s /storage/emulated/0/PCAPdroid-mitm/mitmproxy-scripts/run_all.py

All of my other scripts are automatically initialized and their addons passed to mitmproxy. Even on the severely underpowered device (TCL A1) that I'm toying around with right now.. it all works really nicely.

Disclaimer: This particular script deals almost exclusively with Python module loading methodology. I should probably note that prior to my starting this issue, I'd never written a single line of Python.. so somebody with more experience with the language might be able to provide a better methodology to accomplish this task.

[emanuele-f]

I played around with that adblock script (mentioned earlier)

This is a great start for such a feature! I haven't checked the code in detail, but I assume that a linear search is performed, which combined with regex match time causes such a delay. Is there any strategy to avoid such linear search?

so a single script could be used to bootstrap an entire directory of other scripts

This looks good, I will check in more detail at implementation time. I don't mind being ideomatic at this time, the priority is to allow people to use their own scripts. Thinking about this, it would be great to have a repository from which people can search and download scripts easily directly from the app. I have some concerns about this, in particular regarding maintenance time and cost and content moderation, to avoid the spread of malicious scripts. Maybe for now a simple web page, listing links to suggested scripts, could be enough.

[warren-bank]

regarding the performance.. and implementation details.. of the upstream adblockparser library.. which my script copies.. nearly verbatim.. the only changes were:

• to add the "r" r"regex-pattern" string notation in two places..
  where it was called for.. and the python compiler was pointing out the problem.. but only as "warnings"
• to remove any reference to re2..
  which removed 2 parameters to the class constructor..
  and some code that performs runtime checks to conditionally load the module

The only big improvement that I can see.. is figuring a way to use re2 on Android. I believe that re2 is actually a part of the Android framework.. but I have no idea if any of the available Python wrappers could work.. to allow a script to be able to use the native library. To be able to move from re to re2 is (according to the library README) a 1000x speedup.

[warren-bank]

regarding the lack of a defacto repository for mitmproxy scripts.. I agree, it would be useful.. something akin to what greasyfork does for userscripts.. though, I wouldn't wish that maintenance nightmare on anybody.

the analogy is a good one.. especially since both types of script have major security implications.

continuing with that analogy.. even though I run tons of userscripts in Tampermonkey.. I've never once used one from greasyfork.. the code quality is horrible.. it's just quicker and easier to write my own (as needed) from scratch.. than to spend the time auditing and fixing somebody else's crap code.

I imagine the same would hold true for user contributions of mitmproxy scripts.

[emanuele-f]

Regarding re2, native packages need cross compilation with chaquopy. Such a speedup would justify time spent trying to build it. When I have some time I can look into building it, as I already have the environment ready.

Regarding the repo, I agree that a curated list is better than having a free to access repo. I've just released the new PCAPdroid version with the js injector support, I plan this stuff for the next release

[warren-bank]

I know that I've already said this, but it bears repeating.. I'm absolutely loving this! :heart:

WebMonkey has two major inherent limitations:

1. CSP can block userscripts from running
2. userscripts cannot be injected into iframe windows

PCAPdroid mitm provides a workaround for both, which can be turned on as-needed, and doesn't require any integration with WebMonkey.. it just works.

1. the disable_csp script fixes the 1st
2. the modify_html_files script provides a solution to the 2nd

---

Even though both my modify_html_files and your js_injector are capable of conditionally injecting JS code into HTML documents, all of the userscripts that I've written for Android depend upon the enhanced APIs provided by WebMonkey.. and I like the way it automatically updates my scripts.. so I'm not inclined to migrate away from it.. but PCAPdroid mitm can make it even better. Simply turning the VPN on.. gives WebMonkey super powers :smiley:

[warren-bank]

There is one thing that I'd like to propose..

in PCAPdroid.. under Settings.. there is currently:

• TLS decryption [toggle]
• Mitm setup wizard
• Additional mitmproxy options

I would propose adding an additional setting:

• Tunnel HTTP through mitmproxy [toggle]

The reason for including such an option.. is that the enhanced functionality made available through mitmproxy scripting.. as you've already embedded js_injector.. shouldn't only be limited to HTTPS traffic. Or rather, allowing the functionality to be extended to HTTP traffic should at least be an available option.

---

update: I spoke too soon.. I just ran a quick test, and realized that my assertion was wrong.. when mitmproxy (PCAPdroid mitm) is running, HTTP traffic does tunnel through the proxy.. and all mitmproxy scripts are executed.

sorry.. for the false alarm :shrug:

[emanuele-f]

Yeah, PCAPdroid can direct to mitmproxy any TCP connection. Only UDP is not currently supported.