Add browsing scanning feature through mitmproxy

[mp3butcher]

not to merge right now..just for review to know if you'll have some interest about such a feature...

How the feature works: When browsing through proxy all traffic is sent to OpenVideoDownloader on port 1200 as json and then can be analysed easily to find standard streaming tech in request response (HLS,Dash,partial-content video) then url and request headers are sent to ydl for download...

Check build here: https://github.com/mp3butcher/youtube-dl-gui/releases/latest (Note: I dl mitmproxy from https://github.com/mp3butcher/mitmproxy for zip convenience but you can check I haven't changed anything).

I added macos build too but haven't check if it works, so feel free to report macos problem here

A typical use case for this feature is when page of the video is dynamically loaded with javascript. But what's interesting is that it passes headers to ydl such as the mandatory referer for cross domain embed videos.

How to use scanner:

1. When scanner activated the first time don't forget to install mitmproxy authority certificate (cf image)
2. I recommend to use browser proxy switcher extension such as this extension to easily enable/disable use of proxy while browsing (set it up proxy=localhost and port=15930)
3. Browse to a streaming content and let the scanner capture the video

[StefanLobbenmeier]

Interesting… is there a particular use case for this? Or in which ways is this better than just setting the proxy setting and starting mitmweb yourself?

[StefanLobbenmeier]

Actually had another look and I think it would be easier to achieve a similar result via

--print-traffic Display sent and read HTTP traffic

[mp3butcher]

Actually had another look and I think it would be easier to achieve a similar result via

--print-traffic Display sent and read HTTP traffic

This option only output the traffic of ydl not the browser traffic

The purpose of the feature is to scan browser traffic to find videos (only HLS and video partial content for the moment)

[mp3butcher]

Any though? Do you think it worth to work on a proper pr? or do i keep my version as a fork?

[StefanLobbenmeier]

Currently I still did not quite get the use case for it, so likely keep it as a fork. I will take another look in December though when I have some more time

[StefanLobbenmeier]

The purpose of the feature is to scan browser traffic to find videos (only HLS and video partial content for the moment)

so basically like a browser extension?

[mp3butcher]

The purpose of the feature is to scan browser traffic to find videos (only HLS and video partial content for the moment)

so basically like a browser extension?

A browser extension can't do what ffmpeg can do.

How the feature works: When browsing through proxy all traffic is sent to OpenVideoDownloader on port 1200 as json and then can be analysed easily to find standard streaming tech in request response (HLS,Dash,partial-content video) then url and request headers are sent to ydl for download...

One can easily extends traffic analysis to find more urls compatibles with ydl As credentials generally pass through headers so it's also a way to pass automatically credentials to ydl

I also just code something working for livestream but it's a bit dirty (not sure a ffmpeg pass might be required for the file to be clean)

[mp3butcher]

Currently I still did not quite get the use case for it,

A typical use case is when page of the video is dynamically loaded page with javascript...so you can't get the url without further digging. But what's interesting is that it pass headers to ydl such as mandatory referer for cross domain embed videos.