WebCast

Android app to extract video (file/stream) URLs from websites and watch them elsewhere (internal/external video player, Google Chromecast, ExoAirPlayer).

Tour

• BrowserActivity is shown when the app is started, and includes:

  • a very basic web browser
  • address bar
  • WebView
  • actionbar icons:
  • bookmark toggle
    • add/remove current website URL to/from list of persistently saved Bookmarks
  • actionbar menu items:
  • Bookmarks
    • open drawer on left: Bookmarks
  • Videos
    • open drawer on right: Videos
  • Settings
    • open SettingsActivity
  • Exit
    • close all UI and exit the app
  • drawer on left: Bookmarks
  • contains a persistent list of:
    • website URLs that have been saved via the bookmark toggle icon
    • video URLs that have been saved via the Videos drawer
  • click a list item to:
    • open website URL in WebView
    • watch video URL
    • SettingsActivity determines the particular action to be performed
  • long click a list item to:
    • rename
    • delete
  • drawer on right: Videos
  • contains a transient list of video URLs that have been found on the web page that is currently loaded in the WebView
    • this list is cleared each time the WebView navigates to a new web page
  • click a list item to:
    • add video URL to list of persistently saved Bookmarks
    • watch video URL
    • SettingsActivity determines the particular action to be performed
  • long click a list item to:
    • delete

• SettingsActivity is started from the actionbar menu in BrowserActivity, and includes:

  • Video Player to select whether to watch videos using..
  • internal w/ Chromecast sender
    • start VideoActivity
  • external
    • start Activity chooser w/ an implicit Intent
    • action
      • android.intent.action.VIEW
    • data
      • video URL
    • type
      • mime-type for format of video
    • extras
      • referUrl
      • (String) referer URL
      • used by ExoAirPlayer
  • ExoAirPlayer sender
    • start ExoAirPlayerSenderActivity
  • HLS-Proxy configuration
    • start Activity chooser w/ an implicit Intent
    • action
      • android.intent.action.VIEW
    • data
      • http://webcast-reloaded.surge.sh/proxy.html#/watch/${base64_video}/referer/${base64_referer}
    • type
      • "text/html"
    • start HlsProxyConfigurationActivity
    • only when there is no Activity having a matching Intent filter
      • which should never happen, since any standard web browser should offer to handle this Intent

• VideoActivity is started when a video URL is watched using the internal video player, and includes:

  • ExoPlayer
  • displays an icon in lower right corner of video controls toolbar to toggle fullscreen mode on/off
  • Chromecast sender
  • displays an actionbar cast icon when at least one Google Chromecast is detected on LAN
  • when connected to a Chromecast
    • video URLs are communicated to the receiver app running in the Chromecast
    • the Chromecast receiver app loads the video URL in an embedded HTML5 video player
    • transfer of video data occurs directly between the Chromecast and the server that hosts the video URL
    • transfer would not be effected by any of the following events:
      • VideoActivity stopped
      • BrowserActivity stopped
      • WebCast app exited
      • Android device powered off
  • list of video URLs
  • click a list item to:
    • play video URL
    • if connected to a Chromecast:
      • on Chromecast
    • otherwise:
      • on Android, in ExoPlayer
      • all HTTP requests include the referer url

• ExoAirPlayerSenderActivity is started when a video URL is watched using the ExoAirPlayer sender, and includes:

  • WebView that loads a single web page
  • URL of the web page depends on version of Android
    • Android 5.0 and newer
    • version using ES6+ modern javascript
    • Android 4.x and older
    • version using ES5 compliant javascript
  • URL hash contains:
    • #/watch/${base64_video}/referer/${base64_referer}
  • web page reads data from URL hash and pre-populates fields:
  • video url
  • referer url
  • web page reads data from cookies and pre-populates fields:
  • host
  • port
  • https
  • provides a basic UI to control any ExoAirPlayer receiver app that is reachable through the network

• HlsProxyConfigurationActivity is started when a video URL is watched using HLS-Proxy configuration on a device without any available web browser, and includes:

  • WebView that loads an ES5 compliant web page
  • URL hash contains:
    • #/watch/${base64_video}/referer/${base64_referer}
  • web page reads data from URL hash and pre-populates fields:
  • video url
  • referer url
  • web page reads data from cookies and pre-populates fields:
  • host
  • port
  • https
  • clicking the Load Player button performs the following tasks:
  • configures a new video URL that redirects the HLS manifest through HLS-Proxy
  • redirects WebView to another ES5 compliant web page that provides optional endpoint destinations for the proxied HLS manifest
    • ES6 web page for ExoAirPlayer sender
    • ES5 web page for Chromecast sender
    • in WebView on Android 4.x and older:
      • video player functionality does not work
    • in WebView on Android 5.0 and newer:
      • video player functionality works
      • Chromecast sender functionality does not work

• ---

Important Caveats

• some video URLs may play in WebCast and ExoAirPlayer, but cannot play on Chromecast or other external video players

  • this can occur when a video URL is hosted by a server that uses the Referer HTTP request header to restrict access, which is a common strategy
  • WebCast and ExoAirPlayer have the functionality to configure the value of this header for each unique video URL
  • Chromecast receiver apps cannot change the value of this header because they are restrained by standard browser security policies
    • the specs for XHR and fetch forbid changing certain HTTP request headers, including Referer
    • the WebCast Chromecast receiver app
    • attempts to change the value of this header
    • reveals in the remote debugger console that this attempt raises the warning:
      • Refused to set unsafe header "referer"
  • other external video players would need to:
    • read the referUrl extra in the starting Intent
    • configure its HTTP client library to change the value of this header
  • HLS-Proxy provides a convenient general-purpose workaround
    • setup for integration with WebCast:
      1. install HLS-Proxy
         • npm install --global "@warren-bank/hls-proxy"
      2. install WebMonkey
         • app to open HLS-Proxy configuration
      3. in WebMonkey
         • install WebCast-Reloaded userscript
         • adds enhanced functionality to the ES5 compliant web page that provides HLS-Proxy configuration
         • enables a button labeled: Start App
         • when clicked:
           • start Activity chooser w/ an implicit Intent
           • action
             • android.intent.action.VIEW
           • data
             • proxied HLS manifest URL
           • type
             • "application/x-mpegurl"
      4. in WebCast
         • Settings > Video Player > HLS-Proxy configuration
    • usage:
      1. run HLS-Proxy
         • hlsd --port 8080 --req-insecure --useragent "Chrome/90"
      2. in WebCast
         • navigate internal WebView to a page having the desired HLS video stream
         • open the Videos drawer (on right)
         • click on the URL for the desired HLS video stream
         • click: Watch
         • an Activity chooser will start
         • WebMonkey will be included in the list of apps that contain a matching Activity
         • click: WebMonkey
      3. in WebMonkey
         • configure the location of your running instance of HLS-Proxy:
         • host
         • port
         • https
         • click: Start App
         • an Activity chooser will start
         • WebCast will be included in the list of apps that contain a matching Activity
         • click: WebCast Video Player
      4. in WebCast
         • click the Chromecast sender icon to cast the proxied HLS video stream to a Chromecast device

• the Android System WebView component is wholly responsible for the web browser experience

  • this component has a complicated history
  • without going into detail:
  • on versions of Android < 5.0
    • the WebView component is baked into the firmware
    • cannot be updated
    • does a poor job loading modern webpages, as the javascript language (ES6+) and html spec (HTML5) have changed significantly
  • on versions of Android >= 5.0
    • the WebView component is a standalone application
    • can be updated
  • on versions of Android >= 7.0
    • the WebView component is superseded by a component of the Google Chrome web browser, when it is installed and enabled
    • can be updated

• ---

Organization of Git Repo

• stale branches
  • 01-foundation
  • collection of small apps to develop and test experimental features
  • 02-webcast-httpclient
  • used an external HTTP client library: Apache HttpClient
  • used the Content-Type response header to detect video files
  • 03-webcast-okhttp
  • used an external HTTP client library: okhttp
  • used the Content-Type response header to detect video files
  • 04-webcast-filename
  • uses WebView to download all HTTP requests
  • uses regular expressions to detect file extensions associated with video formats in URL requests
• active branches
  • 05-webcast-filename-media3
  • uses WebView to download all HTTP requests
  • uses regular expressions to detect file extensions associated with video formats in URL requests
  • builds upon the 04-webcast-filename branch, and migrates from ExoPlayer to AndroidX Media3
  • 00-chromecast-receiver-app
  • WebCast Chromecast receiver app
  • gh-pages
  • WebCast Chromecast receiver app
    • the HEAD of this branch should always reference/alias the HEAD of the 00-chromecast-receiver-app branch
  • hosted by GitHub Pages

Highlights of Source Code

• identification of video URLs in outbound HTTP requests

  • BrowserWebViewClient
  • regex to detect video files

• same methodology as implemented by a web browser extension

  • "WebCast-Reloaded" Chromium extension
  • regex to detect video files

• ---

Legal

• copyright: Warren Bank
• license: GPL-2.0