moz-rewrite

Firefox add-on that functions as a light-weight (pseudo) rules-engine for easily modifying HTTP headers in either direction

Summary

Swiss Army knife for anyone interested in writing rules to intercept and conditionally modify HTTP traffic
a distinct set of rules are written for each:
- (outbound) requests
- (inbound) responses
all data sets use file-based persistence
rules can:
- add/edit/remove HTTP headers
- cancel the request
- redirect the request
- save a record of the request
saved requests can:
- be replayed using (supported) external download tools.
  replayed requests include all the HTTP headers and POST data in the original/saved request.

Video Tutorial

2+ hour long screencast that is a detailed walk-through of installation, configuration and usage

Features

regex patterns are used to match rules against the requested URL
rules are applied incrementally, until either:
- all rules have been processed
- a rule is marked as being final
rules are declared within a well-defined data structure.
this data isn't JSON; it is evaluated as javascript.
as such:
- all javascript data types are supported,
  including those that aren't representable using JSON.
  for example:
- comments
- regex patterns
  - literal _{(ie: perl)} notation: //
  - object constructor: new RegExp('')
- functions
- the declaration for the data structure can contain inline code,
  which is interpolated only once during evaluation,
  immediately after the data file is initially read from disk.
  for example:
- storing the output of a helper function as (part of) a static value
- calling an "immediately-invoked function expression" (aka: "self-executing anonymous function"), and storing its output as (part of) a static value
where a function is present, it will be called each time the rule is evaluated.
- rules are evaluated for every request and/or response.
- when functions are called, there will be contextual variables as well as helper functions in scope.
- the contextual variables will allow the function to return a value that is dependent upon the state of the request/response.
- the helper functions provide a library to perform tasks that:
- are commonly used to generate HTTP header values
- provide enhanced capabilities, unrelated to modifying HTTP header values
where inline javascript code is present, the javascript will only be evaluated once.
- this occurs when the rules are read from an external file and evaluated into a javascript array of rule objects.
- when this evaluation occurs, there is no contextual request or response.. so there are no contextual variables in scope.
- however, the same helper functions that are always available to functions (that are defined within the rules data set) will also be available at the time that the rules data set is initialized/evaluated.

Contextual Variables _{(in scope when functions are called)}

both requests and responses
- request.window_location = {}
  keys:
  - href: [string] full URI
  - protocol: [string]
    for example:
  - http:
  - https:
  - file:
  - username: [string]
  - password: [string]
  - host: [string]
  - port: [string]
  - path: [string]
  - query: [string]
  - hash: [string]
  - file_ext: [string]
- request.uri = {}
  
  _{same keys as: request.window_location}
- request.original_uri = {}
  
  _{same keys as: request.window_location}
- request.referrer = {}
  
  _{same keys as: request.window_location}
- request.method [string]
- request.headers = {}
- request.headers.unmodified = {}
  hash of all HTTP headers in original/unmodified request
- request.headers.updated = {}
  hash of all HTTP headers that the rules array data set (for requests) has incrementally modified at the current point of rules processing.
  - this hash is empty before any rules are processed.
  - as rules are processed in sequential order, any rule that matches the requested URL may specify updated HTTP headers which will be applied to this hash object.
  - when rule processing is completed, the final state of this hash will be applied to the HTTP request.
  - a string value will set/update the HTTP header corresponding to its hash key.
  - a boolean false value will be ignored.
  - a null value will remove the HTTP header corresponding to its hash key (if originally present).
request only
response only
- response.headers = {}
- response.headers.unmodified = {}
  hash of all HTTP headers in original/unmodified response
- response.headers.updated = {}
  hash of all HTTP headers that the rules array data set (for responses) has incrementally modified at the current point of rules processing.
  
  _{see additional notes under: request.headers.updated}
- response.status_code [integer]
- response.charset [string]
- response.content_length [integer]
- response.content_type [string]

Helper Functions _{(in scope when functions are called)}

always available
- atob(string_base64_encoded)
  decodes a string of data which has been encoded using base-64 encoding.
- base64_decode(string_base64_encoded)
  
  _{alias for: atob}
- btoa(string_value)
  creates a base-64 encoded ASCII string from a "string" of binary data.
- base64_encode(string_value)
  
  _{alias for: btoa}
- md2(string_value)
  returns the result of hashing the input string using the md2 crypto hash function
- md5(string_value)
  returns the result of hashing the input string using the md5 crypto hash function
- sha1(string_value)
  returns the result of hashing the input string using the sha1 crypto hash function
- sha256(string_value)
  returns the result of hashing the input string using the sha256 crypto hash function
- sha384(string_value)
  returns the result of hashing the input string using the sha384 crypto hash function
- sha512(string_value)
  returns the result of hashing the input string using the sha512 crypto hash function
- format_date(Date | parsable_date_string)
  returns the date as a UTC formatted string
- format_date(null, 1)
  returns the current date as a UTC formatted string
- log()
  for all usage patterns, refer to docs
  helpful tip: configure the javascript console to "enable persistent logs"
both requests and responses
- save()
  prepends a record of the current request to the Output File.
  this record will be available for replay via the view/replay saved requests dialog window.
request only
- redirectTo(string_URI)
  HTTP redirection works for any matching request URL.
  When the matching request URL is the top-level page loading in a browser tab,
  - the HTTP request is cancelled
  - window.location = string_URI
  When the matching request URL is a page resource that's referenced by a top-level page loading in a browser tab,
  - the HTTP request is redirected and the alternate resource is loaded in its place
  For an example of top-level page redirection,
  check out the recipe: redirect search engine queries from Yahoo to Google
  
  For an example that illustrates both top-level as well as resource redirection,
  check out the recipe: redirect Google favicon
- cancel()
  completely cancels the HTTP request.
  
  For an example,
  check out the recipe: light weight ad-blocker
response only

Data Structure

the same data structure (schema) applies to both requests and responses
request and response data are defined separately
the data structure is an array of objects, where each object represents a rule.
each rule can have the following attributes:
- url (required, RegExp)
- headers (required, object or function):
- key = name of HTTP header
- value = determines what action to take
  - new value of HTTP header
  - ignore all previous rules for this HTTP header,
    and leave the original value unmodified
  - remove HTTP header from request/response
- stop (optional, boolean or function):
  when:
- the url pattern of this rule matches the requested URL, and
- processing of this rule is complete
then:
- [false, default]: process next rule (in array)
- [true]: do not process any additional rules
while each request/response is processed by its corresponding rules data set,
when a url pattern match occurs for a rule,
when either the headers or stop attribute of the matching rule object is declared as a javascript function:
- contextual variables will be constructed and made availabe to the function that will convey information about the particular request/response being processed.
- the function is expected to return the proper data type; otherwise, its output will be ignored.
while rules are being processed, an internal list of updates is being created and incrementally updated.
when the processing of rules is complete, this internal list of updates are applied to the request/response.

Simple Examples

sample request rule(s):

[
    /* ****************************************************
     * all requests: add 3 custom headers
     * ****************************************************
     */
    {
        "url" : /^.*$/,
        "headers" : {
            "X-Custom-Sample-Header-01" : "Foo",
            "X-Custom-Sample-Header-02" : "Bar",
            "X-Custom-Sample-Header-03" : "Baz"
        }
    },
    /* ****************************************************
     * secure requests: cancel the 3 custom headers, and stop processing rules
     * ****************************************************
     */
    {
        "url" : /^https/i,
        "headers" : {
            "X-Custom-Sample-Header-01" : false,
            "X-Custom-Sample-Header-02" : false,
            "X-Custom-Sample-Header-03" : false
        },
        "stop": true
    },
    /* ****************************************************
     * all requests: update 1st custom header, cancel 3rd custom header
     * ****************************************************
     */
    {
        "url" : /^.*$/,
        "headers" : {
            "X-Custom-Sample-Header-01" : "Hello",
            "X-Custom-Sample-Header-03" : false
        }
    }
    /* ****************************************************
     * assertion #1: non-secure URL request
     * expected result:
     *     X-Custom-Sample-Header-01: Hello
     *     X-Custom-Sample-Header-02: Bar
     * ****************************************************
     */
]

sample response rule(s):

[
    /* ****************************************************
     * purpose: map an applicable 'content-type' to a finite set of resources
     *          as identified by file extension, when loaded from local hard disk.
     * ****************************************************
     */
    {
        "url" : new RegExp('^file://', 'i'),
        "headers" : function(){
            var $headers = {};
            switch( request.uri.file_ext.toLowerCase() ){
                case 'txt':
                    $headers['content-type'] = 'text/plain';
                    break;
                case 'css':
                    $headers['content-type'] = 'text/css';
                    break;
                case 'js':
                    $headers['content-type'] = 'application/javascript';
                    break;
                case 'json':
                    $headers['content-type'] = 'application/json';
                    break;
            }
            if ( $headers['content-type'] ){
                response.content_type = $headers['content-type'];
            }
            return $headers;
        },
        "stop": true
    }
]

More Complicated Examples

a collection of various interesting rules and useful examples has been dubbed the recipe book
it can be found in its own branch of this repo, named: js/data/recipe-book
users are encouraged to contribute (via push request) additional recipe examples

User Preferences

input: rules data
- HTTP Requests (outbound):
  - Enabled
    
    default: on
  on/off toggle
  - on:
    intercept HTTP Requests and apply its corresponding set of rules
  - off:
    disable this feature entirely
  - Path to Rules File
    
    default: ''
  _{refer to Comments / Implementation Notes for advanced usage}
  - Watch Interval (ms, 0 to disable)
    
    default: 0 (off)
  useful while writing/testing new rules.
  this feature will watch the rules file for changes, and reload its contents as needed.
- HTTP Responses (inbound):
  - Enabled
    
    default: on
  on/off toggle
  - on:
    intercept HTTP Responses and apply its corresponding set of rules
  - off:
    disable this feature entirely
  - Path to Rules File
    
    default: ''
  _{refer to Comments / Implementation Notes for advanced usage}
  - Watch Interval (ms, 0 to disable)
    
    default: 0 (off)
  useful while writing/testing new rules.
  this feature will watch the rules file for changes, and reload its contents as needed.
output: save()
- HTTP Request Persistence:
  - Enabled
    
    default: on
  on/off toggle
  - on:
    the save() helper function will save a record of the request to Output File
  - off:
    disable this feature entirely
  - Path to Output File
    
    default: ''
  _{refer to Comments / Implementation Notes for advanced usage}
  - Maximum Number of Saved Requests
    
    default: 10
  this feature is intended to prevent the Output File from growing too large
  - > 0:
    when a request record is saved to Output File, the data is prepended. If after this addition there are more records stored in the file (ie: N) than the specified maximum number of records (ie: X), then only the first X are retained… and the trailing (N-X) are removed.
  - 0:
    allow the file to grow without any limitation.
tools to replay saved requests
- common settings:
  - Path to Download Directory
    
    default: {DfltDwnld}
  _{refer to Comments / Implementation Notes for advanced usage}
- wget:
  - Path to wget executable
    
    default: /usr/bin/wget
  _{refer to Comments / Implementation Notes for advanced usage}
  - Command-Line Options for wget executable
    
    default: -c -nd --content-disposition --no-http-keep-alive --no-check-certificate -e robots=off --progress=dot:binary
- curl:
  a reference implementation that adds support for this tool exists in a separate branch: js/eval/replay/curl
  
  this hasn't been merged into the js/eval/master branch due to a small incompatability, which is described pretty well across both:
  - the release notes for: js/eval/v2.00
  - the commit message for: 32cb770

Hidden Preferences

extensions.Moz-Rewrite.debug

default: false

boolean
- true:
  enables debug log messages to be printed to the Browser Console
- false:
  suppresses these log messages
extensions.Moz-Rewrite.case_sensitive

default: false

boolean
- true:
  the alphabetic case of the URL and its components are preserved
- false:
  the URL and its components are always normalized to lowercase, which allows regex patterns to be written in lowercase and omit a case insensitive flag

Dialog Windows

Tools -> moz-rewrite -> user preferences
- same Options dialog as:
- Tools -> Add-ons
- Extensions
- moz-rewrite -> Options
- provides a graphical interface for the user to apply changes to the values of (non-hidden / user) addon preferences
Tools -> moz-rewrite -> view/replay saved requests
- saved HTTP Requests:
- list of all saved requests.
  for each, a checkbox is followed by the corresponding URL.
- common form field controls
  - replay selected requests using..
  button that displays a list of all supported download tools.
  this list currently contains:
  - wget
  - interactively identify each partial/incomplete download file
  checkbox that
  - when:
- one or more saved HTTP Requests are selected
- a download tool is chosen/activated from the list
  - if checked:
- for each of the selected saved HTTP Requests, an interactive file picker dialog will open and allow the user to choose the file path for the download.
- this workflow allows using the external download tool to be used to save data to arbitrary paths within the filesystem,
  rather than only to the Download Directory.
- this is particularly useful for when the browser begins a download, but fails to complete.
  in such a case,
- if the request was saved…
  or if the browser can re-request the download, and this subsequent request is saved…
  _{without actually saving the file, and certainly NOT over writing the previously downloaded partial/incomplete file}
- then the interactive dialog would allow the user to browse for this partial/incomplete file download
  - if not checked:
- for each of the selected saved HTTP Requests, the chosen download tool will begin saving/resuming the requested data.
- this data will be saved to a file in the Download Directory
- the filename will be determined by the external download tool.
  factors that the tool may take into consideration:
- a filename component of the requested URL
- a 'content-disposition' header of the response
- command-line options for the tool (in addon preferences)
  - fallback behavior when 'cancel' is chosen in interactive dialog
  checkbox that
  - when:
- interactively identify each partial/incomplete download file is checked
- an interactive file picker dialog is closed by the user without having selected a filepath
- cancel button
- close window (ie: "X") button
  - if checked:
- proceed with download and save to default directory
  - if not checked:
- skip replay of the specific saved request

Comments / Implementation Notes

data sets are stored in external files.
this allows them to be maintained using any text editor.
the addon asks to know the file path to each data set.
one for requests, one for responses.
there are two ways to specify a file path:
- browse for file, which stores an absolute path.
- manually enter the path, which is parsed in such a way that portable/relative paths are supported.
  - when this path begins with one of the following special tokens,
    the token will be replaced with the corresponding directory path.
  - the absolute path of these "special directories" may change from FF shutdown to startup,
    but the relative path will remain valid.
  - these "special tokens/directories" include:
  - {ProfD}:
    profile directory
  - {CurProcD}:
    current working directory (usually the application's installation directory)
  - {ProfDefNoLoc}:
    %installation%/defaults/profile
  - {PrfDef}:
    %installation%/defaults/pref
  - {Desk}:
    user's desktop directory
  - {Home}:
    user's home directory
  - {DfltDwnld}:
    default Downloads directory
  - {TmpD}:
    operating system's temporary files directory
  - sample interpolation values:
  - Windows, PortableApps:
    - {ProfD}:
      C:\PortableApps\Firefox\Data\profile
    - {CurProcD}:
      C:\PortableApps\Firefox\App\firefox\browser
    - {ProfDefNoLoc}:
      C:\PortableApps\Firefox\App\firefox\browser\defaults\profile
      
      _{(note: directory does not exist)}
    - {PrfDef}:
      C:\PortableApps\Firefox\App\firefox\defaults\pref
    - {Desk}:
      REG QUERY "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders" /v Desktop
    - {Home}:
      %USERPROFILE%
    - {DfltDwnld}:
      %USERPROFILE%\Downloads
    - {TmpD}:
      %TEMP%
  - so.. if portability is a concern, then the following file/directory paths should work nicely:
    
    _{(note: the specified paths must already exist; files/directories won't be created.)}
  - {ProfD}/moz-rewrite/requests.js
  - {ProfD}/moz-rewrite/responses.js
  - {ProfD}/moz-rewrite/saved_requests.txt
  - {ProfD}/moz-rewrite/downloads
when enabled (in addon preferences), the addon will watch input data files for updates.
when the path to an input data file is changed (in addon preferences),
or an input data file having a watched path has been updated (identified by its last modification date):
- the file contents are read into a string
- the string is evaluated as javascript
- the return value is validated for having the proper schema
- the rules array is stored (in memory)
when a request/response observer is notified:
- the corresponding rules array is processed, sequentially until complete
- the list of updates are applied to the request/response
the heavy lifting is converting the javascript entered by the user in the rules files into (in-memory) data objects.
- this occurs infrequently, only as-needed.
- the size of the (in-memory) data objects will be very small.
- the performance cost of calling functions within these data objects is trivial.
- this cost is farther reduced by using a strategy that counts the number of functions within the rules array data set during its validation.
- if there are no functions, then there's no need to create the contextual variables that would normally be available (in scope) to functions;
- when it's appropriate to do so, eliminating this step makes the performance cost (of processing the corresponding rules array data set) extremely low.

Alternate Implementations / Branches

js/eval/master:
- rule data files contain javascript, and are evaluated using the eval() function
- security context: system principal
  - pros:
  - has the ability to run protected code from within user-defined functions
  - cons:
  - would (most likely) fail AMO review.
    for security related considerations, they have a general policy to disallow any addon that uses the eval() function.
js/Cu.evalInSandbox/master (default, current):
- rule data files contain javascript, and are evaluated using the Mozilla Cu.Sandbox and Cu.evalInSandbox APIs
- security context: null principal
  - pros:
  - should pass AMO review
  - cons:
  - gives up the ability to run protected code from within user-defined functions
json/master:
- rule data files contain JSON, and are parsed using the native JSON.parse() function
  - pros:
  - would pass AMO review
  - cons:
  - gives up the ability to declare user-defined functions within the rules data sets
  - gives up the ability to perform actions:
    - that are conditional on the state of contextual variables
    - that are only available through helper functions

AMO _{(addons.mozilla.org)}

v1.03 of the json/master branch is available on AMO
v1.00 of the js/Cu.evalInSandbox/master branch is pending review on AMO

License

GPLv2 Copyright (c) 2014, Warren Bank

warren-bank / moz-rewrite

readme

moz-rewrite

Summary

Video Tutorial

Features

Contextual Variables _{(in scope when functions are called)}

Helper Functions _{(in scope when functions are called)}

Data Structure

Simple Examples

More Complicated Examples

User Preferences

Hidden Preferences

Dialog Windows

Comments / Implementation Notes

Alternate Implementations / Branches

AMO _{(addons.mozilla.org)}

License

warren-bank / moz-rewrite

readme

moz-rewrite

Summary

Video Tutorial

Features

Contextual Variables (in scope when functions are called)

Helper Functions (in scope when functions are called)

Data Structure

Simple Examples

More Complicated Examples

User Preferences

Hidden Preferences

Dialog Windows

Comments / Implementation Notes

Alternate Implementations / Branches

AMO (addons.mozilla.org)

License

Contextual Variables _{(in scope when functions are called)}

Helper Functions _{(in scope when functions are called)}

AMO _{(addons.mozilla.org)}