Possible bug in BrowserMobProxyServer.rewriteUrl(match, replace)

[ghost]

Hi all,

I have the following method that I use to redirect traffic:

public void rewriteUrl(String match, String replace) {

        match = "^(https?://)" + match + "(.*)";
        replace = "$1" + replace + "$2";

        logger.Info("Replacing url: " + match + " with the url: " + replace);

        this.mProxyServer.rewriteUrl(match, replace);

    }

And I've noticed that it works perfectly in the legacy net.lightbody.bmp.proxy.ProxyServer, but doesn't work at all in the new net.lightbody.bmp.BrowserMobProxyServer. Just to be exact, it works if I start the server like this:

net.lightbody.bmp.proxy.ProxyServer mProxyServer = new ProxyServer()
mProxyServer = new ProxyServer();
mProxyServer.setPort(8888);
mProxyServer.start();
mProxyServer.setCaptureContent(true);

But doesn't work If I do this:

net.lightbody.bmp.BrowserMobProxyServer mProxyServer
mProxyServer = new BrowserMobProxyServer();
mProxyServer.start(8888);

[jekh]

The rewrite behavior in BrowserMobProxyServer is generally much more consistent than the legacy rewrite rules, and there doesn't appear to be anything unusual about your rules. Can you give an example of a full "match" and "replace" as well as a URL that doesn't work with them?

[ghost]

Not even the example rewrite from the documentation is working. Here's an example code and output, also attached project as file: rewrite-example.zip

package com.company;

import io.netty.handler.codec.http.HttpResponse;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.filters.ResponseFilter;
import net.lightbody.bmp.util.HttpMessageContents;
import net.lightbody.bmp.util.HttpMessageInfo;

public class Main {

    public static void main(String[] args) {

        BrowserMobProxyServer server = new BrowserMobProxyServer();
        server.start(8888);

        if (server.isStarted()) {
            System.out.println("server started on port " + server.getPort());
        }

        server.rewriteUrl("http://www\\.(yahoo|bing)\\.com\\?(\\w+)=(\\w+)", "http://www.google.com?originalDomain=$1&$2=$3");

        server.addResponseFilter(new ResponseFilter() {
            @Override
            public void filterResponse(HttpResponse httpResponse, HttpMessageContents httpMessageContents, HttpMessageInfo httpMessageInfo) {
                System.out.println();
                System.out.println("received response");
                System.out.println("original url: " + httpMessageInfo.getOriginalUrl());
                System.out.println("actual url: " + httpMessageInfo.getUrl());
                System.out.println();
            }
        });

        while (true) {
            // keep process running
        }

    }
}

Output:

received response original url: http://www.bing.com/?a=b actual url: http://www.bing.com/?a=b

received response original url: http://www.bing.com/rms/MobileSiteBase/cc,nc/8d0342e9/e3492d89.css?bu=rms+serp+Mobile%24mobileSite_c%2cMobile%24mobilesiteandroidhigh actual url: http://www.bing.com/rms/MobileSiteBase/cc,nc/8d0342e9/e3492d89.css?bu=rms+serp+Mobile%24mobileSite_c%2cMobile%24mobilesiteandroidhigh

received response original url: http://www.bing.com/fd/ls/l?IG=574353C6151541CEB3A5830E565C0733&Type=Event.CPT&DATA={%22pp%22:{%22S%22:%22L%22,%22FC%22:-1,%22BC%22:-1,%22H%22:-1,%22BP%22:204,%22CT%22:276,%22IL%22:2},%22ad%22:[-1,-1,360,615,360,615,1]}&P=SERP&DA=DB5 actual url: http://www.bing.com/fd/ls/l?IG=574353C6151541CEB3A5830E565C0733&Type=Event.CPT&DATA={%22pp%22:{%22S%22:%22L%22,%22FC%22:-1,%22BC%22:-1,%22H%22:-1,%22BP%22:204,%22CT%22:276,%22IL%22:2},%22ad%22:[-1,-1,360,615,360,615,1]}&P=SERP&DA=DB5

received response original url: http://www.bing.com/fd/ls/lsp.aspx? actual url: http://www.bing.com/fd/ls/lsp.aspx?

received response original url: http://www.bing.com/rms/Framework/jc,nj/8cabc1e5/278b8440.js?bu=rms+answers+BoxModel+config.instant%2ccore%2ccore%24viewport%2ccore%24layout%2ccore%24metrics%2cmodules%24mutation%2cmodules%24error%2cmodules%24network%2cmodules%24cursor%2cmodules%24keyboard%2cmodules%24bot actual url: http://www.bing.com/rms/Framework/jc,nj/8cabc1e5/278b8440.js?bu=rms+answers+BoxModel+config.instant%2ccore%2ccore%24viewport%2ccore%24layout%2ccore%24metrics%2cmodules%24mutation%2cmodules%24error%2cmodules%24network%2cmodules%24cursor%2cmodules%24keyboard%2cmodules%24bot

received response original url: http://www.bing.com/ImageResolution.aspx?w=360&h=615&hash=349a7bb85847e4aeee21d58328590cbb&a=b actual url: http://www.bing.com/ImageResolution.aspx?w=360&h=615&hash=349a7bb85847e4aeee21d58328590cbb&a=b

received response original url: http://www.bing.com/notifications/render?bnptrigger=%7B%22PartnerId%22%3A%22HomePage%22%2C%22IID%22%3A%22SERP.2000%22%2C%22Attributes%22%3A%7B%22RawRequestURL%22%3A%22%2F%3Fa%3Db%22%7D%7D&IG=574353C6151541CEB3A5830E565C0733&IID=SERP.2000 actual url: http://www.bing.com/notifications/render?bnptrigger=%7B%22PartnerId%22%3A%22HomePage%22%2C%22IID%22%3A%22SERP.2000%22%2C%22Attributes%22%3A%7B%22RawRequestURL%22%3A%22%2F%3Fa%3Db%22%7D%7D&IG=574353C6151541CEB3A5830E565C0733&IID=SERP.2000

received response original url: http://www.bing.com/ImageResolution.aspx?w=360&h=615&hash=349a7bb85847e4aeee21d58328590cbb&a=b actual url: http://www.bing.com/ImageResolution.aspx?w=360&h=615&hash=349a7bb85847e4aeee21d58328590cbb&a=b

received response original url: http://www.bing.com/hpmob?a=b&IG=574353C6151541CEB3A5830E565C0733&IID=SERP.5000 actual url: http://www.bing.com/hpmob?a=b&IG=574353C6151541CEB3A5830E565C0733&IID=SERP.5000

[jekh]

The problem with the example was that it did not include the / after the domain. I've updated the documentation to reflect that. I've also added new tests to demonstrate HTTP and HTTPS rewrite rules.

If you're still having issues with rewrite rules, we really need an example that will work against a public website, or a failing unit test.

[ghost]

First of all, thanks for you help. I made sure to include the / after the domain, but it still didn't work. Here's the updated example code and output. I also included a public website in the example:

public class Main {

    public static void main(String[] args) {

        BrowserMobProxyServer server = new BrowserMobProxyServer();
        server.start(8888);

        if (server.isStarted()) {
            System.out.println("server started on port " + server.getPort());
        }

        server.rewriteUrl("http://www\\.(yahoo|bing)\\.com\\?(\\w+)=(\\w+)", "http://www.google.com?originalDomain=$1&$2=$3");
        server.rewriteUrl("https://init\\.mycompany\\.com/sdk/v6\\.4\\.7/\\?(.*)", "https://platform-sandbox.mycompany.com/sdk/v6.4.7/?$1");
        server.rewriteUrl("https://www\\.yahoo\\.com/(.*)", "https://www.bing.com/$1");

        server.addResponseFilter(new ResponseFilter() {
            @Override
            public void filterResponse(HttpResponse httpResponse, HttpMessageContents httpMessageContents, HttpMessageInfo httpMessageInfo) {
                System.out.println();
                System.out.println("received response");
                System.out.println("original url: " + httpMessageInfo.getOriginalUrl());
                System.out.println("actual url: " + httpMessageInfo.getUrl());
                System.out.println();
            }
        });

        while (true) {
            // keep process running
        }

    }
}

server started on port 8888

received response original url: https://ua.mycompany.com/api/rest/v1.1/uniqueusers?applicationKey=3e266cb1&applicationUserId=yotam&sdkVersion=6.4.7 actual url: https://ua.mycompany.com/api/rest/v1.1/uniqueusers?applicationKey=3e266cb1&applicationUserId=yotam&sdkVersion=6.4.7

received response original url: https://init.mycompany.com/sdk/v6.4.7/?applicationKey=3e266cb1&applicationUserId=yotam&platform=iphone&advId=43B32E3C-AAD1-42DC-871A-781A78CBEEC5 actual url: https://init.mycompany.com/sdk/v6.4.7/?applicationKey=3e266cb1&applicationUserId=yotam&platform=iphone&advId=43B32E3C-AAD1-42DC-871A-781A78CBEEC5

And when I tried the yahoo rewrite rule:

received response original url: https://www.yahoo.com/ actual url: https://www.yahoo.com/

Would appreciate any advice :)

[jekh]

I see what's happening here: You're attempting to rewrite the domain name of an HTTPS request. Unfortunately, this is not currently supported, due to a limitation in LittleProxy's connection flow. There is a note about this in the rewriteUrl documentation. You may be able to use the host resolver to achieve the same functionality.