Wrong URI received when using multiple endpoints

[hozuki]

Sample code:

proxyServer.TrustRootCertificate = true;
var ep1 = new TransparentProxyEndPoint(IPAddress.Any, 80, true);
var ep2 = new TransparentProxyEndPoint(IPAddress.Any, 443, true);
proxyServer.AddEndPoint(ep1);
proxyServer.AddEndPoint(ep2);
proxyServer.Start();

OnRequest (OnResponse is similar to this):

Console.WriteLine(e.WebSession.Request.RequestUri);

And then you will see the URIs being printed like:

https://www.abc.comwww.abc.com/ (without path and/or query)
https://www.def.comhttp//www.def.com/g/h?i=j (with path and query)

[honfika]

Could you please tell me how to reproduce this problem? I'd like to fix it. Probably this issue is duplicate of #190

What I tried:

• added your sample code
• added www.def.com to my hosts file
• opened www.def.com from chrome

it shows the url correctly (however it will be an endless loop)

[honfika]

Could you please check the value of the e.WebSession.Request.OriginalRequestUrl?

[honfika]

One more question:) You mentioned in the title of this issue, that the problem occurs with multiple endpoints. Really? So if you keep only 1 endpoint (any of them), it works?

[justcoding121]

An easy way to test transparent endpoint will be to run the proxy in local machine. And then do the dns change in a second machine pointing the site IP to local machine. Second machine can be a laptop or smart phone. If the site is secure we would need to install the root certificate in second machine. I've tested that way in past without the endless loop.

[honfika]

Thanks. Now the transparent proxy is working, but I can't reproduce the problem. Btw: The last parameter in the first TransparentProxyEndpoint should be false.

[honfika]

@hozuki May I close this issue?

[hozuki]

Sorry I forgot this issue. Here's a more detailed description.

I was reverse engineering an app on Android. This application app ignores system proxy settings so I have to use a local DNS forwarder like this one. I run the DNS forwarder on my PC, listening to port 53, and resolve interested requests to localhost.

For instance, Interested domains: www.abc.com:80(1.2.3.4:80) -[DNS forwarder]-> www.abc.com:80(127.0.0.1:80) Not interested domains: www.def.com:80(5.6.7.8:80) -[DNS fowarder]-> www.def.com:80(5.6.7.8:80)

Since the requests are redirected to localhost, they are then able to be handled by proxies listening to local ports, like Titanium.

The set-up is listed below. A router is required.

1. Connect the PC and the Android device to the router.
2. Allocate a static local IP x.y.z.w for the PC on the router.
3. On the Android device, set the DNS address for current network to x.y.z.w.
4. Start DNS forwarder.
5. Start Titanium instance.
6. Start the app.
7. Watch for the console output.

The app only uses HTTPS connections, so I use Titanium to listen to localhost, both port 80 and 443. The code is shown at the end of this comment.

But unfortuantely, I didn't leave any log file... I copied console output, put forward the issue (as you can see I hid the real domain names) and discarded the log. Hmm, unwise acts.

After that I dropped Titanium and tired tried FiddlerCore. It wasn't my first choice because FiddlerCore can't listen to multiple ports. I managed to run it using multiple AppDomains, but I don't have time to solve serialization problems. In the end I stopped the reverse engineering work.

My tablet was broken about a month ago so I can't run the app now, so I can't reproduce this problem. Maybe others can make a simple app, using HTTPS connections, and go through the set-up steps. But really sorry I don't have time for this now.

I also wrote a function to get corrected URIs. It was deleted before testing FiddlerCore. :( It is basically string manipulations under different cases.

And for the question, if you find this issue annoying (disturbs releasing?) sure you can close it. This is only a rare use case, because most people don't use Titanium like what I did. :) If I encounter it in the future, I'd love to send a PR.

---

Proxy code:

public sealed class TestProxy {

    public void Start() {
        _proxyServer = new ProxyServer();

        _proxyServer.TrustRootCertificate = true;

        _proxyServer.CertificateEngine = CertificateEngine.BouncyCastle;

        _proxyServer.BeforeRequest += OnRequest;
        _proxyServer.BeforeResponse += OnResponse;
        _proxyServer.ServerCertificateValidationCallback += OnCertificateValidation;
        _proxyServer.ClientCertificateSelectionCallback += OnCertificateSelection;

        var startupPath = Environment.CurrentDirectory;
        // It is a self-signed certificate, root CA trusted on local machine.
        var certPath = Path.Combine(startupPath, "trusted.cer");
        var cert = new X509Certificate2(certPath);

        var ep1 = new TransparentProxyEndPoint(IPAddress.Any, 80, true) {
//                GenericCertificate = cert
        };
        var ep2 = new TransparentProxyEndPoint(IPAddress.Any, 443, true) {
//                GenericCertificate = cert
        };

        _proxyServer.AddEndPoint(ep1);
        _proxyServer.AddEndPoint(ep2);

        _proxyServer.Enable100ContinueBehaviour = true;

        foreach (var endPoint in _proxyServer.ProxyEndPoints) {
            Console.WriteLine("Listening on '{0}' endpoint at IP {1} and port: {2} ", endPoint.GetType().Name, endPoint.IpAddress, endPoint.Port);
        }

        _proxyServer.Start();
    }

    public void Stop() {
        _proxyServer.BeforeRequest -= OnRequest;
        _proxyServer.BeforeResponse -= OnResponse;
        _proxyServer.ServerCertificateValidationCallback -= OnCertificateValidation;
        _proxyServer.ClientCertificateSelectionCallback -= OnCertificateSelection;

        _proxyServer.Stop();
    }

    public async Task OnRequest(object sender, SessionEventArgs e) {
        Console.WriteLine("> Request: {0} {1}", e.WebSession.Request.Method, e.WebSession.Request.Url);
    }

    //Modify response
    private async Task OnResponse(object sender, SessionEventArgs e) {
        Console.WriteLine("> Response: {0}", e.WebSession.Request.Url);

        var method = e.WebSession.Request.Method;
        if (method.ToLowerInvariant() == "head") {
            Console.WriteLine("head!");
        }
    }

    private Task OnCertificateValidation(object sender, CertificateValidationEventArgs e) {
        if (e.SslPolicyErrors == System.Net.Security.SslPolicyErrors.None) {
            e.IsValid = true;
        }

        return Task.FromResult(0);
    }

    private Task OnCertificateSelection(object sender, CertificateSelectionEventArgs e) {
        return Task.FromResult(0);
    }

    private ProxyServer _proxyServer;

}

[honfika]

Could you please tell me how to setup this FiddlerDnsProxy? When I start it, there is only a combobox with 7 IP addresses and a start/stop button.

[hozuki]

Sure. After launching FiddlerDnsProxy, select the IP address corresponding to what your router allocates, then press "start/stop". When you are done with DNS forwarding, press the button again to exit.

In my case, the router allocates addresses beginning from 192.168.1.101. My PC has the local address 192.168.1.102 on my wireless network adapter (192.168.1.101 is used by RPi). So this is the address allocated by the router, and other IPs are not visible to the tablet. Then select "192.168.1.102" for the DNS forwarder and it will work.

[honfika]

Thats ok, but where shoudl I configure this?

For instance, Interested domains: www.abc.com:80(1.2.3.4:80) -[DNS forwarder]-> www.abc.com:80(127.0.0.1:80) Not interested domains: www.def.com:80(5.6.7.8:80) -[DNS fowarder]-> www.def.com:80(5.6.7.8:80)

[hozuki]

You don't need to. I used domain filters in my program, but it is not related to this issue. You can observe the text printed out in the console (without filters the quantity may be large).

[hozuki]

Hi I'm here again. I have time to redo the tests.

This time I printed OriginalUrl and they are fine:

config.uca.cloud.unity3d.com:443
api.uca.cloud.unity3d.com:443
data.flurry.com:443

And some example printed Url (from the same startup process):

https://config.uca.cloud.unity3d.comconfig.uca.cloud.unity3d.com:443
https://api.uca.cloud.unity3d.comapi.uca.cloud.unity3d.com:443
https://data.flurry.comdata.flurry.com:443

By the way, I simply modified HOSTS this time and I didn't use a DNS forwarder.

Thanks. @honfika

Source code (using NuGet package version 3.0.213):

    internal sealed class Program {

        private static void Main(string[] args) {
            Console.TreatControlCAsInput = true;

            Console.CancelKeyPress += ConsoleOnCancelKeyPress;

            var proxy = new ProxyServer();

            proxy.CertificateEngine = CertificateEngine.BouncyCastle;
            proxy.RootCertificate = new X509Certificate2("FiddlerRoot.cer");
            proxy.CertificateManager.TrustRootCertificate();

            proxy.AddEndPoint(new TransparentProxyEndPoint(IPAddress.Any, 80, true));
            proxy.AddEndPoint(new TransparentProxyEndPoint(IPAddress.Any, 443, true));

            proxy.BeforeRequest += ProxyOnBeforeRequest;

            proxy.Start();

            while (true) {
                    var key = ReadKey(true);
                    if ((key.Key == ConsoleKey.C && key.Modifiers == ConsoleModifiers.Control) || key.Key == ConsoleKey.Enter) {
                        break;
                    }
            }

            proxy.Stop();

            Console.CancelKeyPress -= ConsoleOnCancelKeyPress;

            proxy.CertificateManager.RemoveTrustedRootCertificates();
        }

        private static Task ProxyOnBeforeRequest(object sender, SessionEventArgs e) {
            Console.WriteLine(e.WebSession.Request.Url);
            Debug.Print(e.WebSession.Request.Url);

            return Task.FromResult(0);
        }

        private static void ConsoleOnCancelKeyPress(object sender, ConsoleCancelEventArgs e) {
            if (e.SpecialKey == ConsoleSpecialKey.ControlC) {
                e.Cancel = true;
                _shouldStop = true;
            }
        }

        private static bool _shouldStop;

    }

[honfika]

Please try the latest beta package: https://www.nuget.org/packages/Titanium.Web.Proxy/3.0.256-beta

[hozuki]

It's still failing. When using version 3.0.256-beta, ProxyOnBeforeRequest is not even called.

[honfika]

Oh...

Could you please print the exception?

proxy.ExceptionFunc = exception => Console.WriteLine(exception.ToString());

[hozuki]

Here it goes. There are two types of exceptions.

The first one is thrown before the first communication (messages are translated):

Titanium.Web.Proxy.Exceptions.ProxyHttpException: Error occured whilst handling session request ---> System.UriFormatException: 无效的 URI: 指定的端口无效。 [Invalid URI: the port specified is invalid.]
   在 System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
   在 System.Uri..ctor(String uriString)
   在 Titanium.Web.Proxy.ProxyServer.<HandleHttpSessionRequest>d__160.MoveNext()
   --- 内部异常堆栈跟踪的结尾 --- [end of internal exception stacktrace]

And the second one is thrown during handling communications. One for each session. Here is an example:

https://config.uca.cloud.unity3d.comconfig.uca.cloud.unity3d.com:443
Titanium.Web.Proxy.Exceptions.ProxyHttpException: Error occured whilst handling session request ---> System.Net.Sockets.SocketException: 不知道这样的主机。 [Host is unknown.]
   在 System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   在 System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
   在 System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- 引发异常的上一位置中堆栈跟踪的末尾 --- [end of the stacktrace of the previous location that caused the exception]
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Titanium.Web.Proxy.Network.Tcp.TcpConnectionFactory.<CreateClient>d__0.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 --- [end of the stacktrace of the previous location that caused the exception]
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   在 Titanium.Web.Proxy.ProxyServer.<GetServerConnection>d__162.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 --- [end of the stacktrace of the previous location that caused the exception]
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   在 Titanium.Web.Proxy.ProxyServer.<HandleHttpSessionRequest>d__160.MoveNext()
   --- 内部异常堆栈跟踪的结尾 --- [end of internal exception stacktrace]

[honfika]

Could you please print the following properties with the previous (latest stable) package?

            Console.WriteLine("Request.Url: " + e.WebSession.Request.Url);
            Console.WriteLine("Request.OriginalUrl: " + e.WebSession.Request.OriginalUrl);
            Console.WriteLine("Request.Host: " + e.WebSession.Request.Host);
            Console.WriteLine("Request.IsHttps: " + e.WebSession.Request.IsHttps);
            Console.WriteLine("ConnectRequest.Url: " + e.WebSession.ConnectRequest?.Url);
            Console.WriteLine("ConnectRequest.OriginalUrl: " + e.WebSession.ConnectRequest?.OriginalUrl);
            Console.WriteLine("ConnectRequest.Host: " + e.WebSession.ConnectRequest?.Host);
            Console.WriteLine("ConnectRequest.IsHttps: " + e.WebSession.ConnectRequest?.IsHttps);
            foreach (var endPoint in ((ProxyServer)sender).ProxyEndPoints)
            {
                if (endPoint is TransparentProxyEndPoint)
                {
                    var t = (TransparentProxyEndPoint)endPoint;
                    Console.WriteLine("Endpoint: " + t.Port + " " + t.EnableSsl + " " + t.GenericCertificateName);
                }
            }

Thanks

[hozuki]

Here are two sample outputs:

Request.Url: https://android.clients.google.comandroid.clients.google.com:443
Request.OriginalUrl: android.clients.google.com:443
Request.Host: android.clients.google.comandroid.clients.google.com
Request.IsHttps: True
ConnectRequest.Url:
ConnectRequest.OriginalUrl:
ConnectRequest.Host:
ConnectRequest.IsHttps:
Endpoint: 80 True localhost
Endpoint: 443 True localhost

Request.Url: https://api.uca.cloud.unity3d.comapi.uca.cloud.unity3d.com:443
Request.OriginalUrl: api.uca.cloud.unity3d.com:443
Request.Host: api.uca.cloud.unity3d.comapi.uca.cloud.unity3d.com
Request.IsHttps: True
ConnectRequest.Url:
ConnectRequest.OriginalUrl:
ConnectRequest.Host:
ConnectRequest.IsHttps:
Endpoint: 80 True localhost
Endpoint: 443 True localhost

[honfika]

It is very strange, it seems that the host header already contains the hostname twice. Sorry for bothering you but could you please try the latest beta? (3.0.261-beta or newer)

3 possible results:

• throws exception, it will include the wrong uri.

• does not throw exception, then please print the header, too (with the previously printed information): Console.WriteLine("----" + Environment.NewLine + e.WebSession.Request.HeaderText + Environment.NewLine + "----");

• works, then OK:)

[honfika]

Is this issue fixed?

[hozuki]

I think so.

In version 3.0.261-beta there was only completely silence, no errors, and no request outputs. Version 3.0.370-beta works fine, all the properties are correct. The only problem is authentication always fails (Could'nt authenticate client 'xxx.com' with fake certificate. , traced to Titanium.Web.Proxy.ProxyServer.<HandleClient>d__134.MoveNext()). I've trusted the certificate on the device (Android 4.x), so I have to assume the error happens in Titanium. Anyway, that's another story.

Thanks for tracking the issue.

[honfika]

I think the authencitaction issue was fixed now in issue #339

May I close this issue?

[hozuki]

Sure.