Implement a bare bone proxy

[Sp3ctr3]

Create a forwarding proxy capable of content manipulation. We'll need to manipulate the HTTP data if we are to deobfuscate the data before passing on. Currently we can manipulate data but no forwarding is done.

[anuvrat]

Hi Yashin. I would like to help out with this task. Can you please let me know how to get started?

[jpaulynice]

Hey Yashin,

I can help on this as well.

Jay

[Sp3ctr3]

Excellent! I'll be uploading the code within a few hours. We'll be using the twisted framework for the proxy code. Once the code is up we can get started!

[admwx7]

For full compliance and backwards compatibility with the clients, we may want to do a version check. So when the client makes a request it'll send it's version number (or we can key a general API version here) that the server can then key off of to choose what format the client is expecting the data to be returned in. So if at some point in the future we do a major overhaul of how the steg is done (or change encryption types) we can keep support for the previous encryption type, if the client sends an API version that used the old type we can revert to the legacy use for that request so the client doesn't break. We would want the clients to all be up-to-date, but we also need to provide some legacy support to an extent. Maybe 1-2 API version back for a full deprecation process to allow third-party developers a chance to get their software caught up before the functionality is completely removed.

[Sp3ctr3]

I've added the code here 20a2f7ff8312d1693414bafe86b4838c7b3372ef . It's very basic right now. But you can use it as a proxy server and you can modify the contents too.

[Wingie]

hey i'm trying out the code in that commit and it's giving me this issue when i try to curl it

this is the traceback.

    Traceback (most recent call last):
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/python/log.py", line 88, in callWithLogger
        return callWithContext({"system": lp}, func, *args, **kw)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/python/log.py", line 73, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
    --- <exception caught here> ---
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 151, in _doReadOrWrite
        why = getattr(selectable, method)()
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/internet/tcp.py", line 215, in doRead
        return self._dataReceived(data)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/internet/tcp.py", line 221, in _dataReceived
        rval = self.protocol.dataReceived(data)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
        why = self.lineReceived(line)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/web/http.py", line 1619, in lineReceived
        self.allContentReceived()
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/web/http.py", line 1694, in allContentReceived
        req.requestReceived(command, path, version)
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/web/http.py", line 790, in requestReceived
        self.process()
      File "/Users/wingstonsharon/venvs/prem/lib/python2.7/site-packages/twisted/web/proxy.py", line 147, in process
        port = self.ports[protocol]
    exceptions.KeyError: ''

and my curl log

 $ curl 127.0.0.1:8080 -v                                                                                                                                               [0:44:34]
* About to connect() to 127.0.0.1 port 8080 (#0)
*   Trying 127.0.0.1...
* Adding handle: conn: 0x7fac9b804400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fac9b804400) send_pipe: 1, recv_pipe: 0
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: 127.0.0.1:8080
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server

[sai-manoj-kumar]

I would like to work on this issue. Currently learning about Twisted as I am new to it. Will be able to work on this in few days.

[boxtown]

There are a host of issues with the proxy client. I have to get to class now but I'll either fix them myself later tonight or post on the issues. A starting place might be to actually write constructors for ProxyFactor, Proxy, etc. and use 'self.' on the variables being defined. Right now, in ProxyFactory when you say 'protocol = Proxy', that's not doing anything. You need to have it in the constructor with a 'self.protocol = Proxy()' I believe

[xykivo]

Hi Yashin,

I would like to help with this issue.

How do I get started? Is there some doc/tutorial on setup and testing?

Thanks Dror

[sai-manoj-kumar]

@xykivo Read http://krondo.com/?page_id=1327, for introduction to Twisted. For running the server, just run python server.py

[Sp3ctr3]

@Wingie Are you trying it against pure text websites? I'm not having any problems here with any type of websites. Try a normal browser too. @boxtown Agreed. If you can add the improvements you mentioned, that would be great! @xykivo The resource @ysaimanojkumar is great. There's a similar one at: https://twistedmatrix.com/documents/12.2.0/core/howto/tutorial/index.html

[boxtown]

Figured out the issue @Wingie is experiencing. Since right now, when you run the server, it is only running as a proxy, running curl http://127.0.0.1:8080 is trying to access a webserver that doesn't exist (trying to get the proxy to redirect to itself). A proxy only acts as a go and right now there's nothing to redirect the proxy to. Ergo, Keyerror and the response from curl. The proxy code is fine.