Add twisted endpoint parser plugin

[david415]

Hi! Please review the diff and let me know if there is anything you'd like me to change. The unit tests all pass now!... so I hope you merge it. =-)

Cheers,

David

[meejah]

Great, thanks! I'll have time later on tonight to take a longer look. I'd run pep8 on it ("make pep8") and the localPort etc variables should be local_port etc style to match the rest of txtorcon stuff.

[david415]

ah yeah it is definitely not pep8 compliant... i'll fix that right now.

[david415]

there... pep8!

[meejah]

Awesome, thanks!

I'm playing with a couple things and will push a branch with your stuff and a few things on top "soon" (next day or so).

I was thinking: should we just take out "publicPort=" and make it positional? This would make it more like the tcp strings (e.g. "tcp:80" valid, so would "onion:80") and shorter.

[david415]

Oh yeah I forgot to rename the variables to match style. I can do that soon. Yes... I agree with taking out "publicPort" and making it positional. Much better!

On a related note... endpoint-descriptor-philosophy question: How many tor options should we make available to the endpoint parser? I like the idea that all valid txtorcon endpoint descriptors will work on anyone's system. But maybe this is not a good thing to enforce... because it certainly does seem like an interesting idea to use the system tor's control port to create new hidden services... and in this case it would be smart to allow the user to specify the control port... and this of course would make that endpoint descriptor string not as portable... but maybe that is OK.

[meejah]

I'm leaning towards something wishy-washy like "not too many options exposed". Or "just enough to get the job done". The theory is that if you have a special use-case, you can do the "long way" and make your own TorConfig etc.etc. (or launch tor yourself). Of course, not nice and convenient to endpoint users.

Technically, there's nothing to prevent exposing ALL Tor options...but that seems silly ;)

I expect the main use-case for this server-side endpoint thing is for systems that want to allow themselves to be exposed via a hidden-service...so the end-user of these systems will need a few Tor options exposed, but not all. Maybe it would make more sense to have some way for the "txtorcon user code" (i.e. not the ultimate end-user) to tie in somehow and specify any additional options they need.

Another way to look at it: for anyone needing something special in Tor, they can also launch tor themselves with their favourite options and then do "onion:80:controlPort=1234" to connect to their specially-configured tor. Since Tor has somewhere around 9 billion options, it's tough to predict what special-sauce different use-cases might want.

[meejah]

p.s. I pushed a few things to https://github.com/meejah/txtorcon/tree/david415_endpoint_parser_plugin

What do you think about the "wrapper endpoint" structure of TorLaunchOrConnectEndpoint?

[david415]

Excellent idea! My friend Leif Ryge and I were discussing possible designs for this tor hidden service endpoint... and what we realized is that the Tahoe-LAFS python process will have many endpoints it will listen on and many endpoints it will connect to. In this case we really want these features:

• Each Twisted process (in this case Tahoe-LAFS) would use 1 (one) instance of Tor for all it's client and server communication. We don't want txsocksx to have to be dependent on txtorcon... however the txsocksx client tor endpoint could optionally read a txtorcon module variable specifying the tor socks port. If that fails then the tor client endpoint would try to find the system tor socks port.
• The first time a python process calls the listen method on a hidden service endpoint, the endpoint object will either use the system tor control port OR it will launch it's own tor process. (great... because we can reuse your new code for this...) Which ever it chooses it will then record this information in txtorcon module variables... so that if the same python process later creates more hidden service endpoints they will all use the tor process that was chosen by the first listen method call. This last part about tracking a single tor proc is not implemented yet... but Leif seemed close to finishing a patch for that. Once this is done then I will make the necessary changes to the txsocksx tor client endpoint.

With this design we would not really need to specify the control port. We should probably still have it as an option.. however it isn't needed because if we try a few ports where the system control port might be and we cannot connect to it then in that case we should start tor with it's control port set to an unused port.

By the way... a bit about design patterns I'm using with these twisted endpoints: In Foolscap and Tahoe-LAFS we cannot special case any usage of endpoint objects. Foolscap currently treats all endpoint objects in the exact same way... and in this sense is completely decoupled from the underlying wire protocol; Foolscap is totally decoupled from Tor. Tahoe-LAFS will have interesting Tor-friendly features like converting non-tor endpoint descriptor strings into tor endpoint descriptor strings... when running in "Tor-only" mode for instance. But Tahoe-LAFS in this case still should be totally decoupled from launching Tor AND decoupled from the twisted endpoints api; that is the job of Foolscap. However... Tahoe-LAFS only knows about endpoint descriptor strings... and this makes it's job very simple. Each component in the system has it's own simple responsibilities... this is why I wrote the parsers and why I cannot use extra additional public methods calling into an endpoint object.

[meejah]

If you're lurking here, there is subsequent discussion on tahoe-dev: https://tahoe-lafs.org/pipermail/tahoe-dev/2014-May/009036.html

[meejah]

Re-worked and merged as #74 -- closing.