search

multiformats / go-multiaddr-net

DEPRECATED: Please use the "net" subpackage in https://github.com/multiformats/go-multiaddr.
MIT License
34 stars 26 forks source link

Pluggable design #7

Closed jbenet closed 8 years ago

jbenet commented 8 years ago

(originally at https://github.com/ipfs/notes/issues/37#issuecomment-143518551)

It may make sense to rework how multiaddr-net works into something like this:

n := manet.NewNetwork()
n.Add(manet.IP4)
n.Add(manet.IP6)
n.Add(manet.TCP)
n.Add(manet.UDP)
n.Add(onion.Onion)

l, err := n.Listener("/ip4/1.2.3.4/tcp/1234")
c, err := n.Dial("/onion/<onionaddr>") 

// with shortcuts like:
n := manet.NewThinWaistNetwork()
n.Add(onion.Onion)

"network" is a bit odd name, but does give the idea that these networks are stacked and makes more sense why mulriaddrs work the way they do. "transport" is another possible name, but may not capture the full meaning of "network".

jbenet commented 8 years ago

cc @david415

jbenet commented 8 years ago

cc @whyrusleeping

cryptix commented 8 years ago

Yup, makes sense. I2P and other overlay networks need this as well. I can see how global configuration (like the control port for Tor) can be part of the value that gets added to the "network" but would this allow for passing in configuration of connection specific parameters?

david415 commented 8 years ago

@jbenet OK... I'd like to point out a couple of observations about your code samples in https://github.com/ipfs/notes/issues/37#issuecomment-143518551

onion service key material isn't included in your API design sketch... also tor can be configured with control port authentication... so the tor controller api needs to have even more secret information besides the onion service key private key... and possibly other information.

here's a concise working example of an onion echo server: https://github.com/david415/oniondialer/blob/master/examples/echo.go

I think it's worth mentioning that Twisted Python has a transport plugin system called Twisted "endpoints"; https://twistedmatrix.com/documents/current/core/howto/endpoints.html One notable difference is that the client versus server plugins are referenced with two different plugin name spaces, and are processed by clientFromString and serverFromString respectively.

After looking at Twisted endpoints, it's clear that the multiaddr usage is essentially equivalent to a "client endpoint descriptor string".... because it's what is announced to the world... and clients use it to connect... but it's not really enough information for a server to begin listening.

Let's look at what our plugin system might look like. We don't have a server endpoint equivalent... and I don't see a compelling reason to create such monstrosities as this server side multiaddr onion:

"/onion//onion-key-file/\/home\/human\/myOnionkey/tor-control-port/9151"

So avoiding the ugliness of a "server" multiaddr syntax... What do you think of this?

listenerPlugins.RemoveAllListenerPlugins()
listenerPlugins.Register(listenerType, plugin)

listener := MultiAddrToListener("/onion/<onion-addr>")

In order for the call to MultiAddrToListener to successfully create an onion listener it first needs to have the associated onion service key material. The calling party who registers the onion plugin must also "register" their onion address before using the MultiAddrToListener function.

jbenet commented 8 years ago

@david415 you're totally right

so it would really be more like:

n.Add(onion.NewTransport(onion.ListenerOptions{
    OnionKeyFile:     "echoOnionKey",
    OnionServicePort: 8080,
    LocalAddr:        "127.0.0.1:8080",
    ControlNetwork:   "tcp",
    ControlAddr:      "127.0.0.1:9051",
    AuthCookie:       "/var/lib/tor-cookie/cookie",
})

// dialing out
n.Dial("/onion/<addr>")

// and maybe
n.Listen("/onion/0") // where 0 uses the key in "Onion Key File" ?
whyrusleeping commented 8 years ago

@jbenet I dont like that approach very much. It means that my 'network' is limited to that one set of onion options, when I should be able to specify things in my dial/listen calls.

jbenet commented 8 years ago

@whyrusleeping do you have another API in mind?

this is the goal:

n.Dial(ma.Multiaddr)

but may not be achievable for 100% of the cases

whyrusleeping commented 8 years ago

Yeah.. I dont have a good api in mind. I've been mulling them over and cant really come up with anything I like.

whyrusleeping commented 8 years ago

for it to work right, all of that onion dialer information needs to be embedded into the multiaddr

jbenet commented 8 years ago

not sure if all, i can imagine putting the private key in the addr, but the proxy details still on the transport object (the proxy details are less relevant)... Unless the proxy cannot listen on more than one key.

another option is to set the listener opts on a map on the transport

t := onion.NewTransport()

// add transport to the network
n.Add(t)

// t keeps a map[addr]onion.ListenerOptions
t.AddProxy("<onion-addr>", onion.ListenerOptions{
    OnionKeyFile:     "echoOnionKey",
    OnionServicePort: 8080,
    LocalAddr:        "127.0.0.1:8080",
    ControlNetwork:   "tcp",
    ControlAddr:      "127.0.0.1:9051",
    AuthCookie:       "/var/lib/tor-cookie/cookie",
})

// and than can do
n.Listen("/onion/<onion-addr>")

and another option is to just forgo the nice plugging altogether :(

onion.Listen(onion.ListenerOptions{
    OnionKeyFile:     "echoOnionKey",
    OnionServicePort: 8080,
    LocalAddr:        "127.0.0.1:8080",
    ControlNetwork:   "tcp",
    ControlAddr:      "127.0.0.1:9051",
    AuthCookie:       "/var/lib/tor-cookie/cookie",
})
cryptix commented 8 years ago

I'm with @whyrusleeping wrt Configuration. Setting up Listeners beforehand might be doable like this, but configuring outgoing dials (TOR, for instance, allows to specify in which country an exit-node is located. In I2P you can tweak tunnel length and a lot of other parameters per connection, too) beforehand sounds unwieldy, because you simply might not have all the information until user interaction or that data comes from somewhere else.. Readding the transport with new a new config to the network feels ikcy.

for it to work right, all of that onion dialer information needs to be embedded into the multiaddr

I don't see how this is true though... @whyrusleeping did you mean, assuming we want it as simple as this, every Param would have to be in the multiaddr? Aggreed, I wouldn't want that either. Doesn't it mix where to call with how to get there? The latter is up to the calling party IMHO.

whyrusleeping commented 8 years ago

did you mean, assuming we want it as simple as this, every Param would have to be in the multiaddr?

Yeah, thats what i was going for.

After sleeping on it, I think that we might have to go with something like:

func Dial(addr ma.Multiaddr, opts ...interface{}) (Conn, error) {

this allows us to call it as: Dial(addr) for the simple connections, and it also allows us to call: Dial(addr, onion.Options{....})

The same goes for listener

jbenet commented 8 years ago

Agree with @cryptix and @whyrusleeping .

Though, i think @whyrusleeping's point about it being in the multiaddr (or at least that being one way of making it possible) makes sense. One of the goals of multiaddr is to allow single-string configuration of transports, so that we can pass those easily from the CLI, from config files, etc, that may not understand the specialized transports at all.

Maybe, what we're missing here, is that this is not just onion, but really, a specific way of accessing onion, namely through a local tcp proxy. I think this does make sense: 

# proxy onion over a local tcp service
/ip4/127.0.0.1/tcp/9051/onion/<onion-addr>

and this makes sense for both dials out and listens.

the tricky part with listens is other options, like AuthCookie and the private key.

/ip4/127.0.0.1/tcp/9051/onion-listen/<private-key>:<auth-cookie>/onion/<public-key>

is starting to get long and redundant.

@whyrusleeping i do like the Dial with interface{} options. And probably worth prototyping. Though again, i think being able to have single-string config is valuable.

@david415 i wonder what part of this listener config is really not meant to be part of the user configuration, but rather just a part of the "network stack implementation". what i mean by that is, the multiaddr string does not specify "for network tcp, make sure to use the net pkg's TCPDial function". This happens by virtue of the implementation of tcp in https://github.com/jbenet/go-multiaddr-net/ and it's a specific choice by the package author. People could chose to implement this with a C++ library instead, or with a userland TCP stack, etc. etc. Why is this relevant? because we may be able to separate the ListenerOptions above into three sections:

What about i2p? could we take a look at what connections there would look like, to get some inspiration? And DNS records?

cryptix commented 8 years ago

The length frightens me a little. Also encoding file paths in it.... (Could maybe be map[string]string to nick name them?). And I don't like the idea of having two representations of an address. (Think cat .ipfs/config and ipfs id output.

Though...

/ip4/127.0.0.1/tcp/9051/onion-listen/<private-key>:<auth-cookie>/onion/<public-key>

I just got this idea: There could be a multiaddr proxy that uses ssh intelligently to build forwarding's. Basically socks5 for the dial case but, you know.. Listen at mars from your code? I like the look of this! /ssh/mars/tcp/1337

what i mean by that is, the multiaddr string does not specify "for network tcp, make sure to use the net pkg's TCPDial function". This happens by virtue of the implementation of TCP.

For none-raw sockets, with TCP/UDP you are also at virtue of your systems kernel implementation of how it handles IP, no? I guess this is where implementation and layers rob up against?

What about i2p? could we take a look at what connections there would look like, to get some inspiration?

Well... Fire away :)

I2P is really toiled towards p2p. It uses a kademlia-ish dht for its netdb. You basically dial key hashes in the end, like with ipfs. You don't have numeric tcp/udp ports and just create new keys and endpoints on the go. (You can choose with stream and datagram though - there are lots of options to tweek those.. datagrams with no sender addr for instance.). One of the funkier features and setups which is also related here are encrypted leaseSets (endpoints), where you can't build a tunnel connection if you only have the key hash but also need a corresponding symmetric key to ask the netdb to build you a tunnel to it. Otherwise /ip4/127.0.0.1/tcp/9051/sam-stream-listen/ might actually work. (That's how the SAM bridge works. It can hand you new tcp ports locally that end up on the other side.) But do you want to open a previous key or a new one? How long should it be? etc.. Lots of options. Naming (name.i2p to $keyHash.b32.i2p) is done in each client node with an address book (like /etc/hosts maybe) which can be subscribed to or you ask a jump service which you trust for new ones. I'd really like to make multiaddr i2p aware, I think its a good fit.

jbenet commented 8 years ago

@cryptix leaseSets sound great actually. TRTTD.

on listen, wouldn't it be:

/ip4/127.0.0.1/tcp/9051/sam/0

?

what would other multiaddrs look like? like the ones you distribute out

str4d commented 8 years ago

Just adding my name to the participant list. I would definitely like to see I2P support (being one of the devs :stuck_out_tongue_winking_eye:) and am happy to answer questions.

A few points (from reading the previous two comments):

str4d commented 8 years ago

Having read a little more about multiaddr, I would imagine that distributable string multiaddrs for I2P would look like:

jbenet commented 8 years ago

thanks @str4d ! and thanks for all the work on I2P.

those multiaddrs LGTM.

whyrusleeping commented 8 years ago

The interface i've been working with for the utp codebase is like this:

type Dialer interface {
    Dial(Multiaddr) (Conn, error)
    Matches(Multiaddr) bool
}

So you can create anything that implements the dialer interface and construct a network with it like so:

n := NewNetworkThings()
n.AddDialer(myutpdialer)
n.AddDialer(mytcpreusedialer)
n.AddDialer(myfallbacknetdialer)

Then when you call dial on the nework, it will find a dialer that can dial the given address, and use it. The primary reason i went with this method is that for utp (and udt and friends) we need to be able to use the listener socket to dial out on, This becomes possible now:

list := NewUtpListener(address)
utpd := UtpDialerFromListener(list)
mynet.AddDialer(utpd)
david415 commented 8 years ago

What's the status? Did we settle on a multi-addr format for onion service addresses?

david415 commented 8 years ago

It's no big deal if we only use these multiaddrs for clients to connect to... but if we want onion services to work then we must specify the key somehow. Either we put the entire key in the multiaddr or a filename containing the key. Of course in the future when tor prop 224 is resolved the key will be an ed25519 key... so it'll be much shorter.

david415 commented 8 years ago

i think this ticket can be closed now.

whyrusleeping commented 8 years ago

@david415 agreed