Host.SetStreamHandler before/after Host.Connect has different behavior

[felixSchl]

Handlers registered via Host.SetStreamHandler behave differently depending on if they were registered before or after connecting a certain peer. If Host.SetStreamHandler is called before Host.Connect, then it won't be invoked immediately when the remote peer calls Host.NewStream. It will only be invoked if the remote peer then also makes a call to Stream.Read or Stream.Write (on the stream returned from Host.NewStream). If, however, Host.Connect was done first, followed by Host.SetStreamHandler, the handler is invoked immediately after the remote peer calls Host.NewStream.

Compare below code for a demonstration:

In this example, there is no output:

import (
    "context"
    "fmt"

    "github.com/libp2p/go-libp2p"
    "github.com/libp2p/go-libp2p-core/network"
    "github.com/libp2p/go-libp2p-core/peer"
)

func main() {
    host1, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2AddrInfo := peer.AddrInfo{ID: host2.ID(), Addrs: host2.Addrs()}

    host1.SetStreamHandler("foobaz", func(stream network.Stream) {
        rid := stream.Conn().RemotePeer()
        fmt.Printf("host1: got stream (foobaz) %s\n", rid)
    })

    host1.Connect(context.Background(), host2AddrInfo)

    host2.NewStream(context.Background(), host1.ID(), "foobaz")

    select {
    case <-context.Background().Done():
    }
}

Whereas in this example, the output is:

host1: got stream (foobaz) QmZ76ZZiws8JHFWmNymcc2icjEE7gVZUXWFGPHjaK649g1

package main

import (
    "context"
    "fmt"

    "github.com/libp2p/go-libp2p"
    "github.com/libp2p/go-libp2p-core/network"
    "github.com/libp2p/go-libp2p-core/peer"
)

func main() {
    host1, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2AddrInfo := peer.AddrInfo{ID: host2.ID(), Addrs: host2.Addrs()}

    host1.Connect(context.Background(), host2AddrInfo)

    host1.SetStreamHandler("foobaz", func(stream network.Stream) {
        rid := stream.Conn().RemotePeer()
        fmt.Printf("host1: got stream (foobaz) %s\n", rid)
    })

    host2.NewStream(context.Background(), host1.ID(), "foobaz")

    select {
    case <-context.Background().Done():
    }
}

Most curiously, reading (or writing) to the returned stream also triggers the handler. It makes sense to me that Stream.Write would cause this, but I find it odd that Stream.Read also triggers the handler:

package main

import (
    "context"
    "fmt"

    "github.com/libp2p/go-libp2p"
    "github.com/libp2p/go-libp2p-core/network"
    "github.com/libp2p/go-libp2p-core/peer"
)

func main() {
    host1, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2, err := libp2p.New(context.Background())
    if err != nil {
        panic(err)
    }

    host2AddrInfo := peer.AddrInfo{ID: host2.ID(), Addrs: host2.Addrs()}

    host1.SetStreamHandler("foobaz", func(stream network.Stream) {
        rid := stream.Conn().RemotePeer()
        fmt.Printf("host1: got stream (foobaz) %s\n", rid)
    })

    host1.Connect(context.Background(), host2AddrInfo)

    stream, err := host2.NewStream(context.Background(), host1.ID(), "foobaz")
    if err != nil {
        panic(err)
    }
    stream.Read(make([]byte, 0))

    select {
    case <-context.Background().Done():
    }
}

Version Information

github.com/libp2p/go-libp2p-examples
cloud.google.com/go v0.26.0
github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7
github.com/BurntSushi/toml v0.3.1
github.com/Kubuxu/go-os-helper v0.0.1
github.com/aead/siphash v1.0.1
github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803
github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd
github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd
github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723
github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
github.com/btcsuite/winsvc v1.0.0
github.com/client9/misspell v0.3.4
github.com/coreos/go-semver v0.3.0
github.com/davecgh/go-spew v1.1.0
github.com/dgraph-io/badger v1.5.5-0.20190226225317-8115aed38f8f
github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f
github.com/dustin/go-humanize v1.0.0
github.com/fsnotify/fsnotify v1.4.7
github.com/go-check/check v0.0.0-20180628173108-788fd7840127
github.com/gogo/protobuf v1.2.1
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/mock v1.1.1
github.com/golang/protobuf v1.3.0
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/google/go-cmp v0.2.0
github.com/google/uuid v1.1.1
github.com/gorilla/websocket v1.4.0
github.com/gxed/hashland/keccakpg v0.0.1
github.com/gxed/hashland/murmur3 v0.0.1
github.com/hashicorp/golang-lru v0.5.1
github.com/hpcloud/tail v1.0.0
github.com/huin/goupnp v1.0.0
github.com/huin/goutil v0.0.0-20170803182201-1ca381bf3150
github.com/ipfs/go-cid v0.0.2
github.com/ipfs/go-datastore v0.0.5
github.com/ipfs/go-detect-race v0.0.1
github.com/ipfs/go-ds-badger v0.0.2
github.com/ipfs/go-ds-leveldb v0.0.1
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8
github.com/ipfs/go-ipfs-util v0.0.1
github.com/ipfs/go-log v0.0.1
github.com/ipfs/go-todocounter v0.0.1
github.com/jackpal/gateway v1.0.5
github.com/jackpal/go-nat-pmp v1.0.1
github.com/jbenet/go-cienv v0.1.0
github.com/jbenet/go-temp-err-catcher v0.0.0-20150120210811-aac704a3f4f2
github.com/jbenet/goprocess v0.1.3
github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89
github.com/jrick/logrotate v1.0.0
github.com/kisielk/errcheck v1.1.0
github.com/kisielk/gotool v1.0.0
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23
github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.1.0
github.com/libp2p/go-addr-util v0.0.1
github.com/libp2p/go-buffer-pool v0.0.2
github.com/libp2p/go-conn-security-multistream v0.1.0
github.com/libp2p/go-flow-metrics v0.0.1
github.com/libp2p/go-libp2p v0.1.0
github.com/libp2p/go-libp2p-autonat v0.1.0
github.com/libp2p/go-libp2p-blankhost v0.1.1
github.com/libp2p/go-libp2p-circuit v0.1.0
github.com/libp2p/go-libp2p-core v0.0.1
github.com/libp2p/go-libp2p-crypto v0.1.0
github.com/libp2p/go-libp2p-discovery v0.1.0
github.com/libp2p/go-libp2p-kad-dht v0.1.0
github.com/libp2p/go-libp2p-kbucket v0.2.0
github.com/libp2p/go-libp2p-loggables v0.1.0
github.com/libp2p/go-libp2p-mplex v0.2.1
github.com/libp2p/go-libp2p-nat v0.0.4
github.com/libp2p/go-libp2p-netutil v0.1.0
github.com/libp2p/go-libp2p-peer v0.2.0
github.com/libp2p/go-libp2p-peerstore v0.1.0
github.com/libp2p/go-libp2p-pubsub v0.1.0
github.com/libp2p/go-libp2p-record v0.1.0
github.com/libp2p/go-libp2p-routing v0.1.0
github.com/libp2p/go-libp2p-secio v0.1.0
github.com/libp2p/go-libp2p-swarm v0.1.0
github.com/libp2p/go-libp2p-testing v0.0.3
github.com/libp2p/go-libp2p-transport-upgrader v0.1.1
github.com/libp2p/go-libp2p-yamux v0.2.0
github.com/libp2p/go-maddr-filter v0.0.4
github.com/libp2p/go-mplex v0.1.0
github.com/libp2p/go-msgio v0.0.2
github.com/libp2p/go-nat v0.0.3
github.com/libp2p/go-reuseport v0.0.1
github.com/libp2p/go-reuseport-transport v0.0.2
github.com/libp2p/go-stream-muxer v0.0.1
github.com/libp2p/go-stream-muxer-multistream v0.2.0
github.com/libp2p/go-tcp-transport v0.1.0
github.com/libp2p/go-ws-transport v0.1.0
github.com/libp2p/go-yamux v1.2.2
github.com/mattn/go-colorable v0.1.1
github.com/mattn/go-isatty v0.0.5
github.com/miekg/dns v1.1.12
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1
github.com/minio/sha256-simd v0.1.0
github.com/mr-tron/base58 v1.1.2
github.com/multiformats/go-base32 v0.0.3
github.com/multiformats/go-multiaddr v0.0.4
github.com/multiformats/go-multiaddr-dns v0.0.2
github.com/multiformats/go-multiaddr-fmt v0.0.1
github.com/multiformats/go-multiaddr-net v0.0.1
github.com/multiformats/go-multibase v0.0.1
github.com/multiformats/go-multihash v0.0.5
github.com/multiformats/go-multistream v0.1.0
github.com/onsi/ginkgo v1.8.0
github.com/onsi/gomega v1.5.0
github.com/opentracing/opentracing-go v1.0.2
github.com/pkg/errors v0.8.1
github.com/pmezard/go-difflib v1.0.0
github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a
github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572
github.com/spaolacci/murmur3 v1.1.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0
github.com/syndtr/goleveldb v1.0.0
github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1
github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc
github.com/whyrusleeping/go-notifier v0.0.0-20170827234753-097c5d47330f
github.com/whyrusleeping/mafmt v1.2.8
github.com/whyrusleeping/mdns v0.0.0-20180901202407-ef14215e6b30
github.com/whyrusleeping/multiaddr-filter v0.0.0-20160516205228-e903e4adabd7
github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee
go.opencensus.io v0.21.0
golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f
golang.org/x/exp v0.0.0-20190121172915-509febef88a4
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6
golang.org/x/sys v0.0.0-20190412213103-97732733099d
golang.org/x/text v0.3.0
golang.org/x/tools v0.0.0-20190226205152-f727befe758c
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522
google.golang.org/appengine v1.4.0
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19
google.golang.org/grpc v1.19.0
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
gopkg.in/yaml.v2 v2.2.1
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099

[Stebalien]

Case 1: Register before connect.

In this case, both peers will send the known supported protocols on connect using the identify protocol. When that happens, we use something called "lazy" stream negotiation.

1. When we call NewStream, we find a protocol (out of the list specified by the user) that we know the peer speaks.
2. We then create a special stream that negotiates this protocol lazily. We can do this because we know the negotiation will succeed.
3. Finally, when the user actually tries to use the stream (call read/write), we negotiate the protocol.

We do this because:

• Not waiting for the protocol negotiation saves us a round trip.
• If we're going to write immediately after creating the stream, negotiating lazily allows us to send one big packet instead of two small packets (i.e., we don't have to send the protocol in one packet, then the message).

Case 2: Register after connect.

In this case, the identify handshake didn't tell our peer about this protocol. We don't know if they speak the protocol so we negotiate up-front.

---

Note: we negotiate on both read and write because opening a stream and then waiting for a response without writing anything is a valid protocol (that's how the identify protocol works, actually).

[felixSchl]

Thank you very much for the explanation. That sounds very sensible.