[Feature Request] Expose custom transport data streams

[HexaField]

WebRTC and potentially other transports support media streams (such as video and audio). It would be great to use libp2p directly to expose the relevant hooks/interfaces to enable this functionality, as opposed to opening a second connection and reimplementing a lot of what simple-peer / libp2p-webrtc-peer and libp2p-webrtc-star do.

The requirements I can identity thus far are only to add a way to (dynamically pass a stream)[https://github.com/ipfs-shipyard/simple-peer#dynamic-videovoice] via the peer.addStream function and add an event hook for the peer.on("stream") event.

[vasco-santos]

Thanks for reaching out @HexaField

When a transport connection is initially created we create a MultiaddrConn. This MultiaddrConn has the raw connection, which in the webrtc-star transport is the SimplePeer instance. The libp2p upgrader will upgrade it into a libp2p connection which does not include the raw connection (SimplePeer instance in webrtc-star). With this in mind, adding the raw connection into libp2p-connection is quite simple.

We need to take a few things into account though. Libp2p is a modular stack with multiple transports and consequently, transport agnostic. Exposing the raw connection will mean that the underlying API for the raw connection will be different according to the transport being used. This might create issues in the application layer as users will need to take into account these differences in the application layer. Other than that, exposing the raw connection might lead into unexpected behaviours in libp2p, such as if a raw connection is directly closed in the raw connection.

Overall, I think that this feature is valuable for several application use cases. Perhaps we can try to find a balance and expose raw connection functions per configuration.

A potential way of doing this would be to add to the interface-connection:

{
  transportFunctions: { }
}

In the transport layer, each transport would have a exposeTransportFunctions:

  exposeTransportFunctions: (rawConn) => { addStream: rawConn.addStream }

With a design like this, the user would be able to expose transport based functions to use in the application layer (if they exist). A simpler approach could be to add a transport string identifier in the interface-connection so that users could validate that a connection is using a given transport before trying to do something on the raw connection (directly exposed).

I think using the configuration approach would be better as it would be less prone to application layer surprises and less magical. Users might not understand why a given rawConn has the addStream and others not, while if explicitly configured they will understand that it is transport specific. However, it turns libp2p configuration even more complex. This would be an advanced user use case anyway, and if we have an example on how to configure this it should be straightforward.

What do you think @HexaField @jacobheun ?

[HexaField]

Thanks for the reply!

My thoughts looking through how this works is that a simpler fix might be to add a boolean option to the libp2p config exposeRawConn which is used in libp2p.upgrader._createConnection.

If exposeRawConn is set to true then the rawConn is simply assigned to the Connection. This would not cause any breaking changes, just an update to libp2p and interface-connection, and would put the burden of managing the proposed functionality on the application layer in an opt-in basis.

I do agree that adding transport tags to Connection is useful here.

[vasco-santos]

@HexaField sorry for not getting an answer for this yet.

I have discussed this synchronously with @jacobheun but he did not get to a decision yet. One problematic aspect of this is that we are multiplexing the connection with libp2p mplex. Once we expose the raw connection and people can use it, this will affect the multiplexer that will attempt to do its thing and affect exchanged messages.

We hope to get back to an answer for this soon

[julien51]

Hello! Interestingly enough I have exactly need :) Using MediaStream through the libp2p connection would be much better than using the libp2p connection to pass around some signaling to open a new SimplePeer :)

[HexaField]

Would a potential solution include exposing the raw connection internally, such that libp2p modules can access it and provide functionality that way? My thoughts are to have a libp2p-media-streams module which registers itself to the upgrader and is then able to receive raw connections. This way raw connections are not exposed to the application layer, but modules are still able to handle extra transport functionality.

[jacobheun]

I think the simplest solution here is to indeed just expose the raw connection. Right now that's hidden in the Connection interface, but if we expose the underlying MultiaddrConnection you can then access the raw connection and should be able to set up media streams that way. Raw connections may have pretty different behavior based on the type (webrtc/bluetooth/tcp/etc), but that would at least provide developers the opportunity to access that with out overcomplicating the interfaces.

The other option which we could do later anyway, that would be best done with the configuration proposal, would be to make it easier for developers to provide their own upgrader. This is a lot more work for application developers, but provides a lot of options for customizing what's exposed.

[maschad]

Closing as this is abstracted away from consumers intentionally, as a work around consumers can cast a WebRTC transport listener as a type with a peer Connection which will give you access to the WebRTC session.