Add documentation

[marklagendijk]

Is your feature request related to a problem? Please describe. It is not clear how the library can be used. Apparently the library can be used to display RTSP streams in the browser, but there is no clear explanation on what is needed to make this work. E.g:

1. Is it enough to add the library to the browser and point it to a RTSP stream? Or do you need to run some Node.js server as well. If so, how does that work?
2. What does this part of examples/browser/camera/simple-player.js do exactly? Why is there both a ws and rtsp url? What are the paths based on?

   const pipeline = new Pipeline({
     ws: { uri: `ws://${host}/rtsp-over-websocket` },
     rtsp: { uri: `rtsp://${host}/axis-media/media.amp?videocodec=${encoding}` },
     mediaElement,
   })

   Describe the solution you'd like Documentation on the most common scenarios the library will be used for.

[seeitlive]

Hi, 1) You don't need NodeJS. You can do it any sort of web server or equivalent and browser supports Media Source Extension or similar with client side built in h.264 decoding and jpeg decoding. There seems only problem unfortunately I couldn't find any parts of implementation about h.265/hevc. 2) ws is to connect web server, rtsp is to connect camera. It should be physically different unless you use AXIS cameras. AXIS supports web socket service in their camera itself as in the ONVIF standard commented. To get access and stream to cameras, we need both physical camera(s) and web server ( webSocket server at this case if no AXIS cam ). I found rtsp server emulator in this project too. I tested working fine at some sort of AXIS camera with recent firmware. In my feeling of testing result, everything specially performance looks great except no HEVC/265. So I am adding it to my project with h.265 from some separate open source projects. Good luck guys

[steabert]

Very good question, but the answer is very long. This library is mostly a toolbox to deal with RTP streams in JavaScript (most of the stuff works both in a browser or with NodeJS, but not everything). The part of the library that deals with streams is not specifically tied to RTP, we just built a component and pipeline as simple wrappers around the streaming concept to enable us to easily combine stream transformations.

The parts that deal with video/audio streaming are split into different components that handle a specific task, like handling an RTP session, extracting RTP payload, constructing an ISO BMFF byte stream stream. Those components can be used together in different pipelines based on your particular needs, and some default pipelines are available for common cases.

1. If you want to deail with RTP data in the browser, you need to get access to the data somehow, and that will be either via streaming over HTTP or WebSocket. So your server needs to support one of those options. The latter is the easiest and is the one that is implemented, while RTP over HTTP is not available (the reason is that e.g. this would be easiest with streaming fetch which is only recently supported, and there is not standard way of delivering RTP over HTTP, that would depend on how it's implemented). If your server does not have WebSocket, you can run a NodeJS WebSocket server in-between as a proxy for the RTSP server.
2. The ws URI tells the component where to connect to on the WebSocket server, while the rtsp URI is used for the RTSP server.

[marklagendijk]

Right. I have the feeling that this is a very powerful library that is useful for quite a number of different use cases. However, these use cases are not clearly listed and documented. I find it a pity that such extensive library, has such little documentation. I think that because of this the library will remain unused quite often, because it is too hard to find out if it does what is needed (and if so, how).

I would recommend starting with a simple (multi-level?) list of the features of the library in the README. To make the library really usable you would have to document the what, how and why of every feature. The documentation mentions that there are some pre-made pipelines for more common scenarios, so a good starting point would be documenting those.

Although RTP streams is the main focus, the library is not limited to handling RTP streams, or to the browser. Its main focus is to handle streams of messages, and make it easier to stitch together transformations from one message type to another. Contributions of new components/pipelines are always welcome.

This is a very broad definition. If the library indeed has functionality that is very broadly applicable, that functionality should get its own library. On NPM the main philosophy is to keep libraries small. You could make it an ecosystem with a structure like this:

• Main pipeline library
• Multiple small libraries that are compatible with the pipeline library
• Third parties would create compatible plugins.

Back to my original questions.

1. I didn't know RTP could be sent over websockets in a standardized way (I'm not familiar with any of the technical details of RTP). Is this the RTSPOverWebsocket that ONVIF describes?
2. This keeps confusing me very much. I would expect only a ws url, because as you mentioned, the stream is send over a websocket. The only way I can see the rtsp url being used is if the client somehow tells the ws server that it wants to receive a rtsp stream from that url, in which case the server would be a proxy.
3. Can this library be used to proxy a RTSP server? If so, how?

[steabert]

1. Yes, that seems to be the case looking at the ONVIF documentation.
2. It is confusing, but as you already hinted at, the WebSocket connection is just a tunnel, a proxy to the RTSP server. Whatever you send over the WebSocket connection gets sent to the RTSP server.
3. Yes, we use that to make the examples work. We run a regular RTSP server on localhost, and then run a separate WebSocket proxy that looks at an incoming RTSP request, parses the address, and then opens a TCP socket to that RTSP address, and then forwards all the data to the RTSP server. Note that when used with e.g. a camera, the RTSP IP address in the RTSP URI might not necessarily have any meaning, as the WebSocket proxy in that case just forwards a known RTSP server on the same system. The proxy for the examples is very simple though, for a proper production proxy it's better to use existing packages that support a TCP over WebSocket use-case.

[lekoaf]

@marklagendijk, perhaps the media-stream-player might be something for you to try instead? It's a React implementation of this library specifically targeting AXIS cameras, with a fully fledged video player out of the box.