Discoverable, decentralized, cached live video streaming

[m-anish]

As a subset of #18 , one of the requests from Zanskar to focus on the live video streaming aspect of the mesh network.

The current implementation involves running vlc on a windows laptop and using its web camera and microphone to create a live video stream, and then asking users in different places to open that live stream in their vlc instances. This works well, but suffers from:

• The fact that this does not happen over a browser, making discovery hard!
• There is no caching of any sort happening in the mesh nodes, so for every new user, the same amount of additional streaming load is created on the network. For example, if the livestream is 300kbps bitrate (video+audio encoded data), and there are 10 users watching, 3mbps of overall network bandwidth are consumed.

What if there could be some kind of caching implemented on the mesh nodes.

I came across this project, whose readme also lists other potentially similar projects.

We can break this down in two parts:

• Discoverability
• Operational efficiency

[m-anish]

This might be interesting to try out.

[m-anish]

See this related issue. Some promising options there.

[m-anish]

Okay! So I did some more research on this:

1. I tried YouPHPTube, which claims to be able to broadcast to a website from a mobile phone. I setup the service on a VM smoothly enough, but am working through issues getting the streaming stack up. This seems like a traditional working solution - the safe bet. I will update if I get things operational. This also comes with an encoder, so whatever formats people upload videos in, it will convert them to web formats accordingly. Amazing!

2. This is a stretch goal - hlsjs-p2p-engine. Decentralized video playing, with support for live streaming. Since this is a stack, we will have to build a basic video website (or use one already available - will need to research). But this will be an AMAZING solution in a decentralized setup.

3. There is also this solution - streamroot based on similar lines, but not sure how much of it is open/proprietary.

4. RTCMultiConnection also looks promising, but the demos don't seem to load, and the project has been inactive since the last 2-3 years.

5. The link above is the work of the same person who has created this aggregated page of all his webrtc work. Lots of potential. (@pdey something for you to tinker with!). Do we have in house basic app development expertise?

6. Dat project seems interesting too!

7. Salsify Future - research project at stanford promising dramatic performance improvements!

8. Tribler Experiments over webtorrent.

9. p2p over webtorrent with some known limitations.

10. peerstreamer-ng looks promising to build upon, even if the documentation is sparse.

[m-anish]

Update. I was able to get basic live streaming to work with YouPHPTube. So we have some kind of a working solution now.

Onwards towards something smoother, and closer to being more decentralized.

Separately, I will work on a playbook to get YouPHPTube into IIAB.

[holta]

Separately, I will work on a playbook to get YouPHPTube into IIAB.

How realistic/practical might it be for educators to "publish" short videos from their Android phones[*] (or short screencasts/tutorials from their laptops) to YouPHPTube on IIAB?

[*] whether 100% homebrewed or from WhatsApp, or YouTube itself...which contains a ton of very thoughtful educational vids and visualizations (amid the swamp ;)

ASIDE: many YouTube science demonstrations are more vivid than PhET science simulations...these 2 artforms should really be combined side-by-side!?

[m-anish]

Another update. Me and @sboruah were able to test that she could livestream from her phone's camera, and I could watch it in a browser.

There is one limitation: The live stream does not get recorded directly onto the site, and it seems that the feature is enabled through a paid plugin. There is a work around though, as it is easily possible for those videos to be recorded on the server somewhere, where they may be presented (manually), through a simple online form, to the encoder as part of this suite.

[m-anish]

@holta for what its worth, streaming, transcoding requires significant compute resources, so if IIAB is running on decent hardware (i3 NUC or so), then it is entirely possible.

[m-anish]

Bumping milestone. Good progress happened on this over the week.

[m-anish]

One challenge with p2p hls webrtc based solution seems to be that we might have to build apps, or hunt for existing ones, that can do a live stream.

[mikkokotila]

Let's see...maybe not. Maybe we just have to allow it through a browser by simple means. Let's see.

[m-anish]

I'm talking of mobile phones. People accustomed to using apps, so it might be the preferred way. Desktop will have to be browser based.

[mikkokotila]

Well, I think people are more accustomed to go to a browser than to an app, even if you count all apps combined vs. browser. But surely browser will win every time a custom made app for a single purpose.

[m-anish]

Also, I have little knowledge of app development, but maybe the security model in phones also encourages apps rather than webcamera+mic permissions in a browser. Anyway, I dont know very well about this, so perhaps that's why I think someone more adept needs to answer this.

[mikkokotila]

It makes sense to have a protocol where we always assume that something can be done in a browser, because it is easy and it does not introduce anything else and allows us to focus on core capabilities instead, prototype fast, etc. etc. and only if that absolutely fails, we consider introducing something new (like an app). Apps are kind of like cognitive garbage...just like we should not use new materials if we can use garbage, we should not use apps if we can use browser.

I'm talking of mobile phones. People accustomed to using apps, so it might be the preferred way. Desktop will have to be browser based.

Precisely, which further makes the case for the browser based approach 100%, as it is same regardless of device and platform. You can be on apple or samsung, or you can be on your laptop or tablet or whatever, it does not matter, you know exactly what to do to access the livestream (or whatever else it may be).

[mikkokotila]

Also, I have little knowledge of app development, but maybe the security model in phones also encourages apps rather than webcamera+mic permissions in a browser. Anyway, I dont know very well about this, so perhaps that's why I think someone more adept needs to answer this.

No. In app you give the permissions to the app, in browser you give permissions to the browser. Access to camera is 17 characters long syntax and works almost universally. (does not work on edge I think).

Once I have the setup in home, we will know much more about all of these things as certainty.

[mikkokotila]

Possibly meaningful here >> https://recordrtc.org/

[mikkokotila]

Some browsers directly support streaming with <video src="http://skynet/stream.m3u8">.

So if I understand correct, you could broadcast from your end in the m3u8 file, and then in the portal landing page and some other page we can broadcast in the network what is live right now, the user clicks, which then just opens the page where the stream is broadcasted.

[m-anish]

Possibly meaningful here >> https://recordrtc.org/

Following on from there... https://www.webrtc-experiment.com/RecordRTC/simple-demos/video-recording.html

Can you check if this works within a mobile browser?

[mikkokotila]

That does not work on my desktop either. MacOSX (old OS) + Chrome.

[m-anish]

Yeah, they say so in the comments.

Do you have an android phone where you can test?

[mikkokotila]

https://www.webrtc-experiment.com/RecordRTC/simple-demos/video-recording.html

This does work on iphone (old). Also I realized that my laptop webcam is disabled so it could never work. I will buy cheap android phone so can test.

[mikkokotila]

Also, I realized that we do have some flexibility initially in terms of supporting streaming, as long as watching is widely supported. For example, if both iphone and android phone works with RecordRTC, then that would do it.

EDIT: I'm studying more about the HTML / Media Source direction and will hopefully start testing things soon in a live network :)

[m-anish]

There are two reasons why we might consider moving to webrtc/hls/newer technologies.

1. They run in-browser, are latest and greatest, are under active development, and offer performance gains.
2. The offer real tangible immediate network bandwidth savings.

Personally, I'm most interested in the second part, but I do fully agree that this is the future.

So,

1. I am going to wait a while before I make a playbook for YouPHPTube
2. If we are able to make a basic web app with these new technologies, test it thoroughly on many platforms and see it works, then we go with that
3. If the above doesnt happen, I'll make the YouPHPTube playbook.

[mikkokotila]

Ok I've seen enough.

I have now established an understanding, roughly, of what is out there, what are the limitations, what's coming etc. The biggest thing I see is related with formal internet privacy regulation.

WebRTC is the most invasive and easy to exploit technology in the current standard stack, and can be readily used to fingerprint users, see through VPN to the originating IP etc so it's definitely too early to adopt anything WebRTC based.

For everything else, which relates with the other big issue I see, the eco-system being currently really fragmented and it seems like nothing supports "everything", there are interesting choices but it seems that everything is still "up and coming".

I think that we would get very quick early wins, probably it would take a couple of hours max to set something up, and then face endless browser/device compatibility issues.

Let's go with YouPHPTube and come back to this topic later.

[m-anish]

I think hlsjs-p2p-engine is worth a serious try! I got it to work in a very simple series of steps.

• Open a terminal and type the following commands

  
  #make a test working directory and cd to it
  mkdir test && cd test
  #clone hlsjs-p2p-engine
  git clone https://github.com/cdnbye/hlsjs-p2p-engine.git
  #create a live stream from the webcamera (the command below works on Linux, might need changing on MAC
  ffmpeg -f v4l2 -i /dev/video0 -vcodec h264 -f alsa -i pulse -acodec aac -strict experimental -pix_fmt yuv420p playlist.m3u8 

start a simple http server on port 8000 in another terminal

python -m http.server

* Open the file `./hlsjs-p2p-engine/demo/videojs-demo.html`
* Change the m3u8 source line in the file above from whatever it is to `http://localhost:8000/playlist.m3u8`
* In a browser, open `http://localhost:8000/hlsjs-p2p-engine/demo/videojs-demo.html`. You should see a livestream!
* In another browser, open the same link, and the stream should open there as well.
* Wait for a few seconds to see the statistics being updated (this might not work if your browser has any kind of tracking or ad-blocking turned on)
* You should also be able to see this in vlc. Open network stream: `http://localhost:8000/playlist.m3u8`
* Remember to clean up the many `.ts` files and the `.m3u8` file that are created afterwards (housekeeping)

### Next step
* Create a local [signaling service](https://docs.cdnbye.com/#/en/signaling) instead of using the default one.
* Use local js files instead of using them from the cdn.

Please test at your end, and let me know if this looks promising from the POV of creating a webapp around it. (@mikkokotila @pdey )

[mikkokotila]

Ok, another part of the puzzle is solved.

<video autoplay></video>

<script>
const constraints = {
  video: true
};

const video = document.querySelector('video');

navigator.mediaDevices.getUserMedia(constraints).
  then((stream) => {video.srcObject = stream});
</script>

This yields a seemingly compatible way to prompt the user to allow access to device camera and immediately start broadcasting it.

Note that this (and I think all other similar approaches) only work with https.

[mikkokotila]

One upside here is that the same allows screenshots and as far as I understand, screensharing.

[mikkokotila]

Ok, so I'm happy to say that after trying +20 different approaches, I have now peer-2-peer one-to-many live streaming working on IIAB! :)

More tomorrow.

[mikkokotila]

TODO:

• get socket.io server running locally
• test that everything works on the mesh as well
• test recording
• test many-to-many

Then finally after that, create a simple UI that allows going to a URL and starting a broadcast or watching one of the currently live broadcasts (we might want to separate these).

[m-anish]

Ok, so I'm happy to say that after trying +20 different approaches, I have now peer-2-peer one-to-many live streaming working on IIAB! :)

More tomorrow.

Would be interesting to know whether you tried the multiwebrtc or the hlsjs-p2p approach, as the latter seems more cross-browser compatible.

Anyway, will wait for more updates!

[mikkokotila]

That's right, we're on RTCMultiConnection.

As you point out, for the time being this is not supported by edge (and I think IE) so that would be something to consider in comparison to hlsjs-p2p. It seems that given the choice, we want ie/edge support, or?

[m-anish]

We definitely want more support, and also prefer building upon known, well used, cross platform containers, the likes of which hlsjs-p2p claims to support.

Another thing to consider here is, all kiwix videos are rendered within video.js, so we could potentially alter the code within the kiwix html pages to use hlsjs-p2p. It might be useful in some situations, for eg: a teacher requesting a class to open a particular video, and that being streamed p2p. (So this is one potential application outside of live-streaming).

RTCMulticonnection looks compelling from the point of view of enabling video to be recorded and streamed at the same time. I think one thing to think about is to separate the "streaming" and the "viewing" portions, if that helps makes things easier.

[mikkokotila]

We definitely want more support, and also prefer building upon known, well used, cross platform containers, the likes of which hlsjs-p2p claims to support.

I don't think that RTCMultiConnection is any worse "supported". I'd say better, because it makes effort to work with php, jquery, etc. We're going to end up using just one of the so-called 'containers' so having support for many does not seem to be a strong qualifier.

Another thing here is that it seems to offer us everything we are looking for, and then if user wants to use some other player, that could be possible as well. This totally covers the use case you had mentioned, where a recording is streamed p2p by the teacher to students, while they might be doing something else as well on their desktop (or not). As I understand, in "Khan's eco-system" every possible use-case is already covered. Most have copy-paste demos.

Having learned from you the way to evaluate open-source project, it seems that RTCMultiConnection wins on all accounts. Also, I like the fact that the guy is solely dedicated just to webRTC. Yesterday when I asked a question, he replied in minutes and directed me to one of his other packages.

When it comes to IE/Edge ... well ok, I agree we should have wide support, but if the only thing that is not supported is IE/Edge ... I'd ask the question about how long will microsoft support them as the thorn they have been in web developers flesh for 25 years, and make it into something more compatible. There is benefit for having mesh users adopt Chrome, Brave, or Firefox anyways in my view, so the worst case would be to have a full screen attention grabber that says "click here to install so and so browser".

I'm not saying that we should choose one and other though, just highlighting the other side of the argument.

[m-anish]

I also feel, besides this, we should setup some kind of test environment, and actually test both the solutions in our typical mesh setup and see real performance.

Aside: Also, in my basic testing on linux mint (based on ubuntu LTS running up-to-date-firefox) multiwebrtc p2p video broadcast didn't work (while it did in brave). So I am less convinced of broad support until i see results.

Video.js for example, is in use by Kiwix and being used in IIAB for the past 3+ years atleast, and is known to be reliable. There have been kinks with browser versions in the past, but they have worked quite hard to get to a stage where it works on most of the browsers IIAB typically is accessed in.

So, one question in context of Kiwix. Would multiwebrtc work from within video.js? What modifications to code are required. In the case of hlsjs, it just seems to be the matter of replacing one js script with another and keeping the rest of the html code exactly the same.

As I understand, in "Khan's eco-system" every possible use-case is already covered. Most have copy-paste demos.

Don't know how this is relevant (and also can you elaborate what you meant here by copy-paste demos). Khan academy is different to Kiwix (offline TED talks, Dalai lama teachings, and many other videos).

[mikkokotila]

Don't know how this is relevant (and also can you elaborate what you meant here by copy-paste demos). Khan academy is different to Kiwix (offline TED talks, Dalai lama teachings, and many other videos).

Simple misunderstanding. My bad for ambiguity. The webRTC guy's name is Khan.

[m-anish]

fwiw, it's great that the maintainer of multiwebrtc is a very responsive person! +1 to that

[mikkokotila]

So, one question in context of Kiwix. Would multiwebrtc work from within video.js? What modifications to code are required. In the case of hlsjs, it just seems to be the matter of replacing one js script with another and keeping the rest of the html code exactly the same.

I do agree, but I guess it's pretty easy to just jump to it yet we might end up being years with it. If we have to pull the trigger on this, then may it be hls-2p2 as plan a.

[m-anish]

This might be a useful pointer to go about building a rudimentary web app https://aaronparecki.com/2016/11/19/15/self-hosted-facebook-live

[m-anish]

Related to #25

[m-anish]

Related issue #17

[m-anish]

Okay, so @pdey @Ishanchr and me made some progress on this. We tested hlsjs-p2p-engine in a video.js container. We setup a schoolserver on an Intel NUC and had two mesh nodes connected to it.

We got encouraging results, and a list of things to do for future runs.

• [ ] The video.js example still relies on some scripts being loaded by a cdn. We need to have them offline.
• [x] The signaling service also needs to be locally installed.
• [ ] We need to look into the p2p sharing parameters, and optimize them over many rounds of testing.

My suggestion would be to work within a fork of the IIAB project. It can live either in my fork, or @pdey or in eka. That may be a starting ground for a PR for wider inclusion within IIAB.

[m-anish]

Another question/thought.

Do you think this might be doable on the mesh nodes themselves?

[mikkokotila]

I can't see any reason why this would not work on just the mesh node.

[mitra42]

@m-anish asked me to comment ... I built the dweb.archive.org site as a decentralized version of the Internet Archive, and am about half way through a year long project dweb-mirror for a offline version of the Archive (suitable for mesh networks etc), it currently can crawl IA resources; serve them fully offline and also act as a proxy (saving as you browse a local server). Its tested on MacOSX; RPi (NOOBs or IIAB); and Rachel3+/WorldPossible though its still got bugs I'm working through.

For video ... we tried WebRTC - it was a massive resource hog, caused browser crashes etc, though we think that was mostly because it was opening lots of connections to DHT's. Like most people in the P2P community we dropped it as far as possible. The one video system we found worked well was WebTorrent, especially because it can easily be setup to have fall-back Http URLs, which works well when you have a mix of popular videos (lots of peers to connect to) and unpopular (no other peers online), so the latter gets seeded by http from an origin server.

I'm part way through a process where videos viewed on the dweb-mirror project will be shared by any peers using webtorrent, in part to get around the well-known issues with bandwidth on Raspberry Pi's.

[m-anish]

@mitra42 thx for commenting!

Did you also get a chance to try hls? We experimented with hlsjs-p2p-engine and got decent results for live streaming and are thinking to build around that? HLS is not 'true' live, and there is a lag of about 10-20 sec but it is perfectly acceptable for our scenario.

Also, re: webtorrent, is there a project that we can test on the mesh we have locally to do some live video streaming? How well/badly does it work on mobile phones? Would love to test. I came across this in my research, but didn't test it as it seemed they had some known limitations, and limited browser support, but perhaps should rethink that decision.

One reason we are considering the hlsjs-p2p-engine is it seems to be well supported across browsers through various containers (video.js et. al.). We are also thinking to integrate hlsjs-p2p-engine in the video.js container within kiwix zim files which are heavily used within IIAB.

[mitra42]

I haven't tried webtorrent for livestreaming, its not our application, our application (at Internet Archive, and in dweb-mirror) is for static video, and in particular to make sure the experience is always as good as a direct connection to the http server.

[mitra42]

With just video and relatively small number of viewers then for live stream then webrtc (which is what hlsjs is built on) may be what you want, I'm just sceptical of it for viewing static videos given all the problems and how many people have abandoned it. Have you also tried it fully disconnected, webrtc has supposedly some single-point-of-failure issues with single rendezvous servers, but maybe hlsjs knows a way around it.

[m-anish]

Hmm, I haven't yet tried hlsjs-p2p-engine fully offline. It needs a signaling server and a few js files. We ran the signaling server locally but the js files were still being served online. I think we should make it a priority to test it fully offline (@pdey ), but I am optimistic that it should work offline.

For VOD, we do not envision a lot of p2p, atleast initially, as the number of concurrent users watching a particular video may be small, but still it might get around cases where when a teacher in a class asks pupils to load a video in classroom laptops/tablets and suddenly there is a lot of load on the server.

Thanks for your comments, this is helpful info. To set some basic context, below are two blogposts about Zanskar, where we hope to test and deploy these technologies (more posts to come in the near future).

https://medium.com/eka-foundation/en-meshed-in-la-la-land-part-1-34e0ce29ea1b https://medium.com/eka-foundation/enmeshed-in-la-la-land-part-2-9799b6f8e9d2

[georgejhunt]

Seems like ipv4 multicast can be brought into the picture in the case where a whole class needs to have access to the same video, at the same time

[mitra42]

If what you are doing is static video, I'd strongly recommend using WebTorrent instead, because I'm guessing that the case of a class watching recommended videos is probably more common than all watching the same video at the same place at the same time.