search

voc / voctoweb

voctoweb – the frontend and backend software behind media.ccc.de
GNU General Public License v3.0
188 stars 56 forks source link

Provide videos on Peertube #342

Closed Nutomic closed 3 years ago

Nutomic commented 5 years ago

I don't know if you are aware of Peertube or not. It's an open source video platform that federates videos between different servers. It would be really cool of CCC videos (specifically 35C3) could be offered on this platform. You could do this by posting videos on an existing instance, or by hosting your own instance.

https://joinpeertube.org/en/

saerdnaer commented 5 years ago

Of course we are aware of Peertube, but we are not shure whats the best solution for our case:

The Web-Torrent part of Peertube is not really needed in our case. In the past our existing supporter based CDN worked better than any torrent based solution. Especially due to rereleases, etc.

Nutomic commented 5 years ago

If you want to use an existing instance, there is a list of instances at Joinpeertube. However, I don't think there is any existing instance that provides enough storage space for all your videos. Maybe you could upload some highlights or popular videos though.

You could host a your own Peertube instance as well, under something like peertube.media.ccc.de. The main disadvantage is that you would have to store all the videos twice. And I doubt you want to completely replace your existing site with Peertube.

It might be the best option to implement Peertube compatibility directly, at least basic features so that other instances can follow it.

ping @Chocobozzz @rigelk

rigelk commented 5 years ago

The best course would be to implement some ActivityPub so that other PeerTube instances can follow media.ccc.de and display your videos. Forget WebTorrent if you think you can handle the load, it's optional anyway.

Now, I'm not sure what is the minimum to implement in order to join the federated network of videos (I've not written that part of PeerTube's code :)), but @Chocobozzz would know. Listing what's basic compatibility would be a first.

ghost commented 4 years ago

@saerdnaer

As peertube is decentralized we could host our own instance, but we already have media.ccc.de...

You could substitute voctoweb with a PeerTube server and then re-create the current look and feel of voctoweb into a PeerTube theme (PeerTube now has support for custom themes https://docs.joinpeertube.org/#/contribute-plugins). So the end-users won't even notice that media.ccc.de is using a different back-end.

Maybe we should implement ActivityPub in voctoweb, so it becomes compatible with Peertube instances?

IMO this is another great option as well!

a courteous ping @Chocobozzz

rigelk commented 4 years ago

re-create the current look and feel of voctoweb into a PeerTube theme (PeerTube now has support for custom themes docs.joinpeertube.org/#/contribute-plugins)

Voctoweb provides a slightly different feature set, and suggesting to just ignore that and go with PeerTube would be insulting the work done so far. Much like RSS, exposing the collections as ActivityPub is a perfect case for interoperability, and IMHO a much better option.

Matthias84 commented 4 years ago

As there are existing mastodon accounts @c3voc@chaos.social, @c3voc_releases@chaos.social I guess there is a idea that fediverse might be a place worth to stay :smiley:

I analysed the featureset of media.ccc.de frontend and compared with current peertube instances:

Ok so, I tried to summarize the pros / cons of the two different approachs:

pros - switch 2 peertube instance

pros - add peertube compatible actvitiypub API

So how can people & your grateful users assist on this feature?

Maybe it's a good first step to add a publishing to a existing peertube instance? (e.g. for conferences conf.tube ) So we invest a minimum of energy and get feedback by the operator about performance and limitations. You might also contact users which already mirrored c3voc videos:

There seems also already a script for uploads: https://git.lecygnenoir.info/LecygneNoir/prismedia

saerdnaer commented 4 years ago

@Matthias84 We can give you an VM where you can setup a peer tube instance if you want... :-)

Matthias84 commented 4 years ago

@saerdnaer er Thanks for the offer, but i lack oft time and skills to geht this job done. Would it be OK, if I seend out a General CfP on thr fediverse?

derchrisuk commented 4 years ago

The only valuable option in my opinion is to add Activitypub to Voctoweb

https://docs.joinpeertube.org/#/api-activitypub

Matthias84 commented 4 years ago

Send out a call for participation: https://framacolibri.org/t/ccc-need-somebody-for-setup-peertube-hosting/9488

derpeter commented 4 years ago

@Matthias84 Offering a VM might be a little bit to quick of an response. We CAN'T just give someone from the internet a VM. This might be possible after knowing the person well and agreeing in the team about it. Please edit you call and remove that offer.

danimo commented 4 years ago

@Matthias84 In addition: While I think it would be nice to add fediverse capabilities to voctoweb, I do not see us abandon voctoweb for peertube. You are implying in your post that it would be an option. It is not. It's perfectly possible to add a fresh UI to voctoweb, but that takes a UX person, not a whole new software stack.

lukas2511 commented 4 years ago

I kinda like the idea of Peertube, but I don't see this happening.

As a replacement for voctoweb it would be tons of work porting all data over, fixing the links, changing workflows with all of our contributors, find a solution for existing mirrors (which mostly only support rsync+http/https), etc. It might have all the features we need (and more), but they all work completely differently than what we had before and every single piece of our automated and manual pipeline would be incompatible, links and apis would be broken, etc.

As a standalone system it wouldn't receive much care and would probably always be outdated and in some ways broken. We already have enough systems to handle with a team that seemingly becomes smaller every other month...

Matthias84 commented 4 years ago

@derpeter I added a note to avoid wrong expectations. At mastododon there is drq offering help to test peertube, if you are interested?

@danimo @lukas2511 I'm sorry if my suggestion looks like a final decision of the VOC. In the end it is completely up to you. I see that keeping an infrastructure running is a challenge and extending it, is also a hard task. This is why I suggested to make use of software which is maintained with a bigger developer base?

Sounds like adding peertube API and a separate peertube instance are no features, which others might work on?

derpeter commented 4 years ago

@Matthias84 so far we dont understand how peertube solves any problem but only see that it introduced new issues. You are completely right that it would be nice to have a software with a bigger developer base, no discussion about that. But i just dont see how peer tube can help us.

Bittorrent is a terrible technology breaking the internet in its current core (TCP fairness). Even with webseeds torrent clients break your servers as the ignore chunk sizes and request limits which ends up in ddos situation. Further it adds overhead in terms of bandwidth and processing. It also removes privacy completely from the equation, everybody can see all the time which talks you are watching. Also it costs more energy to run hundreds of seeders and the infrastructure around them. Nearly every year people show up and tell us we should enable torrents back, use ipfs or some other distributed filesharing stuff. Sometimes they even set up instances. It always ends at the point where someone would need to have 10TB of files in a distributed file sharing thing and keep them online. So far no one invented a magic trick how this could be achieved. At the end you end up with running the same amount of servers you need for a normal cdn but with a more complex software stack and less privacy for our users.

It seems that peertube in theory can work on a normal CDN but removed a lot of benefits and makes it frontend with a api incompatible to everything we use. Also we run our own player in our streaming and media platform and working on it. Im not sure how that could be used with peertube.

My understanding is we will end up maintaining are very complex software not in the way it is meant to be used and therefore probably not supported by its developers to have no benefits.

I might just miss the point where peer tube solves some of the points or even has advantages. But looking at your comparison above it seems it has only missing features in contrast to what we currently have.

I`m happy to be proven wrong here but my feeling is there is just some misunderstanding on the problem to solve here

it goes without saying that having pubsub in vocotoweb (the software behind media.ccc.de) would be nice

eest9 commented 4 years ago

Hi,

@derpeter I'm not sure if i get it right, but it seems to me as the conversation shifted from "we don't know how to federate our videos with peertube instances" to "what is even the benefit of implementing federation".

One of the main benefits would be, that if we federate our videos we easily would get indexed by sepia search and therefore we get a bigger international audience. And of course our videos would be accessible through other peertube instances and their users (not to forget the fediverse over all).

I personally also see societal benefits. media.ccc.de distributes their videos also via YouTube, why should we then build fences to the Fediverse? Opening the gates to PeerTube would flood the system with loads of high quality talks form several types of conferences. This would make it easier for people to switch from silos like YouTube to fields like Peertube. But I understand that this wider perspective doesn't change anyone's mind.

Why we should implement ActivityPub and federate with Peertube instances? Become findable by users of the Fediverse especially over sepia search.

rigelk commented 4 years ago

No disrespect intended to @Matthias84 but their summary shouldn’t be used as the sole basis to decide if PeerTube and/or its ActivityPub dialect are worth the time investigating. As mentioned, "pubsub" would be nice to have, and you kind of already have it with RSS, but that is missing the main point @eest9 mentions.

As this has nothing to do with federation, what follows is probably out of topic, but I’m willing to shed light what was said of PeerTube’s handling of file delivery, which seems to be - aside from feature-parity and external governance - @derpeter ’s main gripe with PeerTube:

Bittorrent is a terrible technology breaking the internet in its current core (TCP fairness). Even with webseeds torrent clients break your servers as the ignore chunk sizes and request limits which ends up in ddos situation.

I’m not fond of BitTorrent either when it comes to streaming in a web page, but that doesn’t mean what you say applies here:

everybody can see all the time which talks you are watching

This is simply not true, please check the about page of any PeerTube instance. If you are unhappy with the answer, again PeerTube can function with no p2p tech, just serving files by itself.

with a api incompatible to everything we use. Also we run our own player in our streaming and media platform and working on it. Im not sure how that could be used with peertube.

Videos can be streamed from their origin URL, with a predictable scheme once you know their UUID. NewPipe figured it out, with their own HTTP player.

peertube in theory can work on a normal CDN but removed a lot of benefits

We show how to run a S3-powered CDN in our documentation so that it serves files directly without proxying them via the PeerTube server. @derpeter could you expand on what benefits we removed?

drequivalent commented 4 years ago

Hello!

If you're looking for a guy to do the job of installing and/or maintaining your Peertube instance, I could be that guy and I'm looking for a job. Helping to expand the Fediverse would be a dream job for me. Also, I always thought that media.ccc.de should always have been accessible to the Fediverse, as it will bring more audience and interaction.

You can contact me on the Fediverse at drq@mastodon.ml

derpeter commented 4 years ago

Hi,

@derpeter I'm not sure if i get it right, but it seems to me as the conversation shifted from "we don't know how to federate our videos with peertube instances" to "what is even the benefit of implementing federation".

One of the main benefits would be, that if we federate our videos we easily would get indexed by sepia search and therefore we get a bigger international audience. And of course our videos would be accessible through other peertube instances and their users (not to forget the fediverse over all).

Thats not my point. As i said having a pubsub API is a nice addition.

I personally also see societal benefits. media.ccc.de distributes their videos also via YouTube, why should we then build fences to the Fediverse? Opening the gates to PeerTube would flood the system with loads of high quality talks form several types of conferences. This would make it easier for people to switch from silos like YouTube to fields like Peertube. But I understand that this wider perspective doesn't change anyone's mind.

Why we should implement ActivityPub and federate with Peertube instances? Become findable by users of the Fediverse especially over sepia search.

TBH i never heared about sepia before but yes if thats help people finding the content thats great. Replacing media.ccc.de with peertube, replaceing our CDN with torrent stuff no that is just a bad idea for all the reasons stated before. So for there where no comments on the issues, that would be very helpfull in this discussion.

derpeter commented 4 years ago

Bittorrent is a terrible technology breaking the internet in its current core (TCP fairness). Even with webseeds torrent clients break your servers as the ignore chunk sizes and request limits which ends up in ddos situation.

I’m not fond of BitTorrent either when it comes to streaming in a web page, but that doesn’t mean what you say applies here:

  • WebTorrent uses WebRTC, which doesn’t have the aforementioned fairness issues.

Yes, but we would still need servers that serve those files. These servers need to run a webrtc server which non of our cdn nodes do and as we only operate some of them we just cant change that. To my knowledge webrtc does not prevent the ddos issue but hopefully the clients implement it nicer then torrent clients so far.

In a world where we get webtorrents to work with all CDN nodes we end up with having less efficient cdn then now with no benefits over HTTPS links.

Where do you see a benefit here?

  • PeerTube instances can be set to offer WebTorrent, or HLS (which performs better and is also p2p-enabled in our version), or both, or none and just provide videos directly.

What would be the point of that? Again we need to operate enough servers so serve those files. 10TB is nothing we can hope the community will mirror. So we operate the same amount of servers but have more trouble to do so third party Mirrors run webservers not peertube.

  • WebSeed clients are mostly visitors of your frontend (that is, browsers which respect chunk sizes).

everybody can see all the time which talks you are watching

This is simply not true, please check the about page of any PeerTube instance. If you are unhappy with the answer, again PeerTube can function with no p2p tech, just serving files by itself.

I was referring to the p2p version. Of course if we play unicast from a server than we are back to more complicated HTTPS downloads and the same applies.

with a api incompatible to everything we use. Also we run our own player in our streaming and media platform and working on it. Im not sure how that could be used with peertube.

Videos can be streamed from their origin URL, with a predictable scheme once you know their UUID. NewPipe figured it out, with their own HTTP player.

So far i could not find out how updating releases would work here without breaking scheme. Also what would be the benefit?

peertube in theory can work on a normal CDN but removed a lot of benefits

We show how to run a S3-powered CDN in our documentation so that it serves files directly without proxying them via the PeerTube server. @derpeter could you expand on what benefits we removed?

So if we run a dedicated server based CDN which is not federated at all and uses webtorrents instead of HTTPS we have the same we have now. We will need to do the same things like now. But we changed the software stack so its only working with peertube and not the millions of clients it works right now with. All the media players and so that not implement webtorrent but HTTP. So why on earth should we do this?

Don't get me wrong, i said it before: federation would be awesome. In terms of allowing search engines to index, having different video platform seamlessly integrate our content. Actually there are other peoples operating vocoweb (the software behind media.ccc.de) instances and its not nice that the content is not shown in the other ones. I get all this an would love to the a solution implemented. But i see no point in changing our cdn to webtorrent based solution. So if its possible to include https links in peertube and use peertube as an alternative frontend, that would be awesome. Forcing webtorrents just for the sake of having webtorrents seems wrong to me.

rigelk commented 4 years ago

You. Don't. Need. WebTorrent. With. PeerTube.

I don't know why you think our server implements WebRTC, but PeerTube's server part has nothing of the sort. It is also relying on efficient file serving over HTTPS, much like your CDN.

Matthias84 commented 4 years ago

(I wasn't aware on TCP fairness troubles of bittorrent (thanks!) but not sure if this is the right place to discuss other projects technical decissions?)

media.ccc.de is a decentralized solution which has IMHO also some negative implications from an external POV (hard to find the page, hard to search for content, interoperability and linking to other video platfoms, interact with video content, ...) So idea of fediverse is to interconnect instances and create a more powerfull universe (read: diverse, easier to query, easier to navigate, ...). Esp. the idea of peertube is to unite different communities and offer interoperability to search & discuss videos (and share / mirror them of course). IMHO it's worth to join this network and support the idea of a distributed alternative to Youtube (Which AFAIK is close to the goals of CCC and the hackerethics?).

I can't answer what is the easiest / durable / ... solution. I just can offer my personal perspective, that I really appreciate the work of C3Voc in technics and recordings and that I feel sad, that you need to mirror your talks to Youtube, to gain popularity to the non geekish audiences (like universities, artists, ...) and to get in contact with international users / conferences / channels / ... as well. As operating this Youtube channels takes resources (and IMHO it's worth) it might be also worth to invest some time to add a peertube compatible API or to operate a separated Peertube instance. But this is just my personal impression that CCC might be a player which is worth to join the fediverse, as a lot of hackers have also found a new social media home. (Even worth, I can't offer any help because I have no admin experience in peertube and have not the time resources to setup hosting or start learing ruby to extend voctoweb with W3C activitypub flavor of peertube)

derpeter commented 4 years ago

You. Don't. Need. WebTorrent. With. PeerTube.

I don't know why you think our server implements WebRTC, but PeerTube's server part has nothing of the sort. It is also relying on efficient file serving over HTTPS, much like your CDN.

I think i didnt say any of this. But lets just close this here from my side. Again Federated == nice, p2p stuff just dont works for our use case. media.ccc.de is a webite voctoweb is a software.

Please feel free to continue the discussion without me.

saerdnaer commented 4 years ago

Result of discussion in media.ccc.de meeting on 2020-10-28:

derchrisuk commented 3 years ago

I just installed a test instance. While checking the settings, I could not see an easy way to import a subset of files from our CDN. Uploading a file from URL will result in the file being downloaded to the server. So we would need a way to import local videos into peertube, with playback from CDN and same for federation. Looking through the plug-ins I could not see something which could help with this.

Matthias84 commented 3 years ago

There seems to be a official importer which makes use of youtube-dl that is able to understand media.ccc.de content and metadata: https://docs.joinpeertube.org/#/maintain-tools?id=peertube-import-videosjs

I'm not sure how to import local videofiles, but maybe this is worth a question to the peertube community? https://framacolibri.org/t/bulk-videos-import/10124

derchrisuk commented 3 years ago

The problem is these are all imports, meaning the file will be downloaded to the server. That is not what we want to achieve, as we already have the files and don't want a 2nd file store. Just reference them as a local filepath or better as URL which would point to the CDN version. With the current setup PeerTube seems to be no option for us.

Matthias84 commented 3 years ago

yes, they confirm, that a non mirror approach will doesn't work https://framacolibri.org/t/re-share-existing-online-videos/10304

is it possible to setup an optional test instance for a limited amount of content? e.g. experiment with the upcomming rc3? (e.g. to underline the idea of a decentralized congress?)

lukas2511 commented 3 years ago

yes, they confirm, that a non mirror approach will doesn't work https://framacolibri.org/t/re-share-existing-online-videos/10304

is it possible to setup an optional test instance for a limited amount of content? e.g. experiment with the upcomming rc3? (e.g. to underline the idea of a decentralized congress?)

That doesn't really make much sense. We are already providing a rather big CDN for our videos and have quite a few independent external mirrors. Copying over data to a new platform just to then have to provide a CDN again* seems really really really stupid.

@derchrisuk I'd suggest to stop working on this if it can't integrate nicely with what we already have.

The only option I still see viable would be integration of pubsub into voctoweb. That seems to be the only real benefit of peertube anyway (at least for us) and if we integrate it then we don't need a standalone peertube instance anymore.

*Because let's be honest, even with all fancy federation and p2p features and everything with our amount of content nearly 100% of the traffic would still have to be served by us

rugk commented 3 years ago

Hmm has not the outage of the rc3 shown that this could be a useful feature, for high load, at least? When a video gets popular, p2p could reduce the load of the videos when they are watched at an event/live.

lukas2511 commented 3 years ago

Hmm has not the outage of the rc3 shown that this could be a useful feature, for high load, at least? When a video gets popular, p2p could reduce the load of the videos when they are watched at an event/live.

The outage during rc3 was an attack and it took out big parts of our frontend infrastructure until we did lots of optimization to stay online. Our mirrors staid online the whole time and finished recordings were available from multiple sources. Peertube neither has streaming capabilities nor would it be a good choice for multi-day long stream dumps. Also since none of the peertube instances I've tried were even stable during normal operation I wouldn't expect much during an attack. Maybe one or two videos would have staid online, but sorry, that's not worth the effort and infrastructure.

rugk commented 3 years ago

Yeah, I never said you should switch to PeerTube or or of their instances. Just would be awesome if the Peertube (p2p) technology were integrated in Voctoweb.

derpeter commented 3 years ago

@rugk peer tube would have been even worse in this scenrario. Lets first distinguished, there is live streaming (which is not part of this discussion and peertube) and media.ccc.de which is.

If a ddos attacks the peertube instance that is the origin of files all federated tubes will also be not able to help here. Even if they have the files already locally synced it will still not work. As far as we know there is no way do decentralize this despite from using traditional load balancing/failover in front of your peertube. This would render peertube pretty much useless as there is no benefit anymore.

On the other hand our CDN was never down and all mirrors where still capable to deliver the content. Also media.ccc.de it self was never down, only the DNS servers where down for a short time. As lukas mentioned this has been addressed now. Also we are working on distributing the frontend and database further.

DNS would also affect peertube. DDossing peertube is much easer then ddossing a our CDN as long as you don't operate the same amount of peertube server as CDN servers and give them the same bandwidth and capacity. At the moment you do this you end up with a less efficient CDN which is hard to load balance. So again peer tube solves non of the problems but makes them worse.

As stated before federation and voctoweb would be nice. p2p solves non of the issues. fedration does not need p2p file distribution to work.

BUT: all recordings are public, all metadata is available in a public API. No one stops you from setting up a peertube instance, write an importer (we checked but didn't find a working solution) and operate that. We will be happy to link to it.

Also we had torrents in the past. As we thought that's nice to have p2p. Turns out in most cases the torrent where loaded via the webseeds which are our HTTP cdn hosts, so no p2p. People who shared the torrents via torrent where not able / willing to do that for a long time. Files where we fixed e.g. audio issues where still in torrent network and people where complaining about long fixed issues. And last but not least performance and privacy was worse.

So while p2p and federation sounds nice on the paper in reality there are problems than where never solved and to at least to my understanding how networks work will never be solved.

rugk commented 3 years ago

Okay, that is interesting to hear, although it totally destroys the (only) one promised advantage of peertube: That you e.g. in your LAN (and the c3 is a perfect example for that), can distribute the load and do not need to re-download the video each time. :thinking:

lukas2511 commented 3 years ago

Okay, that is interesting to hear, although it totally destroys the (only) one promised advantage of peertube: That you e.g. in your LAN (and the c3 is a perfect example for that), can distribute the load and do not need to re-download the video each time.

Distributing the load in terms of traffic was never an issue, so it's not a benefit for us at all.

eest9 commented 3 years ago

It's not the same as implementing ActivityPub or host a peertube instance, but at least new video releases are now available as toots. Someone created an RSS to Mastodon bot for media.ccc.de: https://botsin.space/@mediacccde

lukas2511 commented 3 years ago

It's not the same as implementing ActivityPub or host a peertube instance, but at least new video releases are now available as toots. Someone created an RSS to Mastodon bot for media.ccc.de: https://botsin.space/@mediacccde

We already have a more "official" channel for that https://chaos.social/@c3voc_releases but it seems our bot died.

derpeter commented 3 years ago

Its not a bot but the releasing pipline. It was disabled, i currently try to figure out why but it will come back. It also twitters usually.

Update: It was disabled as the link URLs where wrong because of some artefacts of merging schedules from different tools and studios in combination with relive.

yoe commented 3 years ago

FWIW, I (FOSDEM video team member) have been looking at this too, with pretty much the same tradeoffs :-)

If you guys are doing pubsub as part of voctoweb, then it really only works for you.

I've been thinking of creating something that does pubsub for a bunch of video URLs, where you basically throw metadata at a daemon (e.g., various video URLs, a description, a license, a HTML page, etc), and it serves it using the ActivityPub protocol.

Nothing exists yet, but early research suggests that it should (theoretically) be possible to implement this on a standalone URL, with just a few files dropped into your /.well-known directory so that you would be able to subscribe to your "regular" URL, and that's it.

saerdnaer commented 3 years ago

@yoe That would be really nice – can you keep us informed?

neufeind commented 1 year ago

Recent news from peertube says they have a feature to include content from "remote channels", from a separate platform. Could that help getting external videos listed into a peertube-instance for media.ccc.de - and from there also listed in the Fediverse, in searches set (but still keep it hosted on media.ccc.de)? https://joinpeertube.org/news#release-4.3