search

Chocobozzz / PeerTube

ActivityPub-federated video streaming platform using P2P directly in your web browser
https://joinpeertube.org/
GNU Affero General Public License v3.0
12.74k stars 1.46k forks source link

IPFS to store videos #494

Open alxlg opened 6 years ago

alxlg commented 6 years ago

I think that what limits PeerTube adoption is that instances are perfect for personal/organization use but not to build a free service like YouTube where everyone can upload videos without limits. The issue is that storage has a cost and videos make the necessary storage grow quickly.

IPFS (InterPlanetary File System) can be used to solve the storage issue because every user can store the files by himself but it doesn't have a way to browse and interact with them. PeerTube instead has an awesome UI that can be used by everyone.

Would it be possible to combine PeerTube and IPFS? Ideally the instance administrator would limit the classic upload for each user but eventually let users upload videos by specify an IPFS address. I guess when a second user browse a PeerTube instance and want to watch a video hosted on IPFS, PeerTube provides it by reading from IPFS and not from its local storage. PeerTube instances would cache IPFS contents like IPFS users and admins would monitor IPFS cache impact on their storage. If a PeerTube user wants to be sure its video is available he just have to keep it on IPFS with his machine. This could have another advantage: if the used PeerTube instance won't be available anymore its users won't need to upload videos on other PeerTube instances if they are on IPFS: they would just "upload" the IPFS addresses.

I will be grateful to those who answer by denying or confirming my assumptions.

rigelk commented 6 years ago

Ideally the instance administrator would limit the classic upload for each user but eventually let users upload videos by specify an IPFS address. I guess when a user […] wants to watch a video hosted on IPFS, PeerTube provides it by reading from IPFS and not from its local storage.

@alxlg indeed that's more or less how I envisioned the potential use of IPFS. But then there's the fact an IPFS endpoint is not a webseed[¹] nor holds versions of different quality. In other words, IPFS would only be a second class citizen feature-wise.

¹: let me extend a bit on that issue. The fact is that we use WebTorrent (BitTorrent/WebRTC) on the client side to watch videos. It provides a handy pool of content seeders and direct browser connection. Watching a video via IPFS would mean to replace entirely that component with an IPFS client in the browser. So it's not just thinking of a different storage/upload mechanism.

If you have any ideas as to how to solve these problems, I'm all ears :)

P.S.: we have also not heard a lot about IPFS performance-wise when it comes to streaming videos.

alxlg commented 6 years ago

@rigelk thanks for your reply!

I had not thought of different quality versions of videos. Since IPFS is really low in the stack the only solution I can think of is storing a different IPFS file for each version. The user should be able to specify the IPFS address for each quality version he wants to maintain... This doesn't seem user-friendly but with a desktop client that automatically manage versions on IPFS it could gain adoption... Ideally the desktop client could use some API to upload the video to PeerTube by specifying many IPFS addresses. Desktop client's users should just pick a video from their HDD and the desktop client would generate different versions, upload them to IPFS and load the addresses to a PeerTube instance.

It seems like a big amount of work but promising to me and the idea could get many contributors.

rigelk commented 6 years ago

@alxlg we're not even close to writing a Desktop client. This is a non-option considered our resources.

I was considering leaving the video uploaded without transcoding, thus leaving a single quality available. It's always better than no video at all.

But now that I come to think of it, about ¹: do we even have to replace the webtorrent client for ipfs videos? If we could manage to mark ipfs endpoints as WebSeed, we could just use them under the hood of webtorrent by making the webtorrent client aware of them.

alxlg commented 6 years ago

@rigelk in fact I did not intend to replace the video player. I thought that the server could run both PeerTube and a IPFS node, and the PeerTube instance see the file cached with IPFS like local files... I hope it has sense now...

This feature doesn't depend on a desktop client, but it would just help normal PeerTube users to automatically store their videos locally.

I would be happy to store some videos with IPFS on my HDD without a PeerTube instance that need much more maintenance.

I think the change would mostly be in PeerTube UI providing a way to upload a video through a IPFS address and not uploading the entire file to the PeerTube instance. Of course the server admin should configure it to run IPFS...

Openmedianetwork commented 6 years ago

Would not a webRTC torrent app running in the users PC do the same thing when she/he leave it seeding the video? This also allows user to create "seedboxs" using RSS auto downloading torrent apps. Easy simple/KISS way of "distributing" the video hosting.

https://github.com/Openmedianetwork/visionOntv/wiki/Seedbox

alxlg commented 6 years ago

@Openmedianetwork good point! I think your proposal is easier to implement but using IPFS too could have some advantages. For example, I'm pretty sure that if an instance of PeerTube is no longer available an user can reupload his/her IPFS videos on another instance just sharing IPFS addresses, much better than reuploading video files! Do you think this could be achieved with torrents/WebTorrents too? Would changes to PeerTube be needed? Maybe uploading a *.torrent file instead of a video file?

rigelk commented 6 years ago

@alxlg using an IPFS address or a *.torrent file yields the same import capabilities. See #102. The only advantage of IPFS I see is that there are pinning brokers. (For BitTorrent too? I didn't check)

Openmedianetwork commented 6 years ago

issues with testing webrtc http://hamishcampbell.com/index.php/2018/05/18/testing-peertube-webrtc-seedboxes/

rigelk commented 6 years ago

@Openmedianetwork this has nothing to do with IPFS. Please find a related issue and detail your problem there, not in a blog post.

Openmedianetwork commented 6 years ago

Sorry this was a update for alexig that my suggestion for webRTC torrent as an alternative for seeding dues not appear to work after actually testing. Will start a new thread after further of tests.

Chocobozzz commented 6 years ago

Closing this issue since we have no really use cases for IPFS for now.

NightA commented 6 years ago

@rigelk @Chocobozzz IPFS can serve as a backup for local storage and dedicated seeding pools, as it effectively transfers each newly added file to an entire pre-existing network of 300+ peers from the get-go. In such an instance, PeerTube might not necessarily even have to double as a WebRTC-based IPFS node but simply run along side a regular one (which in-itself can be optional), first to ensure the files initially propagate through the network and secondly to provide an optional method for retrieving them through a local gateway. However simply linking to an IPFS address should suffice if it would be possible to configure a PeerTube instance to use external public gateways for retrieval.

In a case where a PeerTube instance should go down with no pre-existing seeding pools in place, as long as the videos are still present on IPFS, it should be possible to retrieve them by simply following each video's address (that presumably was shared beforehand with other federated instances). This way each video will remain accessible and therefor could be later conventionally reseeded via a different instance.

As a by-product, if it'll be possible to authenticate each user's identity, perhaps it might also be possible to use this method for transferring channels between different PeerTube instances.

poperigby commented 4 years ago

@Chocobozzz shouldn't this be reopened? It seems what @NightA mentioned was a pretty good idea.

LoveIsGrief commented 4 years ago

I'm actually interested in implementing this, but I think a roadmap should be discussed.

What I propose:

Phase 1: Server uploads to IPFS and stores hashes of videos

This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used. That will allow mounting ipfs to a directory and designating all videos to be stored there / storing them in one place : IPFS.

I assume there's a json or a table in the db with the videos, where a field or column for the hashes of the video files can be added.

Phase 2: Import from IPFS

In this phase, the user will have the option to provide a hash that the server can download and process. Maybe it's possible to check the filetype before downloading it in order to save bandwidth, I dunno.

If phase 1 is done intelligently, hashes already present in the db will be rejected e.g fil

If phase 1 is done intelligently, hashes of videos already in the db can be rejected since that would create an unnecessary duplicate. But if another server uses the same hash with different properties (title, description) that might not be good. Up for discussion

Phase 3: Syncing with other instances and downloading hashes

This is separated from Phase 2 only if the code is in a separate area. Importing from IPFS might be a different procedure from receiving a video file another instance.
Since I don't know what data is sent over activity pub, I assume it's either the torrent or a link for a server-to-server API call, which include information about the video e.g title, description, resolutions and (of course) hashes.

Phase 4: (wishful thinking) ipfs:// links for videos

Users running their own IPFS nodes with IPFS companion could then stream using IPFS.
I haven't actually done a lot of research into this, so I don't know if it's possible. Maybe plugin for videojs would be necessary - I dunno.

End goal

Instances running IPFS nodes and using that to download and pin hashes, which would allow :

rigelk commented 4 years ago

Server uploads to IPFS and stores hashes of videos

This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used. That will allow mounting ipfs to a directory and designating all videos to be stored there / storing them in one place : IPFS.

Where is it uploaded exactly? A third party (a pinning service?)?

ipfs-fuse requires a the Go IPFS runtime alongside, so this will complexify the deployment.

Syncing with other instances and downloading hashes

This is trivially done by adding another Link object in the ActivityPub Video.url field.

Users running their own IPFS nodes with IPFS companion could then stream using IPFS.

What bothers me is that users are expected to have this extension and a running IPFS Go runtime.

I haven't actually done a lot of research into this, so I don't know if it's possible. Maybe plugin for videojs would be necessary - I dunno.

I haven't found any videojs plugin for ifps, or its companion extension.

Instances running IPFS nodes and using that to download and pin hashes, which would allow :

* greater resilience to takedowns or simple storage failure

* additional data sources since users could host the data too

* possibly less bandwidth consumption if Phase 4 actually is possible and is done
LoveIsGrief commented 4 years ago

Where is it uploaded exactly? A third party (a pinning service?)?

It's pinned locally and uploaded when somebody else requests it over IPFS. If someone does so over a third party, then they'll have it, but not pinned.

ipfs-fuse requires a the Go IPFS runtime alongside, so this will complexify the deployment.

That depends. It's very possible to do that in stages too:

What bothers me is that users are expected to have this extension and a running IPFS Go runtime.

This is not a proposal to force users to use IPFS, merely to give users the option. Right now they have the opportunity to use HTTP or webtorrent. This would merely be another one

I haven't found any videojs plugin for ifps, or its companion extension.

Yes, that would have to be developed if necessary. The companion provides the useful feature of redirecting URLs to IPFS instances :

Requests for IPFS-like paths (/ipfs/{cid} or /ipns/{peerid_or_host-with-dnslink}) >>are detected on any website.
If tested path is a valid IPFS address it gets redirected and loaded from a local gateway, e.g:

https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
→ http://127.0.0.1:8080/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR

I assume we use HLS for streaming, so serving a different .m3u8 playlist with /ipfs links would be the only work required.

Your final points are valid, but I don't see the harm in providing an additional option. In case you missed it, I am willing to implement this, so your workload would be reduced to code-reviews and handling future bugs (since no code is perfect).

Of course, if you are firmly against having it in the original code base, I'll investigate if a plugin can be written and if not, simply fork it.

rigelk commented 4 years ago

This is not a proposal to force users to use IPFS, merely to give users the option. Right now they have the opportunity to use HTTP or webtorrent. This would merely be another one more.

Your final points are valid, but I don't see the harm in providing an additional option.

:+1:

In case you missed it, I am willing to implement this, so your workload would be reduced to code-reviews and handling future bugs (since no code is perfect).

I am not sure we could handle the future bugs part, especially when dealing with technologies we don't use regularly. And since merging code means taking responsibility in its maintainance…

Of course, if you are firmly against having it in the original code base, I'll investigate if a plugin can be written and if not, simply fork it.

I would suggest waiting for @Chocobozzz to answer about that - regarding plugins, not everything can be changed via their API. Depending on how many and where changes to the codebase are required, the plugin API could be expanded to facilitate your changes.

Now that being said, implementing it directly in the codebase at first is not a bad idea. It is not time lost, as this will serve as a POC and help us understand the reach of the needed changes - and thus the potential changes to the plugin API.

NightA commented 4 years ago

Where is it uploaded exactly? A third party (a pinning service?)?

To add a bit more details to what @LoveIsGrief mentioned, when a file is added to IPFS it is given a UnixFX structure, cryptographically signed, hashed, registered as a content-identifier (CID) in an IPLD, broken into sets of Merkle-DAG blocks and then distributed peer-to-peer using a DHT.

* resilience can be achieved in IMHO simpler ways. Right now what makes a video's origin instance disappearance a problem is that the BitTorrent tracker is the instance. Sharing the video over DHT alongside would already solve the problem, without changing our infrastructure.

From what i understood, unlike the DHTs utilized with torrents, the IPFS network doesn't focus on seeding each file individually rather focuses on distributing the individual file-blocks themselves among many peers. In which case, a typical IPFS nodes doesn't really "seed" individual files, rather only temporarily caches blocks of various others and only stores complete sets of blocks for specific files when those are explicitly pinned. Otherwise, the cache gets garbage collected and erased after a specific time period that's configured on each individual node.

So unlike with a PeerTube instances that goes under with a specific torrent which happened to be seeded from one specific location (the instance itself), once the same file gets cached throughout enough IPFS nodes, it has this sort-of a grace-period for being retrieved. During that period it has the chance of being saved/pinned on another IPFS node or be imported unto another hypothetical PeerTube instance that supports retrieval from IPFS.

The IPFS node in this regard also doesn't have to have anything to do with PeerTube instances to begin with, as it simply provides the files as long as there's someone requesting them.

TL;DR - Torrent DHT's only replace a tracker and in that regard only point to files that may or may not be seeded anymore. IPFS provides a P2P CDN of sorts that can cache those files independently of their initial seeding PeerTube instance, thus preserves them over a pool that operates independently and is not restricted to any particular instance and/or file. 

* users can already host the data, with a webtorrent-compatible torrent client.

In practice users who just consume content tend not to do so, unlike IPFS nodes who do so from the get-go upon request. E.g, while the initial "seed" has to come from an IPFS node running along with a PeerTube instance, once the file had propagated through multiple requests, it can be retrieved from other non-associated nodes within a given time-span.

That being said, yes, to ensure the file doesn't disappear from the network there has to be an IPFS node somewhere that pinns some of the content from the aforementioned PeerTube instance, which is a concept that sounds similar to a basic seedbox. However considering the pre-existing given network of peers that can automatically participate, this gives more chances in terms of availability for those files.

* less bandwidth consumption is already achieved with WebTorrent: people watching share their bandwidth (and not just those with an extension)

With IPFS there are also dedicated nodes that run on Independent servers/VPS's who distribute the content, in addition to users who just seed some of it for the duration of its run and then move along.

So essentially if a video can be streamed from each public IPFS gateway within a pre-defined list, all the PeerTube instance has to do to offload traffic is just pick and point to one of those IPFS "edge" gateways and serve the page to the user as usual.

ivan386 commented 4 years ago

This phase has the potential of requiring double the disk space since the files will be stored on disk normally, then uploaded to IPFS and pinned. In order to prevent that, ipfs-fuse could be used.

You can just use --nocopy option on add. And --raw-leaves if you want to store it only in ipfs and have same hash as with --nocopy.

ipfs add --nocopy file_name
ipfs add --nocopy -r directory_name

Additionally if you use --chunker=rabin different files will share same parts in it.

ipfs add --nocopy --chunker=rabin file_name
ipfs add --nocopy -r --chunker=rabin directory_name

In torrent file can be added local webseed http://127.0.0.1:8080/ipfs/{cid(hash) of file or directory}. Torrent clients will use it if it will be available.

Webtorrent can fetch public gateways and replase ":hash" with {cid(hash) of file or directory} from local webseed link. And use that gateways as alternative webseeds.

You can add a comment to torrent file with options that used when file added in ipfs. And user than can re add it to ipfs with this options and get same root hash.

ghost commented 4 years ago

Any updates on this issue? IPFS is very important because it's an efficient tool against censorship.

sundowndev commented 4 years ago

I don't think IPFS support can be added to Peertube that easy. My suggestion would be to create a different client-side intended to use IPFS network-based backend. Adding IPFS to Peertube as it would add several compatibility issues and increase code complexity. Also may be a PoC about IPFS video streaming with benchmarking should be done first?

ivan386 commented 4 years ago

@sundowndev Simple player: https://github.com/ivan386/ipfs-online-player Example of how it works: https://gateway.ipfs.io/ipfs/QmX8DUfyL7sVSa61Hwvx4qiTHPcpWrNxvb3XSm5iiqH8Ph/#/ipfs/QmdtE78NHJGByBpoPREMQA142oj9hFPmQRxMniDsbdhw5d

delbonis commented 4 years ago

Can someone explain to me why it makes sense to store videos on IPFS beyond being able to upload/publish them by IPFS address? You're backing a distributed video storage system with another distributed storage system. IPFS works great with small files that people are likely to rehost within some other system, but in PeerTube it just duplicates the necessity of clients sharing video over webtorrent to other peers. It could help simplify the process of sharing videos with other instances but webtorrent already does that. There's some sense to using a Sia host as a backend for storing data within an instance, as an alternative to just the filesystem or S3, but there isn't much advantage (at least not enough to justify the increased complexity) for something as involed as IPFS.

ivan386 commented 4 years ago

At now time all public IPFS gateways can be auto webseeds for webtorrent. Public Gateways will be cache proxy between IPFS peers and Webtorrent peers.

IPFS is better than webtorrent. You can rename file, chage it container and cut part of video without reencoding and IPFS will share unchanged chunks with original file.

Example: I changed container from mp4 to mkv.

ffmpeg -i video.mp4 -c copy video.mkv

And added both files to IPFS with chunker rabin

ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkv

And they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.

alxlg commented 4 years ago

@delbonis you can't seed WebTorrent without a Web browser and it's impossible to do so with PeerTube i.e. you can't take a list of PeerTube URLs, download the videos and tell to your PC to seed them for other PeerTube users.

Recently after years of work libtorrent added WebTorrent support so maybe we will see torrent clients supporting it too. Then PeerTube will need some kind of "seed this video from you torrent client of choice" button.

thomask-gh commented 4 years ago

There's some sense to using a Sia host as a backend for storing data within an instance, as an alternative to just the filesystem or S3, but there isn't much advantage (at least not enough to justify the increased complexity) for something as involed as IPFS.

In that perspective, Tardigrade, a decentralized cloud storage platform, could also be a good option to serve as an S3-like storage bucket. Not the same as IPFS though, but still interesting, maybe to implement via a plugin

I'm just mentioning it so that you know this decentralized cloud platform exists and might be interesting 🙂

manalejandro commented 3 years ago

Hi, you can upload from browser directly with js-ipfs, we have a PoC in ipfs-upload

Regards :up:

ghost commented 3 years ago

I've been experimenting with this, here's a player that kinda works on PeerTube's existing HLS files after importing them into ipfs with --nocopy. This requires a lot of extra CPU to do the hashing on import but it doesn't use much extra disk space.

https://github.com/scanlime/hls-ipfs-player

ghost commented 3 years ago

For what it's worth there is a ton of hype around IPFS and a ton of use cases that make no sense, so I should explain what brought me here:

I was interested in having a more unified data model between the "peer" part of peertube and the actual video storage, and for both of those parts to just generally work a lot better.

Drilling down into the technical details, a big decision is how the data gets hashed. In the current system, we use either WebTorrent (hash everything once the content is fully uploaded) or the novage p2p loader, which is very YOLO and trusts everyone. By contrast, IPFS hashes data in blocks that default to 256kB, max 1MB, which among other things lets processing and replication start before a video upload or stream is finished.

I was investigating architectures that let us keep a consistent data model between different parts of the network (clients, small servers, clustered servers, desktop apps, various caches) and IPFS came up. It's interesting because it solves the P2P problems we were solving on the client in broadly similar but specifically more competent ways.

manalejandro commented 3 years ago

I've been experimenting with this, here's a player that kinda works on PeerTube's existing HLS files after importing them into ipfs with --nocopy. This requires a lot of extra CPU to do the hashing on import but it doesn't use much extra disk space.

https://github.com/scanlime/hls-ipfs-player

Hi i was testing this https://github.com/moshisushi/hlsjs-ipfs-loader and made one hybrid version to do with p2p in the client too, that could be interesting to share two p2p platforms one in IPFS and another in the client, but this doesnt work yet i need some IPFS pending implementations in JS client version, best regards :ok_hand: hlsjs-ipns-p2p-loader.js.txt

ghost commented 3 years ago

I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them.

I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.

ghost commented 3 years ago

Another note on storage overheads: I have about 4.5 TB loaded in so far, and the additional disk usage is only about 8GB. But the memory usage has been consistently fairly high, about 8GB on this system. That's mostly for the DHT i suspect.

One idea that's been kicking around in my head is to build a media server (possibly based on this, https://gitlab.com/valeth/javelin) that can take live streams in/out as well as handling long term storage. One way to do the storage may be to have a very lightweight built-in IPFS server which would manage the hash database and communicating content with other servers as well as with clients that have native ipfs support. Clients would still get web video from the media server side of that daemon over whatever formats they need.

I'm still interested in finding a storage strategy for this which can unify the two formats we currently use (original webtorrent and HLS fmp4) as well as providing a path to introduce future formats without compounding the storage overhead further. The DAG data structure of IPFS may make it possible to do some kinds of container conversions in a way that references chunks of the original data rather than copying it? That's something I still need to experiment with.

ghost commented 3 years ago

And added both files to IPFS with chunker rabin

ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkv

And they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.

That sounds very impressive! Have you calculated the overall space savings though? That's such a small block size that I'd expect the parent nodes that point to all the chunks to themselves require a huge amount of space. Is it enough to make up for avoiding duplicating the video content?

I'd expect that, best case, rabin identifies all the best block boundaries and separates the audio/video streams from the container perfectly at each packet, now you have to store as metadata a reference tree with several hashes per packet.

I'd like to explore this still maybe, but I'm also interested in a way to teach some kind of ipfs-compatible-ish server, possibly either the real go-ipfs, or a light ipfs implementation that's part of a media server, to do remuxing or real-time transcoding to recreate blocks that don't exist. I'm wondering if this could work internally with a mechanism similar to how --nocopy is implemented, where the database stores pointers to filesystem data. In this case the database would store enough information to quickly remux or transcode a section of the input file. This wouldn't prevent having to do the transcode or remux once at first though, in order to determine what the hashes are. It would just be a way to garbage-collect resolutions that aren't used much and then regenerate them quickly as they're needed again.

sundowndev commented 3 years ago

FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)

manalejandro commented 3 years ago

I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them.

I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.

Because you have two p2p networks one behind IPFS it is like work internally, and another in the client level using p2p software to share the content, the plugin i have uploaded could connect two worlds, you are watching a video over IPFS and sharing the content, the advantage of IPFS vs WT is that torrents must be closed and IPFS can be opened to share live content for example. Regards

ivan386 commented 3 years ago

And they have 95% same blocks. And will be 95% source for each other. Each file size is 1.3 GB.

That sounds very impressive! Have you calculated the overall space savings though? That's such a small block size that I'd expect the parent nodes that point to all the chunks to themselves require a huge amount of space. Is it enough to make up for avoiding duplicating the video content?

@scanlime

My mistake. Files sizes is 1.6 GB and 76% of same blocks. 1.8% is metadata. 

ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mp4

Result:

Blocks size:     1 655 040 126 bytes -
File size:       1 624 999 455 bytes =
Metadata size:      30 040 671 bytes (1.8%)
ipfs add --nocopy --chunker=rabin-1024-2048-4096 video.mkv

Result:

Blocks size:     1 654 562 731 bytes -
File size:       1 624 528 837 bytes =
Metadata size:      30 033 894 bytes (1.8%)

I use this script (js-ipfs-same) to compare blocks tree.

Same data: 1 246 038 243 bytes (76%)

But for converting container from MP4 to HLS, the rabin will not work.

ghost commented 3 years ago

FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)

That's interesting, seems like a niche that many folks would appreciate being filled! One question though, you are striving for privacy, but IPFS is really antithetical to that goal by default as it's constantly broadcasting information about what content your node has locally. This seems fine for a public site like most PeerTube instances are, but it might be less appropriate for your use case here.

ghost commented 3 years ago

I'm not sure what you mean by "two p2p platforms" here? If you mean retaining compatibility with webtorrent or with the novage loader, that may be doable but I'm not sure the existing systems are working well enough that it's useful to retain compatibility with them. I'm using that same moshisushi/hlsjs-ipfs-loader project but I forked it to fix some bugs, that's probably what you're running into.

Because you have two p2p networks one behind IPFS it is like work internally, and another in the client level using p2p software to share the content, the plugin i have uploaded could connect two worlds, you are watching a video over IPFS and sharing the content, the advantage of IPFS vs WT is that torrents must be closed and IPFS can be opened to share live content for example. Regards

Ok. I think I get what you're saying. To me the interesting part of using IPFS for this is that you technically would not have two different networks, you would have one network (with different tuning and capabilities for sure) but with the same data model on each side.

I think there's a lot of architectural fuzziness to get lost in here but I find it helpful to think concretely about how the data is being identified and authenticated. With IPFS there's a consistent way to identify and check data across the network. With PeerTube as it stands now, we use bittorrent hashes in part of the system and no hashing at all elsewhere.

Also keep in mind that browsers have a very limited ability to help share video data to others. It might be useful for networks where many clients are sharing data with each other locally through a limited internet connection. But for most PeerTube-like setups I'd expect that the benefit of having better p2p file sharing is in making it easier for loose federations of server operators to mirror each other's data in a way that increases reliability rather than decreasing it :)

ghost commented 3 years ago

In the current system, we use either WebTorrent (hash everything once the content is fully uploaded) or the novage p2p loader, which is very YOLO and trusts everyone.

My mistake, I did not notice that we actually have another hashing system that's used by the HLS P2P loader. That one uses SHA256 hashes computed on entire segments. So, not really compatible with either other system directly, but it wouldn't be a big deal to retain compatibility with the system.

My main issue with the novage p2p loader is that I haven't come across very many situations where it's actually useful. It might help with individual extremely popular videos, but does anyone have those right now? Does it serve blocks fast enough for clients to see any benefit?

PeerTube's redundancy system is almost great... it only works with trusted peers right now though. A single slow or malicious redundancy peer will ruin the client's experience. That's the problem I'm interested in potentially solving with IPFS: having a good way to open the floodgates to crowdsourced data redundancy in a way that actually works. Browsers are really bad at this, they might be able to help a bit but the core of the network really needs to be servers that link together if we will have p2p that fits users expectations for loading video.

alxlg commented 3 years ago

@scanlime

But for most PeerTube-like setups I'd expect that the benefit of having better p2p file sharing is in making it easier for loose federations of server operators to mirror each other's data in a way that increases reliability rather than decreasing it :)

This is exactly what I mean when opening the issue, thank you and all the others for the very interesting replies so far!

sundowndev commented 3 years ago

FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)

That's interesting, seems like a niche that many folks would appreciate being filled! One question though, you are striving for privacy, but IPFS is really antithetical to that goal by default as it's constantly broadcasting information about what content your node has locally. This seems fine for a public site like most PeerTube instances are, but it might be less appropriate for your use case here.

@scanlime What I mean by privacy is analytic and user tracking. IPFS is just a storage, so if you want to keep your storage private, you can create your own IPFS Swarm with a set of peers. From what I've understood, you don't even have to isolate the pinset behind a private network, you just have to configure a trusted peers list. You can then create a gateway to control access to the cluster content. Is that answering your question ? I'm new to IPFS by the way but from what I've read in the docs, IPFS works fine for both cases.

ghost commented 3 years ago

@scanlime What I mean by privacy is analytic and user tracking. IPFS is just a storage, so if you want to keep your storage private, you can create your own IPFS Swarm with a set of peers. From what I've understood, you don't even have to isolate the pinset behind a private network, you just have to configure a trusted peers list. You can then create a gateway to control access to the cluster content. Is that answering your question ? I'm new to IPFS by the way but from what I've read in the docs, IPFS works fine for both cases.

I think you're misunderstanding what ipfs-cluster is for.. it doesn't change who the individual IPFS nodes contact, it's an additional layer on top of ipfs so that groups of servers can agree to reliably serve a set of content together. That's all about reliability, not about privacy.

You have to understand that IPFS (and all p2p protocols I know about) are quite promiscuous, they will just ask anyone they're connected to for the data they need, or broadcast the data they have to random people. To retain privacy, you would really need to have a completely closed network, in which case there isn't much point to the protocol.

This seems fine for PeerTube and is roughly equivalent to our existng level of privacy here. But I wouldn't use IPFS as a basis for creating a project that has privacy as its focus.

sundowndev commented 3 years ago

@scanlime Alright thanks for taking the time to explain that to me. I'll look closer at the design of IPFS and its use-cases. On a side note, what I can do, is simply put IPFS as a storage option in my project. You could then choose between local storage, IPFS or whatever (S3, SFTP...).

I should stop discuss about that there since it's off-topic, but feel free to create an issue in dreamvo/gilfoyle to discuss further about it :)

ghost commented 3 years ago

I'm still experimenting with IPFS. There's a lot to like, but the RAM usage is a real drag, and might be the main reason not to use it in most PeerTube-style video sites. Each IPFS node needs enough memory to quickly invert hashes into disk blocks, and... on my ~9TB dataset the ipfs daemon wants to use around 11GB of RAM even with few peers. The actual hashes are large but i'm not sure how to explain the bulk of that usage. A 32 byte hash for each 256kB block is 128MB per terabyte. (Edited: math was off by 1024x, oops)

ghost commented 3 years ago

FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)

Will it somehow relate to peertube? Maybe federate with peertube?

sundowndev commented 3 years ago

FYI I've started a whole new project backed by IPFS but with different design and goals than PeerTube. Also it targets not only video but also audio content. It's written in Go and depend on FFmpeg. Feel free to give feedback about the design! (although this is still a WIP for now)

Will it somehow relate to peertube? Maybe federate with peertube?

I don't think so as it's not designed to work with Peertube or other P2P platforms. But feel free to discuss about the design in the repository.

ghost commented 3 years ago

It's not a real project yet but I've still been experimenting with an IPFS-backed video server that would be compatible with PeerTube federation. There are a lot of performance hurdles to get over but the large amount of existing tooling for sharing hash-addressed blocks still feels compelling to me.

sundowndev commented 3 years ago

I think you're misunderstanding what ipfs-cluster is for.. it doesn't change who the individual IPFS nodes contact, it's an additional layer on top of ipfs so that groups of servers can agree to reliably serve a set of content together. That's all about reliability, not about privacy.

@scanlime I've seen this repository showing how to setup a private IPFS swarm to host private files. Apparently this is possible by removing the default bootstrap nodes and isolating your peers inside a private network. See also https://github.com/ipfs-inactive/faq/issues/4 where the creator of IPFS himself says that this is part of their goals. So I guess choosing IPFS for privacy usage is possible and not an anti-feature.

ghost commented 3 years ago

See also ipfs-inactive/faq#4 where the creator of IPFS himself says that this is part of their goals. So I guess choosing IPFS for privacy usage is possible and not an anti-feature.

That is 5 years old, there is a much better way now, you can create private IPFS networks by restricting the p2p connection with a pre-shared key. This is only really practical for situations where you are using ipfs in a data center or in a corporate environment, I don't see how that would apply to something like PeerTube intended to be an open network.

If you are using ipfs and you care about privacy you have to understand it at least. Everyone your node connects with can easily find out what content you have and what content you're looking for. Like most P2P systems.

sundowndev commented 3 years ago

Everyone your node connects with can easily find out what content you have and what content you're looking for. Like most P2P systems.

Yes, that's the point of a swarm, this shouldn't be a problem if you control each nodes. But from what I see, using IPFS with a private network is an experimental feature that's not yet supported but which is not against the IPFS's design & goals. But you're right, this is not a feature that PeerTube would need.