Question: Ready recipe for decentralized file storage?

[master255]

@arvidn For example: I have 1000 torrent clients. These clients want to have a shared directory of files. Each client can add and remove files (only his own). Each client is willing to keep a copy of this directory on his computer. I have already figured out approximately how to do this with putDHT and getDHT.

But I would like to hear your version. What do you think about it? Perhaps there are ready-made solutions?

Ipfs, OrbitDB, blockchain - no need to suggest.

[arvidn]

I'm not familiar with an existing system like that built on bittorrent. I experimented with something along those lines in uTorrent many years ago. One feature that came out of that was the ability to distribute encrypted torrents, where a client that has the secret key decrypts/encrypts at the disk layer, so it's stored in plain text. Any client that doesn't have a key will simply download, and seed, a torrent of garbled data.

[master255]

@arvidn Ok. I think you need to develop torrents in that direction. I'll write if I was able to do it with put and get.

[master255]

@arvidn Good news. I have developed an algorithm for a decentralized torrent storage (even with the ability to moderate content and distribute access.) I wrote this algorithm. It already works! I don't understand why no one uses it yet.

[master255]

@arvidn The release of decentralized storage is ready in version 8.0 of my player! We may not know such systems, but they exist. My player now uses this system to store shared playlists. I even added an administration screen that I can use to block users or forcibly delete their files.

I rate the complexity of the system as medium. For example, it is easier than multi-threaded streaming, which has 3D logic.

[master255]

@arvidn @markmdscott @aldenml @gubatron @glassez @proninyaroslav I published an article on how to make a decentralized Torrent storage or even a decentralized database. In two languages: ENG RU

I remind that everything written in the article really works and already used in my player for a long time. For publishing playlists. But it can be used to create billing systems, Torrent Index, slow sites, modifiable torrents, to create a shared cloud storage of any files with the ability to specify quotas and speed limits and much more...

[master255]

@arvidn @markmdscott @aldenml @gubatron @glassez @proninyaroslav I invented two more scenarios to use DHT. Updated the articles. Read them again. Added variant 2 and 3. It's beautiful, but I don't have time to develop it.

[gubatron]

@master255 have you used Resilio (Formerly known as BTSync)? I wonder if you were inspired by it.

I've certainly thought of several scenarios on how to create something inspired by Resilio where you'd have a public directories, users can access them because the folder public keys are known by all the peers in advance.

I imagine cryptographic tools today could allow for deriving shared private keys so you can add/remove content managers that would be allowed to edit folder contents and to republish the torrents representing such data. All clients willing to sync those folders would automatically sync/seed the latest versions of the folders.

If one can create one level of abstraction above the inner workings of DHT put/get and create protocols based on canonical folder structures you could do even a distributed torrent search index and do away with torrent search indexing websites.

The problem that I have with this design is always spam. Which then I go on to think about distributed reputation, and instead of one giant index, have instead a bunch of DHT-based RSS-like feeds that can be used by the clients to create local torrent databases. You solve spam by following the more reputable DHT-based RSS feeds.

[master255]

@gubatron

have you used Resilio (Formerly known as BTSync)?

No. Of course I've never used this program and didn't even know it existed.

If one can create one level of abstraction above the inner workings of DHT put/get and create protocols based on canonical folder structures you could do even a distributed torrent search index and do away with torrent search indexing websites.

For almost a year now, my player has had a shared playlist system. In fact, it is a torrent index. The work of this system is based on a decentralized database that works with DHT and torrents. It can be hacked, but it is quite difficult (you need to disassemble the program and find the private key in it) and will not damage the database. Since it is a distributed database, it has, even access rights. I have an administrative control panel for this system. I can block users by id.

In the article above I described scenarios to use DHT which cannot be hacked. As in blockchain there appear two types of node. Users - who use the decentralized database, and managers - who handle changes in the database. From this you get a managed but decentralized database. This can be used for anything. For a torrent index, for billing, for distributed file storage, for a decentralized Internet.

If you have time for that, I can suggest answers to all the questions about creating such a system. I don't have time to program it. Besides in my country (Russia) DHT via 4G is blocked. And without that there is no point in programming anything. No one will be able to use it.

Also, my player has had editable torrents for a long time. You can create a torrent, publish it to DHT. Send a magnet link to a friend. He will download the torrent. Then you can modify the torrent, update it in DHT. Your friend can download the new torrent from the old link.

I offered to add this functionality to Arvin, but as I understood he didn't like it very much.

[FranciscoPombal]

@master255

(you need to disassemble the program and find the private key in it)

That is not as hard as you think it is for any even slightly motivated attacker with the help of modern decompilers/disassemblers that have heuristics to detect common crypto primitives and secrets.

Posting such information in a public forum isn't doing your app's security any favors either, you are basically "inviting" motivated individuals who come across this post to "take on the challenge" (although security by obscurity, which is what you were relying on up to this point, isn't any better).

[master255]

@FranciscoPombal I think you need to read my article again https://habr.com/ru/post/546612/ I've already updated it. Read variant 2 and variant 3. These variants are impossible to hack.

[FranciscoPombal]

@master255

Yes, I have read it. You describe 3 different systems, but in that article you do not make clear which one(s) your app implements exactly.

You simply sate at the end of it:

A similar system is used in my "Media Library" app, for publishing playlists. I already have even an admin interface for moderation. Everything works successfully. Enjoy it!

One would hope that this refers to the 3rd described system in the article, but it is actually not 100% clear from the presentation/structure of the article.

Then in https://github.com/arvidn/libtorrent/issues/5959#issuecomment-926924975 you stated (emphasis added):

For almost a year now, my player has had a shared playlist system. In fact, it is a torrent index. The work of this system is based on a decentralized database that works with DHT and torrents. It can be hacked, but it is quite difficult (you need to disassemble the program and find the private key in it) and will not damage the database.

Which implies that what you actually implemented in your app is the 1st variant described in the article, not the 2nd or 3rd.

You then go on to summarize the 2nd and 3rd systems from the article in your comment...

In the article above I described scenarios to use DHT which cannot be hacked. As in blockchain there appear two types of node. Users - who use the decentralized database, and managers - who handle changes in the database. From this you get a managed but decentralized database. This can be used for anything. For a torrent index, for billing, for distributed file storage, for a decentralized Internet.

...but then state that you won't implement such variants in your app for a few reasons:

If you have time for that, I can suggest answers to all the questions about creating such a system. I don't have time to program it. Besides in my country (Russia) DHT via 4G is blocked. And without that there is no point in programming anything. No one will be able to use it.

So my point stands - the current system your app implements is vulnerable to a motivated kid with a decompiler.

I've already updated it. Read variant 2 and variant 3. These variants are impossible to hack.

You've updated the article, but what about the app?

[master255]

@FranciscoPombal

So my point stands - the current system your app implements is vulnerable to a motivated kid with a decompiler.

It's not as easy as it sounds. To decompile my player is a very difficult task. And after that, it's even harder to find any keys there. In addition, it is not known in what form the keys are stored. But you won't find them even in their simplest form. That I have checked. And even if you hack my playlist system, you won't get anything. In the next update, which will come out an hour after the hack, I'll just change the keys and the storage method and all users who update will automatically republish the playlists. The system will recover automatically.

You've updated the article, but what about the app?

My player only uses the 1 variant from the article and the editable torrents via magnet links. This is enough for the player. There is no point in embedding other variants into it. And as I said before, I don't have time for it. You or other programmers can try to do something similar. I have given you a hint of a plan of action.

[FranciscoPombal]

@master255 OK, best of luck.

[master255]

I invented a new kind of decentralized database. This is DHT over DHT. Right now there is no time to describe it. But there is nothing complicated there. It just needs bootstrap nodes.

[master255]

@arvidn @markmdscott @aldenml @gubatron @glassez @proninyaroslav The world's first DHT over DHT application. DHT Market

[master255]

Market already supports publishing editable torrents, torrent files, apk links to mega.nz, Apk files.