search

Neos-Metaverse / NeosPublic

A public issue/wiki only repository for the NeosVR project
193 stars 9 forks source link

Use Owner-Free Filesystem for decentralized asset and world storage #3550

Open technobaboo opened 2 years ago

technobaboo commented 2 years ago

Is your feature request related to a problem? Please describe.

It would be ideal to move from Azure Cloud to a more distributed/decentralized storage model to reduce storage costs, avoid future censorship, spread out bandwidth among users, etc.

Relevant issues

No response

Describe the solution you'd like

I think the best decentralized storage method is https://en.wikipedia.org/wiki/OFFSystem personally, because the raw data is anonymized and all raw data is able to be sent p2p, not requiring intermediate nodes for blocks. This would mean you get anonymity with high download/upload rates. The Neos asset URLs would transfer over to OFF URLs very easily too.

Describe alternatives you've considered

IPFS is probably the biggest contender (see #1754) but it has several issues. The biggest 2 are:

  1. Unpopular files need to be "pinned" or they will fade over time due to nobody having them to reupload. OFF solves this by making all blocks multi-use, requiring higher up-front storage usage but over time exponentially less. Unpopular assets would be more likely to be accessible later on as more multi-use data blocks exist in the brightnet.
  2. Files in IPFS are not by default anonymous (https://github.com/ipfs/go-ipfs/issues/6430), as doing this would mean creating proxies that encrypt the data which still does not anonymize metadata. In contrast, OFF blocks are all anonymous garbage until you download 2 and XOR them. This would mean tracking users by the worlds they go to via their assets they download would be much harder if not impossible on the file storage side.

Additional context

No response

jeanahelver commented 2 years ago

Asset storage is part of the current business model. one of the big perks you gain from the patreon is additional storage. If you require the assets to be off the Asure Cloud you can request this with the neos pro license for a "local asset server". But this seems to only for "large scale Neos users such as univerities and companies"-Karel

ZarosTenjin commented 2 years ago

Theres several problems i see with using a distributed filesystem: 1) This requires the resevation of space on the users harddrive besides the already big and always growing cache. We already have trouble with people not noticing how big their cache is getting and suddenly their C: Drive having 0 Bytes left just crashing their System. This would make this worse. 2) I dont see how anonymity of the metadata is required. Especially if the host of the data right now is a corporate run cloud environment. 3) This will make it extremely hard to deal with copyright related matters in the future and this is not taking into account potentially distributing illegal material across computers. this is already a problem with cache, but at least for that you need to be in the session for it to be distributed to you. if its just general storage you suddenly have DMCA material on your computer without you having an intend to do just that 4) Switching to P2P "filesharing" via a distributed storage solution has the downside of also causing additional traffic that is out of their control. To users that might be on a metered connection that is an additional burden that is afaik unneccesary. 5) The burden of "censorship" wouldnt really be solved either since on non compliance it is now private entitites that get targeted instead of a megacorp. if censorship is enforced by government saying oops sorry i have no control is no reason for them to not enforce it anyway. 6) right now the userbase of Neos that is online at any given time is far far away from the critical mass needed to realize something like this. i.e: Conservatively assuming the cloudsize of neos would be 500TB right now and assuming its a nice active day with 250 people online this would require every single online user to donate 2TB of storage for full availability excluding redundancies. its just not feasable right now.

If theres logical disconnects i'm sorry im writing this at work between regular work. will review what i wrote once im home with a nice coffee