search

buckyos / CYFS

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP),is short for CYberFileSystem. https://www.cyfs.com/, cyfs://cyfs/index_en.html.
https://www.cyfs.com/
BSD 2-Clause "Simplified" License
2k stars 276 forks source link
blockchain cryptography named-data-networking named-object-networking ndn non p2p protocol rust web3

readme

CYFS Logo cyfs-lib-ver rust liscence Downloads

CYFS：Next Generation Protocol Family to Build Web3

Introduce

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP). It has a subversive architectural design that everyone brings their own OOD (Owner Online Device) to form a truly decentralized network.

We define Web3 as follows:

After 6 years of research and development, we have achieved the above goals on the CYFS testnet. For the detailed design of the theory and practice of CYFS, please refer to the relevant documents. Open source of CYFS is an important milestone for CYFS, marking that its design has been basically stable and has entered an open community-driven R&D stage. We look forward to having more people join us. We need your passion and strength. Let’s build Web3 that belongs to ourselves together.

This is the repository of the CYFS protocol stack on Github. Although we have built a decentralized GitHub (CodeDAO) on the CYFS testnet to realize the decentralized bootstrapping of CYFS. But before the CYFS mainnet online, we still decided to stay a little longer in WEB2.0 and use this repository as the main community-driven R&D base. After the CYFS mainnet goes online, the entire Repo will move to CodeDAO.

This repository only contains the implementation of the CYFS core system. It does not include the important basic software of the CYFS ecosystem: CyberChat and CYFS Browser, nor does it include SDK implementations for developers of different languages. The following is a list of other important open source projects in the CYFS ecosystem (continuously updated)

Background

CYFS Architecture

The above is the overall architecture diagram of CYFS, which can help to establish a macro understanding of the design of CYFS. The core design of CYFS can be decomposed into the following parts

What key problems are these designs designed to solve? (We believe that "finding the right problem is half done~") You can read "CYFS Architecture" and "CYFS Whitepaper" (coming soon), the amount of content comparison Great, you can understand it with practice~

Start your Web3 journey with “Hello CYFS”

We strongly recommend that you read the "Hello CYFS" series of articles in full to get a complete initial experience of CYFS (the whole process can be completed within 1 hour):

  1. "Compile and deploy DIY-OOD"
  2. "Create Your DID"
  3. "Publish File and Download"
  4. "Tips your friend"
  5. "Publish Website and View"

You can also use the following fast progress to have a first experience of "doing nothing without understand and save time"~

Advance CYFS DEC App develpe tutorial is online

"Decentralized message board development"

Quick Start

The following will build your own Web3 environment by compiling the CYFS source code, and complete the basic experience of Web3 in this environment:

Note that this environment is normally not isolated, but can be added to the CYFS network (we have planned 3 networks: nightly, beta, relase, currently only nightly is available), so the following tutorial does not include compiling MetaChain with source code +SN part. If you plan to build a complete system in an independent environment and need to compile and deploy MetaChain super nodes, please read the document "MetaChain Construction" (Coming soon).

Compile

Prepare an idle Linux host, preferably a VPS, with more than 2G of memory. Preparation:

Run the following command to compile the OOD System.

cd ${cyfs_sources_dir}/src
npm i
node ../scripts/build-standalone.js

During the compilation process of the script, the ${cyfs_root} folder needs to be created. If the current system or current user does not have permission to create this folder, you need to manually create the ${cyfs_root} folder in advance, and assign read and write permissions to the current user. (In the official environment, all components and data of the OOD System will be installed to ${cyfs_root})

${cyfs_root} path:

Compile successfully generates ood-installer

Install the newly compiled OOD System

Before installation, you need to prepare the depends:

Find the newly compiled ood-installer in the src directory and execute

./ood-installer --target solo

After a minintes , the installation is complete.

Activate OOD

In order to experience the process, the cli tool is used here to complete the D.I.D creation process. D.I.D created based on the cli tool can only be used for testing purposes!

  1. Install cyfs-tool: use the command line npm i -g cyfs-tool-nightly to install the nightly version of the cyfs-tool tool
  2. Generating sets of identities

    Use the command cyfs desc -s <save_path> to generate a matching identity file and save it in the save_path directory. If save_path is not specified, it defaults to ~/.cyfs_profile

  3. Bind OOD

    After the identity is generated, copy the two files <save_path>/ood.desc and <save_path>/ood.sec to ${cyfs_root}/etc/desc on the OOD machine and rename it to device. desc and device.sec

  4. Bind CYFS-Runtime

    After the identity is generated, copy the two files <save_path>/runtime.desc and <save_path>/runtime.sec to ${cyfs_runtime_root}/etc/desc on the CYFS browser machine and rename them to device.desc and device.sec

${cyfs_runtime_root} specific path:

Publish your first Web3 website

First prepare the www directory of your website, let's first experience publishing static websites to Web3, and follow-up documents for the construction of dynamic websites will be introduced.

use command

cyfs upload <dir_path> -e ood -t ood

Add the file pointed to by the local to the OOD. The command execution is complete, the local www directory has been uploaded to OOD and the unique URL to the Web3.0 website has been generated (the end of the command execution). The link is cyfs O-Link, which looks like this cyfs://o/$ownerid/$objid, where $objid is the ContentId of the directory.

Browse the website just released

Use the command on any machine with cyfs-tool installed

cyfs get cyfs://o/$ownerid/$objid/index.html

You can download the just-released official website.

Any machine with a cyfs browser installed can use the cyfs browser to open cyfs://o/$ownerid/$objid/index.html and browse the website just released. For the download of cyfs browser, see here

Code guide

Through the above process, you have a basic understanding of the design and use of CYFS. Although the design of CYFS is basically stable, we still have a lot of code to write. We are very eager for your help, but certainly not too much energy to write documentation (details are in the source code~). Here we do a minimalist code introduction, hoping to help you understand the implementation of CYFS faster.

According to the architecture, we know that the core of cyfs:// is the construction and acquisition of Object Linke, and the premise of acquisition is to upload data to OOD at least. The process is as follows:

  1. Start the local protocol stack (cyfs-rutnime)
  2. Calculate the hash of all files in the directory locally, and construct a FileObject with the current PeopleId as the Owner
  3. Generate a Map structure (directory structure) locally, and construct a MapObject with the current PeopleId as the Owner, At this time, cyfs:// has been constructed, but the cyfs:// cannot be accessed at this time
  4. Add the above named objects and named data to the local stack
  5. Initiate CYFS PUT operation to OOD: save the MapObject to OOD and set the access permission to public
  6. Let OOD start MapObject Prepare and save a named data on OOD
  7. MapObject Prepare is completed on OOD, cyfs:// can be accessed

Then the process of using cyfs get to obtain is as follows:

  1. Start the local protocol stack cyfs-runtime
  2. Use the HTTP protocol to initiate an HTTP GET request to cyfs-runtime
  3. cyfs-runtime checks whether the object exists in the local cache
  4. cyfs-runtime initiates a NamedObject query request (the following behaviors are not serial)
    • 4.1 Query NamedObject from OOD
    • 4.2 OOD query local, whether NamedObject exists
    • 4.3 OOD queries MetaChain, whether NamedObject exists
    • 4.4 OOD queries whether the NamedObject exists on the previous hop device according to the Reference information in get
    • 4.5 OOD queries the configuration of Object's Owner Zone through MetaChain
    • 4.6 OOD is configured through Zone, connected to NamedObject's OOD, or connected to NamedObject' Cache, and query NamedObject
  5. After getting the ChunkId, cyfs-runtime calls the Channel interface (NDN semantic interface) of the BDT to request the Chunk
    • 5.1 For the first, small Chunk, get it directly from the associated OOD
    • 5.2 For the second Chunk, it will try to get it from the previous hop (Reference OOD)
    • 5.3 BDT will try to perform multi-source search and multi-source download based on fountain code based on the context information of the application layer
    • 5.4 The router can identify the Chunk request packets sent by the BDT, intercept and forward them, and further optimize the overall load of the network
    • 5.5 Only OOD will upload Chunk
  6. When the first Chunk of the FileObject is ready (passed authentication), the HTTP GET request in step 1 starts to return data

(The above process is shown in the figure below) get-from-cyfs

For the overall design of the CYFS protocol, please refer to Introduction to CYFS Protocol.md

After understanding the logic of the above process, you can read the relevant code according to the following guidelines.

Upload

  1. Start the local protocol stack: util.ts: create_stack()
  2. Construct FileObject: file_recorder.rs: FileRecorder.add_file()
  3. Construct ObjectMap: publish_manager.rs: PublishLocalDirTask.publish()
  4. Add the above Named-Objects and Named-Data to the local protocol stack: file_recorder.rs: FileRecorder.record_file_chunk_list()
  5. Initiate a CYFS PUT operation to the OOD: save the MapObject to the OOD and set the access permission to public: upload.ts: upload_obj()
  6. Let OOD start MapObject Prepare and save Named-data(Chunks) on OOD: upload.ts: run()

Get

  1. Start the local protocol stack: util.ts: create_stack()
  2. Initiate an HTTP GET request using the HTTP protocol
  3. cyfs-runtime checks whether the object exists in the local cache
  4. cyfs-runtime initiates NamedObject query requirements (the following behaviors are usually not serial)
    • 4.1 Query NamedObject from OOD
    • 4.2 OOD query local, whether NamedObject exists
    • 4.3 OOD queries MetaChain, whether NamedObject exists
    • 4.4 OOD queries whether the NamedObject exists on the previous hop device according to the Reference information in GET
    • 4.5 OOD queries the configuration of Object's Owner Zone through MetaChain
    • 4.6 OOD is configured through Zone, connected to NamedObject's OOD, or connected to NamedObject' Cache, and query NamedObject
  5. After get the ChunkId, cyfs-runtime calls the Channel interface (NDN semantic interface) of the BDT to request the Chunk
    • 5.1 For the first and small Chunk, get it directly from the associated OOD
    • 5.2 For the second Chunk, it will try to get it from the previous-jump (Reference OOD)
    • 5.3 BDT will try to perform multi-source search and multi-source download based on fountain code based on the context information of the application layer
    • 5.4 The router can identify the Chunk request packets sent by the BDT, intercept and forward them, and further optimize the overall load of the network
    • 5.5 Only OOD will upload Chunk
  6. When the first chunk of the FileObject is ready and verified, the HTTP GET request in step 1 starts to return data

DecApp development practice

The following series of articles describes how to implement a message board DecApp based on CYFS. this is a long series of articles detailing the general process of developing a DecApp

  1. Introduction
  2. DEC App basic principle
  3. Implement your own message service
  4. Run and debug your own message service
  5. Extended reading: Further explanation of the principles and introduction of some tools
  6. Implement the front end
  7. Send it to your friends and understand the CYFS permission system
  8. Use the emulator to test the app across zones
  9. Extended reading: Decentralized, trusted data, semantic data, data property rights
  10. Add comments
  11. DEC App data security and high availability
  12. Extended reading: Learn more about data exchange contracts and DAOs
  13. Summary

Directory Structure

You have learned from the architecture diagram that the project implementation of CYFS is not small. To prevent you from getting lost, this chapter is a small map that will help you get a basic understanding of the CYFS code structure and understand where the key code will be.

Roadmap

CYFS is currently in the state of dev_testnet (nightly), in this state, all components will be affected by the daily build results by default, and we do not consider how the data structure is backward compatible, and the protection of data security is not enough. whole. This version should only be used for development testing and should not be used in production production.

Our next goal is to complete the launch of the testnet (beta). Relatively nightly, beta will specify strict release standards, and will make the data structure as backward compatible as possible, and can confidently protect users' data security. The stability of beta will be able to support the release of beta products~ The main difference from the release version is assets is fake assets, and there are gaps in stability and performance.

We hope to launch the CYFS mainnet within 18 months. There is no definite launch time for the CYFS mainnet. As long as the asset security, stability and performance of the testnet reach the preset goals, we will enter the release channel of the mainnet.

Welcome to follow CYFS-Stack & Services to learn more about CYFS Roadmap~

Contributing

The dev_testnet of CYFS is designed and implemented by Shenzhen BuckyCloud. The core team of Shenzhen BuckyCloud is inherited from the original Xunlei(NASQ:XNET) infrastructure R&D team. We have rich experience in network protocol and infrastructure development, as well as super-large-scale P2P network design and tuning. experience. After CYFS is open source, BuckyCloud has completed its key historical mission. Now we prefer to call ourselves CYFS CoreDevTeam. In the future, it will be an open team organized based on DAO. The main responsibility is to promote the continuous research and development of CYFS. We welcome all engineers to join:

We are designing a DAO-NFT-based contribution record system. When this Repo is move to CodeDAO as a whole, CYFS DAO Tokens can be issued according to the share size of contributors (fairness is difficult)

License

BSD-2-Clause License
Copyright (c) 2023, CYFS Core Dev Team.