Fair Data Protocol Storage

FDP Storage is a serverless web3 filesystem for organizing users' personal data implemented in Typescript.

Such data is stored using certain structures that allow the data created in one dApp to be interpreted in another dApp. The current implementation allows to create and manage pods (similar to disks in file systems), directories, and files.

The library requires the API endpoint of a Bee node to interact with the data. If you plan to do write operations, you will need to specify postage batch id. To run a local test node trying out the functionalities, you can use FDP Play.

The FDP Storage user account is a wallet based on the BIP-44 mnemonic phrase from which one can create a portable account allowing retrieving the wallet from anywhere by providing a username and a password to the library.

The library can work in the browser, in Node.js and in mobile applications using React Native. There is an implementation of Personal Storage in Golang: https://github.com/fairDataSociety/fairOS-dfs

Project development plans and details of how each of the parts works can be found in FIPs. In this repository, you can create your proposal, which will be considered and taken into account in further development.

Warning: This project is in beta state. There might (and most probably will) be changes in the future to its API and working. Also, no guarantees can be made about its stability, efficiency, and security at this stage.

Install
Usage
Contribute
- Setup
- Test
- Compile code
License

Install

npm

npm install @fairdatasociety/fdp-storage

yarn

yarn add @fairdatasociety/fdp-storage

Use in Node.js and browser

We require Node.js's version of at least 16.x

const FDP = require('@fairdatasociety/fdp-storage');

Use in a browser using a script tag

Loading this module through a script tag will make the fdp object available in the global namespace.

<script src="https://unpkg.com/@fairdatasociety/fdp-storage/dist/index.browser.min.js"></script>

Use in React Native

FDP Storage is ready to work with React Native. But a few shims need to be added to the initialization script of your project to make the library components work.

import 'react-native-url-polyfill/auto' // for bee-js. URL polyfill
import 'react-native-get-random-values' // for ethers.js cryptography
import '@ethersproject/shims' // commong shims for ethers.js
import 'text-encoding' // for fdp-storage to make TextEncoding work

After creating instance of FdpStorage replace bee-js upload method to the new one because of bug #757.

const fdp = new FdpStorage('https://localhost:1633', batchId)

fdp.connection.bee.uploadData = async (batchId, data) => {
  return (await fetch(fdp.connection.bee.url + '/bytes', {
    method: 'POST',
    headers: {
      'swarm-postage-batch-id': batchId,
      'swarm-encrypt': true,
      'swarm-pin': true,
    },
    body: data
  })).json()
}

Usage

Creating FDP account

import { FdpStorage } from '@fairdatasociety/fdp-storage'

const batchId = 'GET_BATCH_ID_FROM_YOUR_NODE' // fill it with batch id from your Bee node
const fdp = new FdpStorage('http://localhost:1633', batchId)
const wallet = fdp.account.createWallet() // after creating a wallet, the user must top up its balance before registration
// Associate the created wallet with the username in the smart contract.
// This method makes the account portable.
// Seed is saved encrypted in Swarm.
// Registration is performed in two steps, first create a registration object
const registrationRequest = fdp.account.createRegistrationRequest('myusername', 'mypassword')
// Then pass this object to the register method
await fdp.account.register(registrationRequest)
// If registration fails, it can be safely retried by passing the same object again

// If necessary, the account can be re-uploaded to Swarm.
await fdp.personalStorage.reuploadPortableAccount('username', 'password')

const wallet = await fdp.account.login('otherusername', 'mypassword')
console.log(wallet) // prints downloaded and decrypted wallet

Creating and using a user without interacting with the blockchain

It is not necessary to register a user in a smart contract and make his wallet portable. You can create a wallet, save the mnemonic phrase locally and import this account to interact with all the data.

// Create a wallet for interacting with data.
// It does not need to be funded.
// Operations in the blockchain will not pass through it.
const wallet = fdp.account.createWallet()

// Get mnemonic phrase of the account.
// This is the key to all data in FDP Storage.
// You need to store in a safe place.
const mnemonic = wallet.mnemonic.phrase

// to access your account, you need to import the phrase
fdp.account.setAccountFromMnemonic(mnemonic)

Creating a pod

const pod = await fdp.personalStorage.create('my-new-pod')
console.log(pod) // prints info about created pod

Getting list of pods

const pods = await fdp.personalStorage.list()
console.log(pods.getPods()) // prints list of user's pods
console.log(pods.getSharedPods()) // prints list of pods that a user has added to their account

Sharing a pod

const shareReference = await fdp.personalStorage.share('my-new-pod')
console.log(shareReference) // prints share reference of a pod

Getting information about shared pod

await fdp.personalStorage.getSharedInfo(shareReference)

Saving shared pod under user's account

await fdp.personalStorage.saveShared(shareReference)

Creating a directory

await fdp.directory.create('my-new-pod', '/my-dir')

Deleting a directory

await fdp.directory.delete('my-new-pod', '/my-dir')

Upload an entire directory with files in Node.js

// `recursively: false` will upload only files in passed directory
// `recursively: true` will find all files recursively in nested directories and upload them to the network
await fdp.directory.upload('my-new-pod', '/Users/fdp/MY_LOCAL_DIRECTORY', { isRecursive: true })

Upload an entire directory with files in browser.

Create input element with webkitdirectory property. With this property entire directory can be chosen instead of a file.

<input type="file" id="upload-directory" webkitdirectory/>

// getting list of files in a directory
const files = document.getElementById('upload-directory').files
// `recursively: false` will upload only files in passed directory
// `recursively: true` will find all files recursively in nested directories and upload them to the network
await fdp.directory.upload('my-new-pod', files, { isRecursive: true })

Uploading data as a file into a pod

await fdp.file.uploadData('my-new-pod', '/my-dir/myfile.txt', 'Hello world!')

// you can also track the progress of data upload
// using the callback, you can track not only the progress of uploaded blocks but also other time-consuming operations required for data upload
await fdp.file.uploadData('my-new-pod', '/my-dir/myfile.txt', 'Hello world!', {
  progressCallback: event => {
    console.log(event)
  }
})

Data can also be uploaded block by block, even without an FDP account. Each block will be secured by a Swarm node. Later, with an FDP account, the data can be finalized in the form of a file.

const data = '123'
const blockSize = 1 // recommended value is 1000000 bytes
const blocksCount = 3
const blocks = []
for (let i = 0; i < blocksCount; i++) {
  const dataBlock = getDataBlock(data, blockSize, i)
  // fdp instance with or without logged in user
  blocks.push(await fdp.file.uploadDataBlock(dataBlock, i, dataBlock.length))
}

// fdp instance with logged in user
const fileMeta = await fdp.file.uploadData(pod, fullPath, blocks)

Deleting a file from a pod

await fdp.file.delete('my-new-pod', '/my-dir/myfile.txt')

Sharing a file from a pod

const shareReference = await fdp.file.share('my-new-pod', '/my-dir/myfile.txt')
console.log(shareReference) // prints share reference of a file

Get information about shared file

await fdp.file.getSharedInfo(shareReference)

Save shared file to a pod

await fdp.file.saveShared('my-new-pod', '/', shareReference)

Getting list of files and directories with recursion or not

// with recursion
const list = await fdp.directory.read('my-new-pod', '/', true)
// without recursion
await fdp.directory.read('my-new-pod', '/')
console.log(list) // prints list of files and directories

Downloading data from a file path

const data = await fdp.file.downloadData('my-new-pod', '/myfile.txt')
console.log(data.text()) // prints data content in text format 'Hello world!'

// you can also track the progress of data download
// using the callback, you can track not only the progress of downloaded blocks but also other time-consuming operations required for data download
await fdp.file.downloadData('my-new-pod', '/myfile.txt', {
  progressCallback: event => {
    console.log(event)
  }
})

// or you can download data block-by-block to combine it later
const blockSize = 1000000
const fileMeta = await fdp.file.getMetadata(pod, fullPath)
const result = new Uint8Array(fileMeta.fileSize)
for (let i = 0; i < blocksCount; i++) {
  result.set(await fdp.file.downloadDataBlock(fileMeta, i), i * blockSize)
}

Deleting a pod

await fdp.personalStorage.delete('my-new-pod')

Checks whether the public key associated with the username in ENS is identical with the wallet's public key

await fdp.account.isPublicKeyEqual('username')

Using FDP instance with cache

const fdpCache = new FdpStorage('https://localhost:1633', batchId, {
  cacheOptions: {
    isUseCache: true,
    onSaveCache: async cacheObject => {
      const cache = JSON.stringify(cacheObject)
      console.log('cache updated', cache)
    },
  }
})

Recovering FDP instance with saved cache

const fdpCache = new FdpStorage('https://localhost:1633', batchId, {
  cacheOptions: {
    isUseCache: true,
  }
})
fdpCache.cache.object = JSON.parse(cache)

There are available function for interacting with DataHub contract. For example to list all available subscriptions:

const subs = await fdp.personalStorage.getAllSubscriptions()

To get user's subscriptions:

const subItems = await fdp.personalStorage.getAllSubItems()

And to get pod information of a subItem:

const podShareInfo = await fdp.personalStorage.openSubscribedPod(subItems[0].subHash, subItems[0].unlockKeyLocation)

Data migration

Starting from the version 0.18.0, pods and directories are stored in different format than the older versions. For all new accounts this doesn't have any impact. But to access pods and folders from existing accounts, migration is required.

Migration is done transparently, but there are some requirements that users should follow.

Pod list is converted to V2 when the fdp.personalStorage.list() method is invoked. So before working with existing pods, call the fdp.personalStorage.list() method first.

Directories are converted on the fly. Here the same principle applies as for pods. The fdp.directory.read() method must be invoked first, before invoking any other operation on the provided directory. The read method will convert not only the provided directory, but also all parent directories. That process happens only once, and all subsequent accesses will work with V2 data instantly.

Existing files are accessible in the new version. But from the 0.18.0 version, files are compressed before upload, which wasn't the case before. To compress files, they must be reuploaded.

Documentation

You can generate API docs locally with:

npm run docs

The generated docs can be viewed in browser by opening ./docs/index.html

Contribute

There are some ways you can make this module better:

Consult our open issues and take on one of them
Help our tests reach 100% coverage!

Setup

Install project dependencies with

npm ci

Test

The tests run in both context: Jest and Puppeteer.

To run the integration tests, you need to use our fdp-play project.

With specific system environment variables you can alter the behaviour of the tests.

BEE_API_URL - API URL of Bee client
BEE_DEBUG_API_URL - Debug API URL of Bee client
BEE_BATCH_ID - Batch ID for data uploading
FAIROS_API_URL - FairOS API URL

There are browser tests by Puppeteer, which also provide integrity testing.

To run the tests for this project, you can use the following commands:

# to run all tests
npm run test

# to run node tests
npm run test:node

# to run FairOS-dfs integration tests
npm run test:fairos

# to run unit tests
npm run test:unit

# to run browser tests
npm run test:browser

The test HTML file which Puppeteer uses is the test/integration/testpage/testpage.html. To open and manually test FDP with developer console, it is necessary to build the library first with npm run compile:browser (running the browser tests npm run test:browser also builds the library).

Compile code

In order to compile NodeJS code run

npm run compile:node

or for Browsers

npm run compile:browser

Maintainers

License

BSD-3-Clause

fairDataSociety / fdp-storage

readme