permanent-seeder

npm version

Permanent Seeder logo, a beautiful seed surrounded with a round and exquisite black border

Install

npm i -g @geut/permanent-seeder

Alternatively you can git clone this repo and build it from source:

# install deps
npm i
npm run bootstrap

# build the dashboard
cd packages/dashboard
cp .env.example .env
npm run build

# start the seeder
cd ../cli
./bin/run stop
./bin/run start

Usage

First, create a base config file, you can tweak it later ;-)

$ permanent-seeder config:init

Then, start the Permanent Seeder daemon

$ permanent-seeder start

:rocket:

Commands

$ permanent-seeder [COMMAND] [--OPTIONS]

Config

$ permanent-seeder config:[init|get]

init: creates the base config file for the Permanent Seeder. This is a .toml file that will live in ~/permanent-seeder/settings.toml.
get: returns the settings from the CLI. Useful when you are changing values and want to be sure they are pick up by the Permanent Seeder.

Default settings:

## Permanent seeder path (will be completed on config:init)
path = 'permanent-seeder'

## Enable stats recording
save_stats = true

## [optional] Preferred swarm port
# swarm_port =

## [optional] Api service settings
# [api]
# port = 3001

## [optional] HTTPS certificate and key
# [api.https]
# cert = "/path/to/cert.crt"
# key = "/path/to/cert.key"

## keys.endpoints = array of configs per endpoint
[[keys.endpoints]]

  # Where to fetch keys
  url = 'http://localhost:3000'

  # Frequency between fetchs (in minutes)
  frequency = 5

  # Hook to parse response
  hook = 'endpoint-hook.js'

## [optional] To add another endpoint, uncomment and complete next lines:
# [[keys.endpoints]]
#   url =
#   frequency =
#   hook

Start

$ permanent-seeder start

Bootstrap a Permanent Seeder instance that will keep up running in the background. If you change settings, you will need to call start command again.

Status

$ permanent-seeder status

It will return instance status. If it is running and some basic stats.

Stop

$ permanent-seeder stop

Stops the current instance.

Dashboard

$ permanent-seeder dashboard

Opens the dashboard app in a browser. If you want to manually access the dashboard, it can be found in: localhost:3001

:warning: Note: The dashboard app runs in http://localhost:3001. If you deploy the Permanent Seeder on a server and wants to access the dashboard from the outside, you may need to setup a reverse proxy.

Sample nginx reverse proxy config

``` upstream dashboard-nodejs { server 127.0.0.1:3001; } server { listen 80 default_server; listen [::]:80 default_server; server_name _; location / { proxy_pass http://dashboard-nodejs; proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; proxy_redirect off; proxy_buffering off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } location /socket.io/ { proxy_pass http://dashboard-nodejs; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ```

Environment Variables

We use the .env.example file as a placeholder for some environment variables that we use internally, like PUBLIC_URL.

If you build the Permanent Seeder from source then you may want to customize some things.

There are two variables that can be used:

PUBLIC_URL: used to define the root path of the dashboard app. DEFAULT = '/'
REACT_APP_API_VERSION: used to indicate the api version to use. DEFAULT = 'v1'

For development there are a few others variables that you can tweak, like the SOCKET_URL and API_URL. Checkout the full .env.example:

REACT_APP_API_VERSION=v1
PUBLIC_URL=/

## Extend eslint config
EXTEND_ESLINT=true

## Optional ENVs only for development
#REACT_APP_SOCKET_URL=
#REACT_APP_API_URL=

Key Management :key:

$ permanent-seeder key:[add|remove|remove-all]

add: Insert a new key Permanent Seeder db, it will start downloading and seeding ASAP.
remove: Removes a single key from the seeder db and also stops seeding it (e.g.: no more announcing to other peers)
remove-all: Removes and unnanounce all the keys in the db.

Logs

$ permanent-seeder logs --[live|all|error]

live: like doing a tail -f of the logs.
all: Show all the logs stored.
error: Display only error logs.

repl

$ permanent-seeder repl

Useful to inspect the Permanent Seeder under the hood. :microscope:

Keys Endpoint

The Permanent Seeder can fetch keys from an external endpoint, i.e: perform a GET request against a particular endpoint. This can be useful if you maintain a service that stores hyperdrive's keys. If that is the case, then the Permament Seeder can fetch those keys regularly. You can think of this like a cron job.

Whilst, we internally expect an Array<{url}>, you can customize and parse the fetch response the way you need it.

To do this, you will need to modify $HOME/permanent-seeder/endpoint-hook.js.

That hook will be called after fetch the response.

You can also tweak the fetch frequency (defined in minutes) and the endpoint url. These options can be found in the settings.toml file:

[[keys.endpoints]]
url = "http://localhost:3000"
frequency = 5
hook = "$HOME/permanent-seeder/endpoint-hook.js"

Delete Keys Endpoint

The Permanent Seeder also allows for another endpoint, this one for deleted keys. With this action the Permanent Seeder can fetch an array of keys similarly to what we defined previously for the keys endpoint (adding).

The setting looks like this:

 [[keys.remove]]
 url = "http://localhost:60620/deletes.json"
 hook = "/Users/deka/permanent-seeder/endpoint-hook.js"

Again, we internally expect an Array<{url}>, but you can customize and parse the fetch response the way you need it.

To do this, you can tweak the hook response. By default the "delete keys endpoint" is using the same default hook used with the keys endpoint.

Finally, the fetch frequency is managed by the keys endpoint frequency. In other words, they run with the same frequency interval.

Update Policy

Here are some rules that we follow to know when to update the keys.

Actions that result in update:

add endpoint is defined and returns a list of keys. delete endpoint is not defined. :white_check_mark:
add endpoint is defined and returns a list of keys. delete endpoint is defined and returns a list of keys. :white_check_mark:

Actions that does not result in update:

add endpoint is defined and returns an empty list. Does not matter if delete endpoint is defined or not. :no_entry:
add endpoint is defined but throws an error :no_entry:
add endpoint is defined and returns a list of keys. delete endpoint is defined but throws an error. :no_entry:

Design

The Permanent Seeder is a CLI tool that can starts a daemon which will seed hyperdrive's keys that you pass into it. Using the CLI you can add, remove keys, check the status and inspect logs.

It also contains a dashboard that you can use to have a visual reference of what is going on with your hyperdrives.

As you can see the project does a couple of things. To do this we decided to use a microservices based approach. We choose to use moleculer as the structural framework behind the Permanent Seeder. This enables multiples processes to communicate each other and at the same time each of these will have a single responsibility/scope. This also give us some room to scale things up if needed. :sunglasses:

Routes

Below we define some internal API routes that can be used to obtain some information about the PS running instance as well as some basic drive queries.

`GET /api/<API_VERSION>/drives/:key`

Returns summary data associated to a specific drive key. Data may contain download information, peers info, seeding status and basic index.json information and version.

`GET /api/<API_VERSION>/drives/:key/size`

Returns only the download information.

`GET /api/<API_VERSION>/drives/:key/peers`

Returns only the peers information.

`GET /api/<API_VERSION>/drives/:key/stats`

Returns only information about the drive's files and its download status.

`GET /api/<API_VERSION>/drives/:key/info`

Returns only the index.json information part.

`GET /api/<API_VERSION>/drives/:key/seedingStatus`

Returns only the seeding status label.

`GET /api/<API_VERSION>/drives/:key/files/:file`

Download the file passed as argument from key drive.

`GET /api/v1/stats/host`

Returns information about the host. CPU load, memory and directory usage.

`GET /api/v1/stats/network`

Returns network information. Include things like online status, swarm information (hole-punching, bootstrapping info and network address), along with transferred bytes.

`GET /api/<API_VERSION>/raw/:key`

Returns the raw data associated with the seeded drive (key). The returned data may include various lifecycle events with their timestamps. This endpoint will return values as long as save_stats config property is enabled. See config.

Filter by Version

You can also query data using the drive version.

`GET /api/<API_VERSION>/drives/:key/versions`

Returns all the versions saved for key drive

`GET /api/<API_VERSION>/drives/:key/:version/info`

Returns only the info field for this specific key and version

`GET /api/<API_VERSION>/drives/:key/:version/stats`

Returns only the stats (files) field for this specific key and version

`GET /api/<API_VERSION>/drives/:key/:version/files/:file`

Download the file passed as argument from a specific drive's checkout (key+version).

Release

For releasing new versions we are using GitHub Actions :rocket: This is the process.

First, we create a new release branch: git co -b username/release-${newVersion}
Then, we need to create the new version. This new version will trigger a changelog update (using chan). To do this just run: npm run release It will prompt you what is the next version.
Create the release pull-request. The title should indicate this: Release: newVersion
- :warning: This is an important step, otherwise the publish workflow won't see the PR.
Finally, when the release PR is approved, a custom action will be triggered publishing the new version to npm, creating a new tag for it and creating a brand new github release :new: :package:

Issues

:bug: If you found an issue we encourage you to report it on github. Please specify your OS and the actions to reproduce it.

Contributing

:busts_in_silhouette: Ideas and contributions to the project are welcome. You must follow this guideline.

Built in collaboration with Liberate Science

Acknowledgments

:clap: Many thanks to @nicomanso for offering some basic infrastructure that was used for early testing and @krahser for his outstanding docker contribution (PR)

geut / permanent-seeder

readme