@beaker/dat-node

A toolkit for writing Dat-based services in nodejs.

const {createNode} = require('@beaker/dat-node')

// instantiate a new dat node
const dat = createNode({
  path: './dat'
})

// get, create, or fork archives
var archive = await dat.getArchive('dat://beakerbrowser.com')
var archive = await dat.createArchive({title: 'My Archive'})
var archive = await dat.forkArchive('dat://beakerbrowser.com', {title: 'My Fork of the Beaker site'})

// you can read & write files within the archive
var filedata  = await archive.readFile('/index.html', 'utf8')
await archive.writeFile('/new/thing.json', JSON.stringify({hi: 'world'}))

// listen to global network events
dat.on('network-changed', ...) // change to network conditions (aka new peer)
dat.on('download', ...) // data was downloaded
dat.on('upload', ...) // data was uploaded
dat.on('sync', ...) // a feed was synced

// get internal information about a dat
await dat.getMtime('dat://beakerbrowser.com') // unix timestamp of last change to the dat
await dat.getDiskUsage('dat://beakerbrowser.com') // in bytes
await dat.getSyncProgress('dat://beakerbrowser.com') // 0-1, where 1 is 100% synced

Provides the same Dat APIs that Beaker browser uses, so that code written which depends on the DatArchive will work here and in the browser.

TODOs

This repo is a work in progress. It still needs:

• [ ] Tests
• [ ] DNS shortnames
• [ ] getSyncProgress() implementation
• [ ] DatNode events
• [ ] Probably more

Examples

👉 simplest-possible-example.js

node examples/simplest-possible-example.js

👉 create-archive.js

node examples/create-archive.js

👉 download-archive.js

node examples/download-archive.js dat://beakerbrowser.com

API

• createNode(opts)
• DatNode
  • dat.listen([port])
  • dat.close()
  • dat.getArchive(url)
  • dat.createArchive([opts])
  • dat.forkArchive(url[, opts])
  • dat.getMtime(url)
  • dat.getDiskUsage(url)
  • dat.getSyncProgress(url)
  • dat.isFullySynced(url)
  • dat.createDebugLogStream([opts])
  • dat.storage
  • dat.swarm
  • dat.dns
  • Event: "listening"
  • Event: "error"
  • Event: "network-changed"
  • Event: "download"
  • Event: "upload" (url, {feed, block, bytes})
  • Event: "sync" (url, {feed})
• DatNodeStorage
  • storage.list()
  • storage.get(url)
  • storage.delete(url)
  • storage.getMtime(url)
  • storage.getDiskUsage(url)
  • storage.getDNSCache(hostname)
  • storage.setDNSCache(hostname, value)
  • storage.clearDNSCache(hostname)
• DatNodeSwarm
  • swarm.join(url)
  • swarm.leave(url)
  • swarm.port
  • swarm.networkId
  • Event: "listening"
  • Event: "error"
  • Event: "peer"
  • Event: "peer-banned"
  • Event: "peer-rejected"
  • Event: "drop"
  • Event: "connecting"
  • Event: "connect-failed"
  • Event: "handshaking"
  • Event: "handshake-timeout"
  • Event: "connection"
  • Event: "connection-closed"
  • Event: "redundant-connection"
• DatDNS
  • dns.resolve(url)
• DatArchive

createNode(opts)

Create a new dat dat. See the DatNode API.

• opts
  • path: String, the path at which all data is stored.
  • autoSwarm` Boolean, swarm dats by default when loaded? Defaults to true.
  • autoListen: Boolean, bind a port automatically after load. Defaults to true.
  • port: Number, the port to bind to. Defaults to 3282.

const {createNode} = require('@beaker/dat-node')

var dat = createNode({
  path: './data'
})

DatNode

dat.listen([port])

Async. Start listening for connections in the dat network. Only need to call if createNode is called with {autoListen: false}, or if dat.close() has been called.

• port. Number. The port to listen for connections on. Defaults to 3282.

dat.close()

Async. Unswarm all dats and stop listening.

dat.getArchive(url)

Create a new DatArchive with the given opts.

var archive = dat.getArchive('dat://beakerbrowser.com')

dat.createArchive([opts])

Async. Create a new DatArchive with the given opts. Will automatically persist and swarm the archive.

• opts
  • title String. The title of the new archive.
  • description String. The description of the new archive.

var archive = await dat.createArchive({title: 'My new archive'})

dat.forkArchive(url[, opts])

• opts
  • title String. The title of the new archive.
  • description String. The description of the new archive.

Async. Create a new DatArchive which duplicates the content of the given archive. Can optionally modify the manifest with opts. Will automatically persist and swarm the archive.

var archive = await dat.forkArchive('dat://beakerbrowser.com', {title: 'My fork of the Beaker site'})

dat.getMtime(url)

Async. Get the modification time of the dat's data. (Tells you the last time new data was written.)

var mtime = await dat.getMtime('dat://beakerbrowser.com')

dat.getDiskUsage(url)

Async. Get the amount of bytes being used by the dat in the local cache.

var bytes = await dat.getDiskUsage('dat://beakerbrowser.com')

dat.getSyncProgress(url)

Async. Get the percentage of the total data downloaded (between 0 and 1).

var pct = await dat.getSyncProgress('dat://beakerbrowser.com')

dat.isFullySynced(url)

Async. Is all of the dat's data cached?

var pct = await dat.isFullySynced('dat://beakerbrowser.com')

dat.createDebugLogStream([opts])

Get a readable string-stream containing the content of the debug log. Useful for providing debugging interfaces.

TODO decide what features this should include

dat.storage

A DatNodeStorage instance.

dat.swarm

A DatNodeSwarm instance.

dat.dns

A DatDNS instance.

Event: "listening"

The swarm port is now bound and listening.

• port. Number.

Event: "error"

A critical error has occurred.

• err. Error.

Event: "network-changed"

A change in the network-connectivity of a given dat.

• url. String. The dat url.
• connections. Number. The number of connected peers.

dat.on('network-changed', (url, {connections}) => {
  // ...
})

Event: "download"

Some data has been downloaded.

• url. String. The dat url.
• feed. String. Which feed received data (e.g. 'metadata', 'content')
• block. Number. The block index downloaded.
• bytes. Number. How many bytes were downloaded.

dat.on('download', (url, {feed, block, bytes}) => {
  // ...
})

Event: "upload" (url, {feed, block, bytes})

Some data has been uploaded.

• url. String. The dat url.
• feed. String. Which feed sent data (e.g. 'metadata', 'content')
• block. Number. The block index uploaded.
• bytes. Number. How many bytes were uploaded.

dat.on('upload', (url, {feed, block, bytes}) => {
  // ...
})

Event: "sync" (url, {feed})

A feed has been fully downloaded.

• url. String. The dat url.
• feed. String. Which feed finished downloading (e.g. 'metadata', 'content')

dat.on('sync', (url, {feed}) => {
  // ...
})

DatNodeStorage

The local data store. Includes all data which is persisted onto disk. Can be found on the DatNode API as dat.storage.

storage.list()

Async. List the metadata of locally-stored dats.

var datInfos = await dat.storage.list()

storage.get(url)

Async. Fetch the stored metadata about the archive. Returns null if not present locally.

var datInfo = await dat.storage.get('dat://beakerbrowser.com')

storage.delete(url)

Async. Delete all cached data for the dat.

await dat.storage.delete('dat://beakerbrowser.com')

storage.getMtime(url)

Async. Get the modification time of the dat's data. (Tells you the last time new data was written.)

var mtime = await dat.storage.getMtime('dat://beakerbrowser.com')

storage.getDiskUsage(url)

Async. Get the amount of bytes being used by the dat in the local cache.

var bytes = await dat.storage.getDiskUsage('dat://beakerbrowser.com')

storage.getDNSCache(hostname)

Async. Get the disk-cached DNS lookup result for the given hostname.

• hostname String.

storage.setDNSCache(hostname, value)

Async. Set the disk-cached DNS lookup result for the given hostname.

• hostname String.
• value String.

storage.clearDNSCache(hostname)

Async. Remove the disk-cached DNS lookup result for the given hostname.

• hostname String.

DatNodeSwarm

swarm.join(url)

Async. Load a dat into memory, add it to the local storage (if not yet present) and begin swarming. Does not need to be called if autoSwarm is true.

• url: String, the url of the dat. Can provide a DNS shortname.

await dat.swarm.join('dat://beakerbrowser.com')

swarm.leave(url)

Async. Stop swarming the given dat.

await dat.swarm.leave('dat://beakerbrowser.com')

Will not remove the dat's data from the local storage (see dat.storage.remove()).

swarm.port

Number. The port being listened to.

swarm.networkId

A 32-byte buffer containing a randomly-generated ID, used to deduplicate connections.

Event: "listening"

The swarm port is now bound and listening.

• port. Number.

Event: "error"

A critical error has occurred.

• err. Error.

Event: "peer"

Emitted when a peer has been discovered.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.

Event: "peer-banned"

Emitted when a peer has been banned as a connection candidate.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.
• details. Object.
  • reason. String. Why was the peer banned? May be 'application' (removePeer() was called) or 'detected-self'.

Event: "peer-rejected"

Emitted when a peer has been rejected as a connection candidate.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.
• details. Object.
  • reason. String. Why was the peer rejected? May be 'duplicate', 'banned', or 'whitelist'.

Event: "drop"

Emitted when a peer has been dropped from tracking, typically because it failed too many times to connect.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.

Event: "connecting"

Emitted when a connection is being attempted.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.

Event: "connect-failed"

Emitted when a connection attempt fails.

• peer. Peer.
  • remoteAddress String.
  • remotePort Number.
  • id String. The ID of the peer on the discovery network.
  • channel Buffer. The key used to arrange this connection in the discovery network.
• details. Object.
  • timedout. Boolean. Did the connection fail due to a timeout?

Event: "handshaking"

Emitted when you've connected to a peer and are now initializing the connection's session.

• connection. Connection.
  • remoteAddress String.
  • remotePort Number.
  • key String. The key of the dat which the peer is exchanging.
  • discoveryKey String. The discovery key of the dat which the peer is exchanging.

Event: "handshake-timeout"

Emitted when the handshake fails to complete in a timely fashion.

• connection. Connection.
  • remoteAddress String.
  • remotePort Number.
  • key String. The key of the dat which the peer is exchanging.
  • discoveryKey String. The discovery key of the dat which the peer is exchanging.

Event: "connection"

Emitted when you have fully connected to another peer.

• connection. Connection.
  • remoteAddress String.
  • remotePort Number.
  • key String. The key of the dat which the peer is exchanging.
  • discoveryKey String. The discovery key of the dat which the peer is exchanging.

Event: "connection-closed"

Emitted when you've disconnected from a peer.

• connection. Connection.
  • remoteAddress String.
  • remotePort Number.
  • key String. The key of the dat which the peer is exchanging.
  • discoveryKey String. The discovery key of the dat which the peer is exchanging.

Event: "redundant-connection"

Emitted when multiple connections are detected with a peer, and so one is going to be dropped (the connection given).

• connection. Connection.
  • remoteAddress String.
  • remotePort Number.
  • key String. The key of the dat which the peer is exchanging.
  • discoveryKey String. The discovery key of the dat which the peer is exchanging.

DatDNS

The DNS manager. Can be found on the DatNode API as dat.dns.

dns.resolve(url)

Async. Get the key of the given URL.

var key = await dat.dns.resolve('dat://beakerbrowser.com')

DatArchive

A dat "archive" instance. See the DatArchive API docs.

const {createNode} = require('@beaker/dat-node')

const dat = createNode()
var archive = dat.getArchive('dat://beakerbrowser.com')
await archive.readdir('/') => [...]