Unify pkgs and styx

[joeltg]

Joel's Current Grand Vision is to unify styx and pkgs so that pkgs has a plugin architecture that can integrate any index (i.e. database index) that satisfies some interface.

In my mind right now this interface is just

type ResourceType uint8

const (
  PackageType ResourceType = iota
  MessageType
  FileType
)

type Index interface {
  Set(path []string, c cid.Cid, t ResourceType)
  Delete(path []string)
}

... which styx can be refactored to implement.

For now each index would be responsible for providing its own usage endpoint (i.e. styx would still attach an API handler to some other port) but that is also something that should be unified under pkgs in the future.

[joeltg]

As of this commit, Styx has replaced its HTTP API with a use-as-a-library Cursor interface, with the idea that pkgs will bundle a StyxIndex adapter that wraps a styx.Cursor as an Index in some way.

Playing around with possible Index interfaces led me to something like this:

type Index interface {
    Add(pathname []string, resource Resource)
    Remove(pathname []string, resource Resource)
    Signatures() []Signature
}

type Signature interface {
    Head() []*ld.Quad
    Domain() []string
}

// Resource is interface type for resources (packages, messages, and files)
type Resource interface {
    ETag() (cid.Cid, string)
    URI() string
}

It feels like unifying query-style access to indices (the "output interface") is just as important as unifying the add/remove functions (the "input interface"), so that's what Signatures() is - a set of query signatures, where each signature is a subgraph (with some blank nodes) and a domain (the subset of the blank nodes in the subgraph that are required to be ground for the signature to match).

For example, an index that lets you look up an ID from a schema.org name (but not the other way around) would have a signature with a head of

_:id <http://schema.org/name> _:name .

and a domain of

[]string{"_:name"}

... but an index that lets you look up a name from an ID would have the same head, but a domain of

[]string{"_:id"}

This still is only an interface for selecting which indices match a query, and not actually querying them (we could potentially re-use the Cursor interface from Styx), but it's a start.

[joeltg]

--- More thoughts --- I think embracing a Cursor interface (or something like it) all the way to end usage might be the best thing to do for queries. I had previously assumed that we'd wrap Cursors in some declarative HTTP API, but maybe it's best to actually expose an interactive websocket/libp2p stream-based interface where actual end users of queries - like javascript frontends etc - actually open a transaction and can page through results interactively before closing it.

This would mean a JavaScript library to expose like

const { Cursor } = require("styx-client")
const pattern = {
  "@context": "ipfs://Qm...",
  "name": "...",
  "properties": { ... }
}
const cursor = new Cursor(pattern, "http://localhost:8086")
cursor.graph // returns the current value as an N3Store or JSON-LD object or something
cursor.next() // advances to next solution
cursor.seek(cursor.domain[3]) // ...
cursor.close()

[joeltg]

--- More thoughts --- I think I have a neat idea of what package servers could be. They should expose two APIs, both over libp2p:

• HTTP REST API
  • PUT/GET/PATCH/etc resources under the root package
  • ETags, headers, content negotiation, etc
  • generally "view and manage resources"
  • proxying over a regular http listener
• JSON-RPC query API
  • json-rpc version 1 for now bc that's what the go standard library has
  • expose the operations on the Cursor interface (which now lives in pkgs/query; styx imports from it)
  • JSON-LD for values/arguments/params (this feels weird since some will just be "value nodes")
  • the RPC format becomes the new source-of-truth for querying
  • all "package indices" implement Cursor
  • pkgs does some magic registering index signatures and routing incoming rpcs to matching handlers
  • In the future, package servers can be clever about composing several different handlers and indices at the same time
  • In the future, index implementations might expect/depend/use on other rpc'd indices?
  • badger transactions are held open during entire rpc sessions
  • proxying over a local websocket? less important than for the HTTP API

Both of these get exposed as libp2p protocols