wchargin
commented
5 years ago Thanks for writing this up! Quick thoughts, will re-read later:
Addresses are often quite long
Yes: note that we are talking about JSON-stringified addresses, which
render NUL characters as \u0000 (six bytes).
The average length of a JSON-stringified address in the SourceCred graph (including the quotes) is 102 bytes:
$ gzip -dc /tmp/sourcecred/data/sourcecred/sourcecred/github/view.json.gz |
> jq '.[1][] | keys | .[]' | awk '{ x += length($0) } END { print x / NR }'
102.385so this expression is likely dominated by
n*A + e*8A
It would be great to do some quick empirical analysis of decompositions of various sizes. Could you please post the JSON files for, say:
- sourcecred/sourcecred
- twbs/bootstrap
- something in between: maybe ipfs/go-ipfs?
Then you or I can use some simple jq queries to investigate exactly
where the bytes stack up. (Doing this on the post-gzip view is less well
defined.)
For instance, while addresses may be about 100 bytes each, floats tend
to be about 20 bytes each when JSON-stringified, and likely compress
more poorly: address formats are extremely regular. (All GitHub
addresses share a common 30-byte prefix, and all edges for a given
repository have an extra ${owner}/${name} in every address. There’s
little entropy.)
It means that the serialized format becomes a part of Graph's public API. Right now this is not the case (
GraphJSONis an opaque type).
As soon as we have such a thing as a public API, the serialized format
will be part of it. The type may be opaque, but there is a contract
that data serialized with toJSON will be readable with fromJSON.
Modify Graph to allow payloads.
As you probably suspect, I really don’t want to do this.
Another way to achieve something similarly general without polluting
Graph would be to have a dedicated compactGraph module whose sole
purpose is to define a serialization format for a graph with payloads.
You avoid parameterizing the graph and dealing with the issues that
arise there (what type parameters do we use on plugin adapters? how
about merge?), and you only have to deal with the extra complexity at
the serialization boundaries.
Comparison of Approaches
I’ll add one:
\5. Store the data in an SQLite database
With no indices, SQLite databases tend to be pretty space-efficient. They don’t have the gross byte inefficiencies of JSON (floats are actually stored in just 8 bytes!). We can also gzip them. See:
This simple schema should(?) suffice:
CREATE TABLE params (
zero INTEGER NOT NULL PRIMARY KEY, -- singleton table: value is 0
self_loop_weight FLOAT NOT NULL
);
CREATE TABLE nodes (
rowid INTEGER NOT NULL PRIMARY KEY,
address BLOB UNIQUE NOT NULL,
score FLOAT NOT NULL
);
CREATE TABLE edges (
rowid INTEGER NOT NULL PRIMARY KEY,
address BLOB NOT NULL,
src INTEGER NOT NULL,
dst INTEGER NOT NULL,
weight FLOAT NOT NULL,
FOREIGN KEY(src) REFERENCES nodes(id),
FOREIGN KEY(dst) REFERENCES nodes(id)
);To do this in production, we’d want to look into SQLite-in-browser. But it should be easy to prototype this on the server to see what the space looks like. Plus, SQLite-in-browser is a reasonable thing to look into anyway.
teamdandelion
Currently, we represent the results of PageRank via the
PagerankNodeDecomposition. The NodeDecomposition is a rich data type in that it allows us to answer the following questions:Thus, the PagerankNodeDecomposition has all the data needed to run the cred analysis workflow of the explorer. (Although the explorer wants some additional plugin-specific metadata so that it can display node names prettily.)
Unfortunately, the PagerankNodeDecomposition is quite large. Now that we want to serialize these results to disk, we should find a better representation, lest we get blocked by the 100MB file size limit on GitHub pages.
Size of a PagerankNodeDecomposition
Why is the PagerankNodeDecomposition so large? Let's analyze it.
Let
nbe the number of nodes in the graph,ebe the number of edges,Abe the size of an address (for convenience we assume that node and edge addresses are the same length),Fbe the size of a float, andKbe the size of a string constant ("SYNTHETIC_LOOP", "IN_EDGE", "OUT_EDGE").Then the size of a PagerankNodeDecomposition is:
n * (K + 3F + A) + e * (2K + 8A + 4F)Addresses are often quite long, so this expression is likely dominated by
n*A + e*8A(Derivation)
At a top level, we have: ``` export type PagerankNodeDecomposition = Map< NodeAddressT, {| +score: number, // Connections are sorted by `adjacencyScore` descending, // breaking ties in a deterministic (but unspecified) order. +scoredConnections: $ReadOnlyArraySize of a Graph
This is very inefficient compared to how we represent graphs.
When we store a graph, we store
nA + e(A+2F). We're able to get this performance because we are careful to only serialize each address once, and thereafter we reference it by index.Optimal Storage
Theoretically speaking, to encode the information in the PagerankNodeDecomposition, assuming that we already have the graph, we only need to store three kinds of information:
Thus, we should be able to store equivalent data in only
n(A+F) + e(A+4F) + F(Note: this excludes additional display metadata for the cred explorer, i.e. we are not storing the human-readable names for each node.)
However: How to actually achieve this?
Less-Optimal But Convenient Implementation
A simple approach would be to store the extra data that we need in maps alongside the graph:
This does not leverage the integer indexing trick (except for in the graph), so we pay
F + nA + e(A+2F) + e(A+F) + n(A+F)for a total ofF + 2nA + nF + 2eA + 3eFor an extranA + eAcompared to optimal.Comparison of Approaches
If we use the pND, then we have to store
n+8eaddresses. If we implement a simple improvement, we store2n + 2eaddresses. If we implement the optimal approach, then we storen+eaddresses.Given this information, I see 4 major courses of action:
We write entirely new code for serializing
CredAnalysisResultas defined above, and reimplement the integer indexing trick. This has the advantage that it is local, but duplicating the integer serialization logic (and, implicitly, coming up with a whole new serialized format for Graph) is very unappealing.We could have a method that calls toJSON on the Graph, and then iterates over the index-order nodes and edges of the serialized, using that to produce an ordered array of node scores and edge weights. Then we can do the reverse on deserialization.
This has the advantage that it involves re-implementing less logic, is straightforward to implement. However, depending on Graph's serialization format seems weird. It means that the serialized format becomes a part of Graph's public API. Right now this is not the case (
GraphJSONis an opaque type).We could make Graph parameterizable by Node and Edge payload types. In this case, the node payload is a number, and the edge payload is an
EdgeWeight. Then the Graph serialization logic can efficiently store the paylods. Then theCredAnalysisResulttype above could be declared as:This has the advantage that it doesn't re-implement graph's serialization, or subvert it. Also, once we have this, we may find other uses for it.
It has some disadvantages too:
Graphclass, and we're quite pleased that it's simple.toJSON/fromJSONWe can just implement the
CredAnalysisResultwith the maps alongside the graph. It's inefficient compared to optimal, but much better than using thePagerankNodeDecomposition. And if the 100MB size limit becomes an issue, we could further navigate around that by splitting it into separate files (for each top-level field) before needing to implement any complicated or hard-to-maintain solution.