trws
commented
3 months ago To get this going, here's the JGF specification website: https://jsongraphformat.info/
We're currently using JGF v1, and I don't think we necessarily want to switch to v2 unless there's a material win for doing so. The main things that I can think of (@milroy I'm pretty sure I'm missing at least one of your suggestions, please correct me) that we've discussed are:
-
we send a lot of default data over the wire. Here's a vertex in flux JGF:
{ "id": "2", "metadata": { "type": "node", "basename": "node", "name": "node0", "id": 0, "uniq_id": 2, "rank": -1, "exclusive": false, "unit": "", "size": 1, "paths": { "containment": "/tiny0/rack0/node0" } } },Just off top of my head:
- no need to send default values
- basename defaults to type
- name defaults to
{type}{id} - rank defaults to -1 when resource doesn't belong to a rank, we could also use -2 to say "same as ancestor" to avoid sending it in almost all cases
- exclusive is almost never used except in allocated JGF, and then defaults to false
- unit is almost never used, and defaults to "each" or "empty"
- size is almost always 1, except for ram
- paths can be calculated entirely from edges, which we're sending separately anyway as long as we guarantee that jgf always includes all ancestor vertices up to the root
- uniq_id always matches the outer "id" field
- the inner "id" is the logical id under the parent vertex, if we have the uniq_id, not sure we need that
With just that, we could send this as essentially equivalent:
{ "id": "2", "metadata": { "type": "node", } },Edges are a bit harder, but since we're using a bidirectional graph finally we should be able to remove the value of the name object, and containment would be the default, so we could take these:
{ "source": "2", "target": "4", "metadata": { "name": { "containment": "contains" } } }, { "source": "1", "target": "5", "metadata": { "name": { "power": "supplies_to" } } },and make them
{ "source": "2", "target": "4", }, { "source": "1", "target": "5", "metadata": { "type": "power" } },All of that is still JGF compatible
- no need to send default values
- Structural sharing, or expansion like GRUG does, as @milroy noted could help a lot. JGF doesn't support this natively, but we could use the metadata to include the information, or use subgraphs with references. Upside to this is the vast majority of our vertices are identical below some point, so we could take say a node, make an exemplar graph as a separate graph, then have a node that says "template these based on this hostlist" or something and we're sending relatively little more than with RV1
- If we're willing to throw JGF the spec out the window, the cheapest, easiest
win would be to switch the edge storage from an array of objects with keys to
an array of tuple-like arrays like this:
[ // [source, target, (optional) type] [2,4], [1,5,"power"] ]Or even switch to a stream of json objects like some other formats use so we can do the same to vertices and process them without having to build the whole object in memory, but this will break any and all JGF parsers.
jameshcorbett
JGF is verbose, and Rabbit-y JGF on elcap systems can become very large. We discussed offline several ways to shrink JGF, both while maintaining the same format and compatibility with the standard (which?) and breaking the standard to achieve greater reductions in size.
Performance tests are necessary to see just how bad the problems are, so we can decide how radical of changes we need to make.