Create Graph Data Explorer CLI Tool

[zicklag]

Right now it's tricky to see what data is in our database, since we use a custom graph format, so we should have a CLI tool that can be used to explore, edit, and export our graph data.

This is kind of a necessity at this point for debugging purposes. We can do a simple CLI interface using ratatui.

• We don't need a lot of features to start. The database itself is still growing and we're still figuring out how it's going to work, so we want to keep things kind of light so it's not too hard to adapt in case the database changes a bit.
• The most minimal thing I want to be able to do is view the key-value documents kind of hierarchically.
  • So at the top level I want to view all of the key-value pairs with one key segment.
  • When I hover over it, it would be cool to preview what's inside it in a split view.
  • If the value is a map, or is a prefix to other keys, then I want to be able to select it to "zoom in" and now list all of the sub-keys with that prefix.
• Other than viewing hierarchies of keys, if there is a Link I want to be able to select the link to jump to whatever node it links to.
• I want to be able to like press backspace or something to go back up in the hierarchy, or backward along the links I've followed, similar to a browser history.
• Finally, if we have basic viewing, it would be good to be able to open a "dialog" or something to add/edit values in the store as well.

It might also be good to have a "raw" browser mode where it just lists every key on the left, and the value of the selected key on the right.

Here's the really raw work-in-progress view of what is currently in the repo:

[zicklag]

Now that we have improved the graph format to be more intuitive and inspectable, we can wait for a full-on data explorer. I'm moving this towards a later step, and a more immediately useful feature would be a way to export the graph database to YAML, which will satisfy the need for seeing into the database as well as be useful for backups.

[gold-silver-copper]

This library could perhaps be useful for prior art, although it doesn't seem too complete at the moment?

https://crates.io/crates/tui-nodes/0.4.0

[gold-silver-copper]

Would be a good idea to create a mock up of what we would want the explorer to look like, and document the features we would want for it.

[gold-silver-copper]

Is it being a CLI tool a hard requirement? egui has really good graph widgets, that may be more clear and usable for a really large database:

https://github.com/blitzarx1/egui_graphs

[zicklag]

Is it being a CLI tool a hard requirement?

Yeah, I think we want to start with a terminal UI because one major use-case would be debugging the production database on a remote server over SSH.

Having an Egui UI would be awesome, too, but terminal is the higher priority.

---

Would be a good idea to create a mock up of what we would want the explorer to look like, and document the features we would want for it.

Definitely, here are my thoughts on it so far:

• We don't need a lot of features to start. The database itself is still growing and we're still figuring out how it's going to work, so we want to keep things kind of light so it's not too hard to adapt in case the database changes a bit.
• The most minimal thing I want to be able to do is view the key-value documents kind of hierarchically.
  • So at the top level I want to view all of the key-value pairs with one key segment.
  • When I hover over it, it would be cool to preview what's inside it in a split view.
  • If the value is a map, or is a prefix to other keys, then I want to be able to select it to "zoom in" and now list all of the sub-keys with that prefix.
• Other than viewing hierarchies of keys, if there is a Link I want to be able to select the link to jump to whatever node it links to.
• I want to be able to like press backspace or something to go back up in the hierarchy, or backward along the links I've followed, similar to a browser history.
• Finally, if we have basic viewing, it would be good to be able to open a "dialog" or something to add/edit values in the store as well.

It might also be good to have a "raw" browser mode where it just lists every key on the left, and the value of the selected key on the right.

Here's the really raw work-in-progress view of what is currently in the repo:

---

So we don't really need like a connected graph view for the database. We want something almost more like a "wiki page browser", where I can just kind of walk through the structure of the graph.

[zicklag]

It's probably best just to continue with our own TUI, but I did just find this which could be good reference: https://github.com/UnkwUsr/nctok

[AnarchistHoneybun]

came here through the ratatui discord also:

1. the discord link for this project is broken, I think
2. I'm willing to put some work in this 🤔 could you share some sort of snippet of the kind of data that needs to be visualized? I'm reading through the specs listed above and can get a rough idea of how to structure the ui elements, but need to know more about the data to write the functions to look it up

[zicklag]

Hi @AnarchistHoneybun! I updated the discord link, and here it is, also.

Regarding the data we want displayed, you can see the gdata Value type and the Key and KeySegment types.

The idea is that we want to be able to view the values in each document, which is a key-value store. The Value type is kind of like a serde_json::Value, but with extra kinds like Bytes and Link.

The Keys are represented with an array of KeySegments, which may be things like strings, bytes, or numbers. The different key segments allow us to make something kind of like a filesystem path, where each segment is like a "folder" and each tail has a value.

The filesystem analogy only goes so far, though, because any key can have subkeys, so "files" can have "files" in them.

What we want to do is allow you to browse the keys kind of like a filebrowser, where you have this concept of a "current working prefix" or breadcrumb, and it shows the keys under that prefix for one level deep.

When you "hover" over a key, then it should show you the value in the right-hand panel, whether that be a string, or a number, or bytes ( which should be base32 encoded for display I think ).

A value type that we'll want special handling for is a Link, which should have an easy way for you to follow the link to the document/key that it references.

I may have already said some of that above, but hopefully that gets the idea across a little better.

[azzamsa]

2. I'm willing to put some work in this 🤔

@AnarchistHoneybun seems you already had a good project with Ratatui https://github.com/AnarchistHoneybun/cl_2048 💯

[AnarchistHoneybun]

tyvm! i love mocking up tui's, that was probably my first full scale project using one. the iteration speed is so fast with these I really love how soon projects start coming together