CLI for interactive exploration of JSON or XML resources

[JohannesLichtenberger]

The idea is to navigate and have commands similar to a shell: https://news.ycombinator.com/item?id=31173562

[JohannesLichtenberger]

CLIs: https://github.com/antonmedv/fx, https://jless.io/

[AlvinKuruvilla]

To be clear, is the intention here to recreate the functionality of jless or fx in java? Are there any specific features these projects lack that you want to include in our implementation?

[JohannesLichtenberger]

Yes, it's a database backed shell, as SirixDB can store arbitrary amounts of data in a single database resource (more than 4Gb for instance).

@mosheduminer however also began to rewrite our web based frontend. The functionality IMHO would be a bit overlapping.

We already have a small Kotlin CLI where we might be able to add interactive exploration of a resource. I think, however that the frontend with interactive views are more pressing. What do you think?

[AlvinKuruvilla]

Personally, I think interactive JSON viewing should be something left for the front end. Since we are getting JSON or XML responses anyway, I feel it would be simpler to pipe the results to jq for prettier viewing in a terminal setting. There are also already react (and svelte, though not too sure about usability) packages for interactive JSON viewing like jless. For an interactive shell environment, we could push the development of sirixsh and make that our standalone project focused on only the shell side of it. In my opinion, it would probably be worthwhile to take a similar approach to the CLI(s), though I don't know how feasible that is.

[JohannesLichtenberger]

I think however, one of the main benefits of SirixDB is also, that you can simply include it as a dependency in other JVM languages. Furthermore you should be able to use SirixDB locally without any network latency. That's why I think that we need an additional shell (bundles/sirix-kotlin-cli) that can be used without starting the HTTP-server. In this case you don't even necessarily retrieve the serialized JSON or XML responses. Internally it's stored as a fine grained DOM alike tree (firstChild/lastChild/leftSibling/rightSibling/parent encoding) and object, array, object key, string value, boolean value, null value and number value nodes, besides the object record value counterparts. Thus, for the CLI we can simply get the nodes in a format which fits (might also be JSON of course).

But I'm agreeing that the frontend IMHO currently is the most important thing. Sadly, Moshe currently has not much time and I'm a backend engineer (though I started an Udemy TypeScript course ;)).

[AlvinKuruvilla]

I see what you mean: If we are able to store our data locally in a DOM style tree structure we could serialize it into a JSON or XML format if I understand correctly. I think it would also be interesting if instead of doing an interactive JSON viewer we stuck with the DOM/graph style and did an app where you can select data (nodes and the like) and run interactive queries that way (just a thought). In any case, this would be an interesting package to add to the organization and I could consider at least sketching out some ideas, just let me know where you would like me to begin

[JohannesLichtenberger]

Maybe, if you're confident with frontend-dev you can work on a brand new frontend (maybe built with SolidJS). @mosheduminer might probably find some time to push a first skeleton (but it's almost empty I think) of a SolidJS based frontend.

Here's a screenshot of a dated (bundles/sirix-svelte-frontend) frontend, but I think either we start a new frontend or we'll have to port it from Sapper to SvelteKit (https://sapper.svelte.dev/faq). I also think, that SolidJS might currently be the better alternative as it's based on JSX which a lot of frontend devs might know from React and it seems to be faster...

https://raw.githubusercontent.com/sirixdb/sirix/master/Screenshot%20from%202020-09-28%2018-50-58.png

[JohannesLichtenberger]

Our idea was to make an initial query whereas the result set has to be a single item (JSON item or XML node). Each node in SirixDB has a unique, stable nodeKey in all revisions. Thus, we can based on this nodeKey visualize a subtree and it's diffs for instance in a tree-view (the right view in the screenshot).

[AlvinKuruvilla]

Yeah, I'm not too experienced with frontend work either, but in the meantime, I could work on a command-line style app to try and achieve a similar purpose (though I would probably end up writing it in Python because that's what I am more familiar with). Having a tool like this accessible through the command line is still worthwhile I feel. Now I have to understand how and where Sirix stores its data and how I can quickly mock some data to work with 😅

[mosheduminer]

@AlvinKuruvilla it might be worth looking at the pysirix CLI included with the python client package. It's rather messy and not so friendly with writing queries, but it does work.

[mosheduminer]

The pysirix CLI also lets you put a bunch of commands in a file and run them one after the other.

[AlvinKuruvilla]

That's a good idea. Thanks for the suggestion, @mosheduminer I'll take a look