Config web ui

[deontologician]

There needs to be a user-friendly web ui for horizon app configuration. Anything in the config file ( #153 ) should be possible to modify through the ui.

We talked about having the ui served with --dev on a different port, but we should probably mount it at /horizon/admin or something. It would modify the config file on disk directly. I don't know how we want the endpoint to work. It should be out of band from the horizon client, so maybe a rest endpoint.

pinging @Tryneus and @dalanmiller for any stuff I've forgotten. Also pinging @mglukhovsky since he'll probably be building the frontend for this

[dalanmiller]

How should the front-end communicate with the server?

Maybe just to get it started, every time a config is changed just POST the entire JSON configuration to the server? I'm thinking maybe for a working prototype something like chrome://flags?

[deontologician]

That sounds pretty reasonable, and simple. We have defaults, so we should probably fetch those from the config file when serving up the admin page

[dalanmiller]

Also, did we determine where the $(APP_NAME)_config.toml file will live? And where is the list of configurable settings?

[deontologician]

The list of configurable settings is mostly up in the air at the moment. We've kind of been using the command line arguments as candidates, but there will be other things as well. Like specifying keys etc

[dalanmiller]

Would we want to start with just being able to live modify the oauth settings at the moment then? What other things should be able to be edited while Horizon is running?

[deontologician]

Hmm, good point that the webui will be up while horizon is running. Probably not worth trying to detect config changes from the server itself, maybe it can just restart the server process with the current config once the user confirms changes?

[coffeemug]

I would also be very very nice to have a few stats that user could look at. The obvious one I can think of is the number of open client connections on the Fusion server; I'll post some more ideas when I think of them.

[deontologician]

@coffeemug You'd mentioned @segphault might be able to do this? If not we should bump it to subsequent

[segphault]

I think translating the horizon.io design language into a user interface is going to be a pretty exciting exercise. I can take a crack at prototyping something. It seems like there doesn't need to be much for the initial release. I think a really simple app with React or Riot and some Epoch graphs would be pretty cool.

[coffeemug]

:+1:

EDIT: I'll think about what should go into the UI and talk to @mglukhovsky about it, and will then talk to @segphault about what we can realistically accomplish.

[deontologician]

@segphault Any chance you could open up a PR for this with your current stuff? I've been using the PR: in progress tag so people don't poke at it while I'm still working on things

[segphault]

@deontologician for the frontend part? Yeah, I think what I'm going to do is add it to the branch that I'm currently working in and we can just hold off on merging it.

[segphault]

Here's an update and a screenshot for those who are interested in the status...

Current goals

• Dashboard: stat tiles, live graph, and recent events
• Request inspector: filter, inspect, and replay/debug requests
• Collection management: list and delete top-level collections/indexes
• Configuration: configure and save Horizon settings

Next steps

• User management: list and delete user accounts
• Data browser: real-time, editable view of data inside of collections
• Server stats: monitoring for all Horizon instances connected to the database

Current Status

The code in the branch currently does the following:

• Creates all of the tables and indexes used for the live stats (TODO: make the tables use soft durability)
• Adds and removes clients from a table as users connect and disconnect
• Logs each request in a table
• Increments/decrements the cursor count for each request so that we can track which requests are live in realtime
• Exposes the statistics through a new set of admin endpoints
• Stat collection is off by default. There's a new flag/option called --stats that will turn it on

I also have a frontend interface prototype that displays some of the tiles, an Epoch graph, and a log of recent requests. I'll publish the source code soon for those who want to try it out and help. This is what it looks like right now:

The style will evolve as the Horizon branding effort produces a more concrete design language that I can adapt. I originally did my first prototype of the frontend design with React and Uber's react visualization library, but I rewrote it with IncrementalDOM and Epoch for various technical reasons. I'm solidly convinced that Epoch is the right choice for the graphs, but I'm really only using IncrementalDOM because it lets me prototype faster—it'll still be relatively easy to switch the project back to React if desirable when the underlying design is more mature.

Features that I need

I haven't started working on a data browser yet because I'd ideally like to see what we do about pagination first. I think that once pagination is built into Horizon, building a proper data browser will be relatively easy.

[segphault]

There are a few architectural issues that still need to be resolved, mainly things that will affect multi-instance Horizon installations. Right now, the current implementation doesn't periodically prune old data, it just completely wipes the stats tables every time you start the Horizon server.

In an environment where there are multiple instances of Horizon speaking to the same database and some of them are going up and coming down at different times, there are some challenges. When an instance goes down, we need to be able to remove all of the client records for that instance and zero out all of the cursor counts on the instance's logged requests.

I have to think more about how I want to handle that, but I think what we're going to want to do is add more instrumentation in the database to track the status of each Horizon instance, so we can use a changefeed to detect when instances connect and drop out.

[niieani]

Another useful feature would be impersonation. To browse the database and clearly see the permissions from the perspective of a selected user / class of users (or whatever the final security model is).

[segphault]

I've been working on the data browser, which can now page through a collection. I still need to add ordering, editing, and etc.

[niieani]

@segphault that's awesome. The UI looks really beautiful too!

[deontologician]

Man, this is so much better than I was expecting to be able to do. Thanks @segphault for swooping in and building this, it looks amazing

[joshwiens]

@segphault - Killed it :)

[casoetan]

This looks fantastic.

Any updates on this?

[dim882]

Is it possible to use the regular rethinkdb web ui in the meantime while this is in progress?

[deontologician]

yes, when you do 'hz serve' it will show you the port number for your rethinkdb admin ui

On Sun, Jul 17, 2016, 11:19 Dan Imal notifications@github.com wrote:

Is it possible to use the regular rethinkdb web ui in the meantime while this is in progress?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rethinkdb/horizon/issues/154#issuecomment-233195799, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAFVlQzkbSh_qOJlOZcCjKtFz8yGvPlks5qWnI0gaJpZM4Hl_9u .

[segphault]

@casoetan: This is still a work in progress. I got past the major architectural hurdles that I described earlier in the thread. My current implementation now actively tracks each horizon instance and can automatically cull records from instances that are no longer sending a heartbeat to the database. I'm currently working on adding support for managing permissions and groups. I don't have any realistic timeline for when this will be available, but it's moving forward.

@mrdanimal: yes, you can use RethinkDB's standard admin UI to manage the underlying tables that Horizon uses. The data explorer, sharding/replication, table management, etc all work exactly as expected.

[lavvy]

Any update? Is there a feature of adding custom reql queries through the web ui

[jhicken]

@segphault are you working on a branch? or a separate project? Can I help? Is there a list of outstanding tasks?

[segphault]

I'm not actively working on this feature at the moment. Due to all of the code churn in Horizon, I've had to rewrite the dashboard's backend integration code three times now. I've been waiting for Horizon plugins to land, because I believe that the plugin system will make it dramatically easier to build and maintain the instrumentation and custom queries that are needed to support this kind of dashboard.

You can find the current proof-of-concept implementation in the ryan_dashboard branch. It's very rough right now and will definitely not be adaptable to current or future versions of Horizon.

I'm currently prioritizing my work on Basejump, but I might revisit the Horizon dashboard at some point after plugins land. If others want to take on this project, I'd strongly encourage you to wait for plugins to land. While some of the queries and logic in my branch might prove instructive, you probably won't be able to repurpose much of the existing code.

[jhicken]

@segphault Thanks thats a very helpful update.

On a side note I saw your presentation on basejump. I am super interested in it. But I looks like you have only made the router public. Is the UI not public for the same reasons you mentioned above? specifically... the churn? I would also be interested in pitching in on that ui.

Sorry everyone for kinda veering off topic.

[emeagenciadigital]

is this dead?