Make a web control panel

HouzuoGuo commented 10 years ago

Go has excellent built-in support for web development, therefore it makes a lot of sense to develop a tiedot web control panel using Go.

Proposed features:

Overview of server performance and databases
Document management
Index management
Query executor

NoahShen commented 10 years ago

:+1: good idea.

The web control will support to manage tiedot in embedded mode or service mode(like API V3) or both?

alexandrestein commented 10 years ago

Great idea!!! ;-)

HouzuoGuo commented 10 years ago

Thanks guys.

It will make use of tiedot's http service as a start.

Would you like to recommend some simple, elegant templates?

matrixik commented 10 years ago

http://www.freebiesgallery.com/admin-panel-template-html-version/ ?

More to choose from: http://www.webresourcesdepot.com/free-html-admin-templates-for-the-backend-of-your-apps/

HouzuoGuo commented 10 years ago

Thanks buddy, that one looks great, I shall meet my old friend Javascript/HTML again soon.

olekukonko commented 10 years ago

@HouzuoGuo Good Idea but here is what i think

Its best one starts with something very simple
Admin Panel and others is just over the top.
I like to take Inspiration from Genghis (Only Missing feature is graph)
Would prefer to use Semantic UI nice simple elements
Lastly we a simple way to to monitor performance so as not to impact actual database performance

HouzuoGuo commented 10 years ago

Thanks.Genghis UI looks really nice - exactly what I wanted.

olekukonko commented 10 years ago

@HouzuoGuo cool .. any update on this ?

HouzuoGuo commented 10 years ago

This weekend I should have some spare time to work on it.

olekukonko commented 10 years ago

@HouzuoGuo i would like to contribute to the control panel if you don't mind.

HouzuoGuo commented 10 years ago

I indeed appreciate that, you are now in collaborator list, please feel free to make commits.

olekukonko commented 10 years ago

@HouzuoGuo Thanks. Have you stared the development on the cpanel if not then i would start development this week using Semantic UI

HouzuoGuo commented 10 years ago

I have not made a start yet, please go ahead with it.

Again I sincerely appreciate your contribution.

HouzuoGuo commented 10 years ago

By the way, in the tiedot "nextgen", most HTTP APIs are still compatible, but there are no longer separate APIs for UID operation - all documents have a UID by default.

olekukonko commented 10 years ago

@HouzuoGuo have been really busy and the holidays could not start the UI as promised. I intend to set some time upper week to start .. sorry for the delay

HouzuoGuo commented 10 years ago

Don't worry,no problem at all!

davesters commented 10 years ago

@HouzuoGuo I have used Tiedot DB with a website project I just launched. I had fun using it, and I would like to contribute. If you have not had any luck on the web control panel, I would love to take it on. I am currently looking for a new side project and thought this would be a good chance to contribute to some open source.

Could you let me know if there is any status by anyone else on this? Is this something I can help out with?

Do you have any thoughts on this. I did see your comment on https://github.com/HouzuoGuo/tiedot-webcp/issues/1 That you planned to make it a part of the main executable. Let me know if you had any specific guidance, or else I could just jump right in and then update you with my plans.

Thanks!

Great DB by the way! It was fun using it, and I would love to help improve it with this and in other ways. I am new to building document DBs, but I have always had an interest, and I love working in Go, so would be good to have a Go specific document DB I could use on a regular basis.

HouzuoGuo commented 10 years ago

@davesters I am very glad to hear that tiedot could be of useful to you, and thanks very much for the initiative! I thought about making a better "front page" for tiedo - a simple page simialr to http://www.serfdom.io because majority of tiedot users are embedding tiedot into their softwares, therefore making a control panel could be of less priority compare to having a fancier front page to make tiedot more appealing. What do you think?

davesters commented 10 years ago

@HouzuoGuo I agree that it would be good to have a nice front page for tiedot, I still think a web control panel is important. It was the one thing I wished I had while I was working on my app.

However, I have no problem working on this as a start, however, I am not much of a designer for creating appealing home pages. Besides your link above, do you have any other kind of a design?

HouzuoGuo commented 10 years ago

@davesters ok! let's go with your preference. What features would you like to support in a web CP? Nextgen branch is near feature-complete, and HTTP API will not take much longer to get there, then we can make a start.

davesters commented 10 years ago

@HouzuoGuo Awesome, that sounds great. I am excited to start working on it.

I think the best thing is to start simple. Perhaps only supporting the basic operations. Here are my thoughts and questions. Let me know what you think and I can get started. I can tweak with any changes to the API as we go along.

Start simple. I really like the Genghis App. That might be a good reference for the simplicity of the UI. I also really like the RavenDB studio interface. It is simple as well and easy to use. I agree that the admin themes linked to above are a bit overkill for something like this.
I would probably use something like Bootstrap or Foundation for the UI framework. Semantic UI looks nice, but I am already familiar with these two frameworks.
I am thinking of just getting it up and running with the basic CRUD operations on documents and collections at first. Then once it is all working, I can start adding things like dealing with indexes or anything else extra.

Questions:

Should I build it all within a sub-folder of the app root folder? Maybe like a "web" folder? Then I can trigger it all off a single base URL like "/admin". So, if they visit "/admin" while it is running in http mode, it goes to the control panel instead of an HTTP API call.
Does it make sense to build the control panel as a single page app? Since once it loads it seems like it would be able to all run off of the API.

I think that is all I have for now. Let me know what you think.

HouzuoGuo commented 10 years ago

@davesters That's great, and thanks very much again.

Talking about genghis app, I'd be sooo happy if tiedot could get a web CP as beautiful as that one.

Are you familiar with AngularJS, or do you feel adventurous about giving AngularJS a try?

I have written down the HTTP API implementations and will document them over the weekend.

BTW your commit access has been enabled.

HouzuoGuo commented 10 years ago

Regarding directory name - "admin" sounds pretty good. Being a simple app, I think a single page should do it.

matrixik commented 10 years ago

I don't want to sound pessimistic but aren't AngularJS too heavy (and slow) for site like this?

Test yourself: http://jsperf.com/angular-vs-knockout-vs-ember/308 http://lhorie.github.io/mithril/benchmarks.html

davesters commented 10 years ago

I agree that Angular and other frameworks like Ember are too heavy and slow in general for my tastes. However, I don't think the size of the framework really matters in this case, since it is an admin interface that would, for the most part, only be running on your local machine.

That being said, I don't really have a preference for the framework being used. I have used Backbone a few times in the past so I am most familiar with it, and I know a little about Emberjs. Never used Angular or Mithril or Knockout before. I am not opposed to any one of them. Mithril does look nice though. There are so many out there, we could debate for days on it.

I would probably lean towards either Backbone or Angular (since HouzuoGuo mentioned it) with jQuery of course. They are both well-suited for single page apps.

p.s. "admin" sounds good to me for the directory name. I am going to start laying things out.

davesters commented 10 years ago

Actually, I looked at Mithtil, and that one looks pretty nice! Very lightweight, good benchmark scores. I wouldn't mind using that one either for the control panel.

Another consideration when using a framework, is typically, you have lots of javascript files, and then these get combined and minified into a single JS file. However, I am not sure what we should do here.

Do we just reference each file individually in the page since it is running locally and does not matter?

Are there Go tools that will compile and minify these files for us? Do we want a dependency like that in this project?

Do I just minify them separately on my own and then just include the minified version into the repo?

I could also use something like requirejs possibly to do the loading for us in the browser.

Thoughts?

HouzuoGuo commented 10 years ago

All right, are Backbone + Bootstrap a good combo?

I wouldn't be concerned about JS size at the moment.

eduncan911 commented 10 years ago

As for directory naming, please don't use /admin. We use that for production admin sites.

How about taking a cue from MySql admin and create one specific to the project name?

/tiedotadmin/

?

Also, i'd make it optional for the user to register rather they want to expose it or not.

Lastly, what kind of security were y'all thinking of? Restricting it IP based would be a request of mine, along with a u/p obviously.

Thnks for the suggestion!

davesters commented 10 years ago

That is a good point. My initial thinking was that this would be running locally on your machine as from what I understand is that most users run the db embedded, and this would not be available from the embedded version.

In the case your were running the db in production in http mode, it would need auth and perhaps a more unique URL. Perhaps the auth work is something that can be done in the future, and for the first iteration we could add another command line param to activate the control panel when running the db that is separate from the regular http command line param. Or, it runs on a diff port so it is not exposed on same port as http API.

I hope that makes sense. Thoughts?

eduncan911 commented 10 years ago

Running on a different port would imply there are two http instances running. In Windows Azure Web Sites, for example, all we have access to is port 80.

I would think that all you'd have to do is register a bunch of handlers upon a simple method - to use whatever existing http instance the user already has. Something like:

package main

import ( 
    "net/http"
    ...
)

func main() {
    # I haven't looked into tiedot's API yet, so I am not sure what tiedotApi is called
    #
    # was thinking of just passing the pointer to &http.Server
    tiedotApi.RegisterCP(&http, ...)
}

That way, you take in the &http.Server pointer the dev is using - either from a raw http program, or a framework like Gorilla or Web.Go that still uses the underlying http framework. Using the underlying http instance would allow the developer to continue using those frameworks, while passing in the base http object.

# within the tiedotApi package
package tiedotApi

import ( 
    "net/http"
)

func (api *TiedotApi) RegisterCP(web *http.Server) error {
    # then use the pointer of *http.Server to register a bunch of handles
    web.HandleFunc("/tiedotcp/", homeHandler)
    web.HandleFunc("/tiedotcp/index/view", viewIndexesHandler)
    web.HandleFunc("/tiedotcp/query/run", runQueryHandler)
}

func homeHandler(w http.ResponseWriter, r *http.Request) {
    ...
}

func viewIndexesHandler(w http.ResponseWriter, r *http.Request) {
    ...
}

func runQueryHandler(w http.ResponseWriter, r *http.Request) {
    ...
}

PS, I am still working on my first Go project. Apologizes for "WTF is that code?" above if it isn't correct. :8ball:

davesters commented 10 years ago

@eduncan911 I think I see what you are getting at, and that is definitely a possibility. I like this idea. The only thing with this way of doing it is that the user would need to include a copy of the control panel folder in their app. In that case, I guess it would be kinda similar to PhpMyAdmin, where the user downloads and runs a copy of the entire control panel app on their server. No immediate issues pop out to me in that method. Perhaps the user would also specify a login or something that is necessary when connecting to the control panel, and we can also let the user specify the base URL too with this method.

However, I think for the beginning, I just want to start simple and get something built and working first. Then we can discuss more and figure out next steps. For right now, I am just planning to build it to run when passing in a command line arg when running tiedot locally (not embedded).

And btw, I am new to Go as well and still learning. I have only built one app in it, but I really like it so far.

HouzuoGuo commented 10 years ago

Thanks very much gentlemen. TIL :8ball: hahaha

My original thought was to serve web control panel under the HTTP API port, in a path such as /admin. ElasticSearch uses the idea to serve both of its restful APIs and plugins - some of which are control panel/monitoring applications.

In the end it'll look like:

(HTTP port 10993) /create (collection API) /insert (document API) ... /admin (HTML page)

Therefore it may not be necessary to have two HTTP instances running, one for API and the other for static HTMLs.

If I got your idea correctly @eduncan911 , do you wish to run the web control panel as a stand-alone application? If so, how about using the following CLI options to customize the CP HTTP path?

./tiedot -mode=httpd -port=10993 -webcp=tdadmin (run HTTP API and CP) ./tiedot -mode=webcp -port=10993 -webcp=wonderful_cp (run CP only)

So that web CP is served under "/admin" (or some other path) by default, and user may use CLI option to change the path.

davesters commented 10 years ago

@HouzuoGuo That is more or less what I was planning based on your previous comments.

Along with being able to specify the URL in the command line, it may be good to also have a way to disable the control panel while running in httpd mode via the command line as well. Or is that possible from your example above by not specifying the "-webcp" option?

On another note, I have already started work on this. I am just in the beginning phase of working out the general layout and getting the main page created. I am doing the work in a branch off of nextgen branch.

Quick note: Nextgen branch has a compile time error on windows 32 bit go. It's fine on 64 bit. Hashtable.go has an int overload compile time error. If I recall, it was like line 32 or 34.

eduncan911 commented 10 years ago

I think I need to actually use tiedot and the command line to see what y'all are talking about. :) It's hard for me to vision and comment on the CL version.

@HouzuoGuo I was coming from a website angle, knowing I have a site (e.g. net/http.Server) already and just wanted to extend it.

But I do see where having it separate, on another port could benefit security by not exposing it to the masses.

HouzuoGuo commented 10 years ago

@davesters thanks for noticing it, tiedot 3.0 probably isn't gonna work on 32-bit platform, I should add a build constraint to only build on 64-bit platforms.

So to summarize our discussions, we're gonna implement these features as a start:

Run web CP as a standalone application, and be able to modify application root URL
Run web CP along with HTTP API, and be able to modify application root URL
Run HTTP API without web CP

Port number is customizable as well, in all of the three scenarios.

The command line will look like:

./tiedot -mode=webcp -port=19093 -webcp=tiedot_webcp
./tiedot -mode=http -port=19093 -webcp=tiedot_webcp
./tiedot -mode=http -port=19093 -webcp=none

By default (if -webcp is not specified), web CP will be accessible on /admin

Sounds good?

I look forward to learn bootstrap and backbone js =)

davesters commented 10 years ago

@HouzuoGuo Thanks for the summary. I am already working on this as mentioned above.

One quick note: I found a nice Go package to embed the webcp files directly into the binary. I think this would be necessary as people would probably be using "go install" on tiedot, or wanting to embed the webcp into their existing apps. When running tiedot from any other folder than the project root, it is unable to find the web files. So, this way, the static website files can be embedded directly into the binary. There would not be any extra work on the user's end, but when making changes/updates to the webcp files, a command would need to be run to translate the static files into go files that can be checked into the repo.

I am using this package right now. https://github.com/GeertJohan/go.rice

It works really well. During development, you don't need to compile the static assets into the binary, and only do it for a final build. Then the webcp can be run from any directory, and it just works.

What do you think? Is there a better way to do this?

matrixik commented 10 years ago

go.rice is great, I convicted alpaca maintainer to move from go-bindata to this one.

One warning: appending files don't work on Windows, but rice embed works fine.

HouzuoGuo commented 10 years ago

Thanks gentlemen, gorice looks nice

davesters commented 10 years ago

@matrixik Thanks for the endorsement. So far go.rice seems to work pretty well. I tested out several of the options. I saw about the appending not working on windows, but I was imagining we would be using the embed feature, and that works just fine on windows.

@HouzouGou I am getting close to having a running version. I want to push my branch up this week so you can look at the code so far. I will let you know when it is there.

Thanks!

davesters commented 10 years ago

@HouzuoGuo I just pushed the control-panel branch to the repo for you to look at. https://github.com/HouzuoGuo/tiedot/tree/control-panel

Please take a look when you get a chance and let me know what you think. Right now, it has pretty much all the functionality, except for a couple of the API calls. It probably has a few bugs, but it seems to work pretty good so far. I tried to keep the interface as simple as possible.

Let me know if you want to make any tweaks or changes. Right now, it only supports the -mode=httpd -webcp=admin arguments. Of course, the "admin" can be anything. It does not support the webcp mode only.

Note - You may need to get the go get the go.rice package to build the branch. If you just run the executable from the root folder, it should just work. If you want to build it with go install, you will need to install the rice package, then run rice embed from within the httpapi folder so it can generate the go asset files, then you can run go install like normal from root folder. Let me know if you have any issues.

Thanks!

HouzuoGuo commented 10 years ago

Hey man, that looks really pretty and awesome, runs smooth as butter!

The only minor problem I ran into is in gorice's source code, box.go line 29 says:

"don't allow dots in box name (they will mess with .syso object file names' arch tags)"

The check was introduced about a week ago (on 4th of June). I had to disable the check in order to launch the admin panel.

HouzuoGuo commented 10 years ago

All right, I worked around it by moving static resources underneath "webcp" package and use gorice from there.

The UI is really impressive, and you were absolutely right - I never realized that a control panel application could make database interactions so much more convenient.

Go from there, this is my plan:

Continue working on the remaining features such as collection scrub and display of memory stats.
Support paging of all documents in query processor, so that they may be presented to user in collection overview page.
(Related to paging documents) Display estimated document count
Allow connecting to another tiedot database
Show equivalent HTTP API call in the page, similar to "Equivalent REST Response" button on Google Cloud UI

davesters commented 10 years ago

@HouzuoGuo Thanks for the feedback! I saw your updates last night when I pulled down the latest to my machine. I am going to look through it again tonight and then do some more work on the last few pieces and the items in your list.

One comment: I think we should wait on the "Allow connecting to another tiedot database". It seems like this could be related to the "-mode=webcp" option you wanted, and I have some questions on that. Or, what exactly did you have in mind here? This may be good to discuss in a separate issue, because this webcp only mode requires quite a bit more work than just the control panel working over HTTP API. Unless you had other ideas of how that webcp only mode should work.

HouzuoGuo commented 10 years ago

No problem at all. Right now the web cp is definitely in a good shape and very useful. I'll merge it into master this weekend.

HouzuoGuo commented 10 years ago

Sorry, on this weekend I haven't been able to make enough progress to merge cp branch to master.

The only thing remaining is to improve query processor to page documents and return.

HouzuoGuo commented 10 years ago

@davesters Hey David, could you please give me a hand? I made two new HTTP APIs for scrolling through a collection's documents:

/approxdoccount?col=A It returns an estimation of how many documents there are in a collection. It is more accurate when there are 1000+ documents.
/getpage?col=A&page=5&total=20 It divides the entire collection A into roughly equally sized 20 pages, and returns the documents on 5th page.

So if the collection has an estimate of 1000 documents, and user wants to see the 3rd of 20 pages, the getpage call may return between 40-60 documents. Page number begins from 0.

I want to implement the following: When user browses a collection, he/she may scroll through all documents by choosing a page number. Every page contains approximately 50 documents - if there is enough room for UI (what do you think)?

Therefore, in the beginning approxdoccount is called to determine how many page links are to be presented; for example, there are 1000 documents, so there are 20 links:

Browse: [1] [2] [3] [4] ... [17] [18] [19] [20]

(Note that on UI the page number begins from 1)

User clicks on a page number, the control panel calls getpage to retrieve the documents.

I gave bootstrap a try, but I'm too unfamiliar with it to make the idea happen. Could you give me a hand please?

davesters commented 10 years ago

@HouzuoGuo Sorry for the late reply. I have been kinda busy the past week or so. I just got the latest from the repo and I am going to start work on these new items. I will report back when I push new updates to origin.

HouzuoGuo commented 10 years ago

@davesters thanks very much & there's no hurry.

By the way, in order to generate a collection of 140k+ documents for UI testing, there is now an option in benchmark command:

./tiedot -mode=bench -benchcleanup=false

That'll make a collection of 140k+ documents under /tmp/tiedot_bench. To use it in webCP, just run:

./tiedot -mode=httpd -dir=/tmp/tiedot_bench

An attempt to view the collection crashes my browser =) perhaps there're too many documents being rendered hehe..

davesters commented 10 years ago

@HouzuoGuo I just pushed some more commits to add the paging to documents.

HouzuoGuo commented 10 years ago

Good news - I have merged control-panel branch into master branch. Over the weekend I shall complete the documentation of new HTTP APIs and write an introduction to the control panel UI.

HouzuoGuo / tiedot

Make a web control panel #34