Future of HTML Interface

[ebolyen]

Hey All,

Let's start the discussion of where the HTML Interface should go, and what it is going to be used for. Where do we draw the line of functionality and how much overlap is acceptable between QiiTa if any?

https://github.com/qiime/QiiTa

[josenavas]

My $0.02: I think that QiiTa will be able to take advantage of the functionality that pyqi will have already implemented. However, I think that the goals will be different: while piqy's HTML interface aims for a standalone HTML interface, in the case of QiiTa we need a interface that can be embedded in a more complex one. I don't know how this will impact the future of the HTML interface, but I think this is something that has to be cleared up from the very beginning, so we can re-design the HTML interface in a way that both projects can take advantage.

@squirrelo, your input here will be really useful, since you are the one that have design the user interface of QiiTa-pet. Having a list of what you really need to include in your interface can highly derive the design.

[antgonza]

I think the overlap is on how the output of the workflows will be shown but nothing else, right? So if that is the case the only thing we need is to have pyqi/Qiita be aware of the session variables (or whatever we use to control sessions). In fact, it will be great if there were an easy to plug output from pyqi. The rest of the Qiita HTML/web-services shouldn't change.

[josenavas]

@antgonza I don't get your point of pyqi/Qiita be aware of session variables. Can you elaborate?

[antgonza]

Basically, you will only display the results from userA to userA and userB shouldn't be aware of them, right? So for that to happen the pages should be "aware" of the session and who created who. Note that this doesn't mean databases or whatever, simply using a validate display function that returns a boolean that can be plugged to any other code should suffice.

[squirrelo]

From my current working with the pyqi web interface, it seems there are two fundamentally different goals going on here: a standalone interface built into pyqi and the ability to generate html pages from pyqi that can be used for another webserver. In an ideal setup, the pyqi script would generate a page that contains form inputs for all options, automatically set to default values if available. I don't know if this is currently possible, but would be nice for the future.

[josenavas]

@antgonza I see your point but I think this is something that Qiita-pet has to take care of, and pyqi shouldn't know anything about it. IMO, Qiita will take only advantage of the HTML forms generation, as pointed by @squirrelo

Being able to show the results directly on Qiita will depend on the output of the executed command. If it already output an html page (as emperor) it'll be trivial. Other kind of outputs will need a bit of work.

[antgonza]

That's fine so really what pyqi needs to create is something similar to what core_diversity_analyses.py does now, right?

[josenavas]

For the output I think that'd work.

I was more concerned about the parameter input for the commands, which I think its the challenging part of Qiita-pet and where it can really take advantage of the HTML interface.

[squirrelo]

That exactly. With this ideal setup I can just regenerate the options page any time the backend pyqi stuff is changed with little fuss.

[josenavas]

Well, the key thing of pyqi is that it generates the HTML on the fly, so you won't need to regenerate anything. You'll be always requesting them to pyqi.

[squirrelo]

I keep forgetting that. Even better. How much overhead does that add though?

[josenavas]

I think that the overhead will be negligible. I was able to generate the XML pages for Galaxy (which essentially performs the same steps) in ms for all QIIME scripts.

[ebolyen]

So from what I am gathering, QiiTa has some use for generating input forms from a pyqi config. However, I am not clear on how QiiTa intends to 'intercept' the values from pyqi, as pyqi also generally acts as a driver for the commands.

Just so we are clear, any current implementation of the HTML Interface isn't particularly valuable to reference. Because as it stands, it cannot survive a router between host and browser. At the very least, it needs a way to handle a job queue and provide data on when jobs are finished.

In my mind, I imagine the ultimate form of the HTML Interface would look like a visual drag and drop unix shell, where commands can be sequenced to run. This would require either dependency injection or native support for the following:

• Storage for execution state
• Parallelization of an Execution Context
• A data-service, rather than direct html as output

Now, this sounds a lot like what QiiTa attempts to accomplish, so lets suppose that this is an unreasonable goal for the HTML Interface

Instead we might choose to allow the HTML Interface to exist as a 'simple' version in which there is no sequencing of commands. This would still require:

• Storage of execution state
• A concept of Execution Context

Neither of these things make a great deal of sense in the context of PyQi unless we chose to allow PyQi to fundamentally handle parallelization in which case, the ultimate form is not necessarily any more difficult.

Assuming we decide that it is not PyQi's job to care about where or how commands are executed, then we are left with an HTML Interface which is similar to what we have now. Additions that make sense would be:

• Templating Support

It should probably be noted that if execution state cannot be stored, then it will never be possible for the HTML Interface to operate beyond localhost except for trivial operations as a timeout delay would occur, or if heartbeats were sent, the user will invariable close the browser anyways.

[josenavas]

For Qiita, I think that the only functionality that we really need is a 'make_html' command, in which we pass the command that we are looking for and it returns the HTML interface. Then Qiita will call the commands directly, providing the inputs as needed.

AFAIK, one of the goals of pyqi is to provide a standard API, which is encoded under the commands folder. That means that we don't really need to use pyqi to call them. Is that right or I'm misunderstanding something?

[ebolyen]

I think that the only functionality that we really need is a 'make_html' command, in which we pass the command that we are looking for and it returns the HTML interface

What do you mean by HTML interface here? A pyqi config file? The HTMLInterface class? or an HTML-formatted string?

Then Qiita will call the commands directly, providing the inputs as needed.

AFAIK, one of the goals of pyqi is to provide a standard API, which is encoded under the commands folder. That means that we don't really need to use pyqi to call them. Is that right or I'm misunderstanding something?

It is true that you can definitely just call commands directly, although at some point QiiTa will start to look like a standard PyQi interface (with added features). This is where I see the overlap of a Good HTML interface and QiiTa.

@gregcaporaso, @jrrideout, @wasade care to comment?

[josenavas]

Sorry for not being clear: by HTML interface I mean an HTML-formatted string that contains the input form for the commands.

True, but the interface that Qiita will use from pyqi is a small portion of the entire front-end that @squirrelo is developing in Qiita (note that the folder Qiita-pet will contain the entire front-end implementation effort). As @squirrelo pointed out before:

"From my current working with the pyqi web interface, it seems there are two fundamentally different goals going on here: a standalone interface built into pyqi and the ability to generate html pages from pyqi that can be used for another webserver. "

I've also re-read your comment about what do you think that the pyqi HTML interface should look like. Although there is a lot of nice stuff to include there, I think that some of your goals are beyond of what an interface is. For example, the parallel execution environment is something that we are going to implement on Qiita-ware, which is the logic part of Qiita, as this is not interface. I'm probably misunderstanding something. If so, probably a call will be good to have...

[wasade]

I see a large overlap with the interaction between the web server, and how Command objects are actually executed, and I think it would be highly beneficial for this logic to have life beyond QiiTa.

[ebolyen]

Ok! As it stands, the current HTML Interface will generate and serve and HTML page from a pyqi html-config file. The HTMLInterface class provides a method command_page_writer which takes a dependecy injection of a write function, this can be basically anything. In the context of the interface, write is <BaseHTTPHandler Instance>.wfile.write but QiiTa can inject anything that takes an argument.

I would agree that many of the goals I have stretch what PyQi is and overlap significantly with your project. However at some point I think we should also realize PyQi provides interfaces, but is not strictly an interface as it also executes them. In this way PyQi acts as a driver and in this vein, management of execution context seems like a reasonable domain of PyQi. And if we accept that, then parallelization is not a particularly different ideal.

However without these concepts, HTML Interface will not be able to move much further than where it is right now. This is also fine. We just need to have this conversation to decide where PyQi will really go.

[ebolyen]

@wasade I agree.

[wasade]

@ebolyen I like the distinctions you just made and completely agree.

[josenavas]

@ebolyen and @wasade thanks for the explanation. Now it makes a lot more sense. I now see the point of including the management of execution context in PyQi.

Have you thought on how do you want to throw the execution in parallel? For example using IPython? I have some code from Qiita prototypes that might be useful.

Note that this will increase the number of dependencies of PyQi and that is probably something that we want to avoid. How about of using a an IPython approach, in which you can specify what are you going to use from IPython?:

pip install ipython[all]

[ebolyen]

I know @wasade has been a proponent of IPython for PyQi, and I have no particular reason to disagree.

[josenavas]

Ok!! I'll be happy to help on this, as I've been already doing some work on it and I was planning to spent some time implementing it on Qiita, so I can just shift the development effort here.

Also, IPython parallel submission is something independent of the HTML interface, right? This way you can also submit them through any other interface.

[ebolyen]

@josenavas Absolutely, parallel would just be something the HTML interface would call upon, but it would be independant.

[wasade]

As an optional dependency. @ebolyen, you were moving previously toward the parallel execution framework being independent of the implementation (e.g., IPython). I'm okay with either, but have found IPython's framework to be friendly and robust. Another way to look at this, is there a specific benefit that we can identify right now in supporting other frameworks? If yes, then we should abstract, if not, then we should focus/optimize for IPython. From the perspective of pyqi, the parallel support would only be available if the necessary underlying frameworks were available (i.e., optional dependency).

Thinking slightly further, it is great that pyqi is very light but there are other possible benefits to a requirement of IPython. In addition, it may be beneficial to centralize the full execution environment including serial commands, which could get touchy if IPython is optional.

[ebolyen]

@wasade I am not entirely convinced either way. There is some benefit to using optional dependency from a user perspective (This would also allow different injections, Torque, IPython, Celery, etc..). However just biting the bullet for a straight IPython dependency would make our lives quite a bit easier and may be better anyways.

[wasade]

IPython parallel supports Torque and SGE. Celery is a beautiful package though, we moved away from it for qiita though as it wasn't as well suited for large compute

On Thu, Feb 13, 2014 at 11:55 AM, Evan Bolyen notifications@github.comwrote:

@wasade https://github.com/wasade I am not entirely convinced either way. There is some benefit to using optional dependency from a user perspective (This would also allow different injections, Torque, IPython, Celery, etc..). However just biting the bullet for a straight IPython dependency would make our lives quite a bit easier and may be better anyways.

Reply to this email directly or view it on GitHubhttps://github.com/bipy/pyqi/issues/254#issuecomment-35012207 .

[josenavas]

My vote goes for IPython. It can be set up to work with a PBS backend, so Torque will be also covered. Some of the work that we've done with Celery shows that it's not that useful as one would expect. @squirrelo found problems using it.

[ebolyen]

Sounds good to me.

[ebolyen]

Here is a new issue to discuss parallelization specifically: #255

[josenavas]

Thanks @ebolyen!

[wasade]

Great, thanks

On Thursday, February 13, 2014, josenavas notifications@github.com wrote:

Thanks @ebolyen https://github.com/ebolyen!

Reply to this email directly or view it on GitHubhttps://github.com/bipy/pyqi/issues/254#issuecomment-35030686 .

[gregcaporaso]

Thanks for all of the discussion everyone. It sounds like we have a plan here, but could someone summarize exactly what it is? I am also fine with an IPython dependency in pyqi.

[wasade]

My impression is that the first goal should be an execution context that supports both serial and parallel processing (i.e., serial Commands still run via IPython parallel). I believe that once this more robust model is in place, that we can then layer richer interfaces on top.

[ebolyen]

@wasade That is my impression as well.

[josenavas]

:+1:

[gregcaporaso]

OK, sounds good.

On Fri, Feb 14, 2014 at 3:57 PM, josenavas notifications@github.com wrote:

[image: :+1:]

Reply to this email directly or view it on GitHubhttps://github.com/biocore/pyqi/issues/254#issuecomment-35133424 .