Web frontend

[mottosso]

Goal

To facilitate collaboration and debugging by visualising publishing activity.

Dashboard

The very first thing a user is exposed to is a summary of events having taken place during the last couple of hours/minutes.

Each summary may also provide information about an ongoing publish, provided each plugin emits relevant information to the database during processing.

Clicking on a summary expands it to a Thumbnail View with some additional information. Clicking on the "maximize" button opens up the Details View.

Details View

The Details View exposes all available data with each publish and is meant as a means of debugging publishes that go wrong.

It can also be used as a minimal "facebook wall" featuring comments and interactive playback of video and possibly annotation on videos to articulate ideas.

Comments

Each publish features a commenting panel using something like Disqus. This way, publishing can be used as a minimal review system in which supervisors leave comments directly onto an asset that was published.

The information can then be re-used in asset libraries, dailies/review systems or pop up at an artists workstation as comments are being made.

Console

A full stack-trace of each executed line is visualised so as to facilitate debugging. Supports folding of blocks of code and inspection of individual variables.

Log

Inspect logging messages produced during the publish and filter by level/severity.

Images & Video

Media produced during a publish can be visualised here.

3D

An automated generation of a minimal version of 3d meshes/pointcaches produced during publish can also be visualised here, using standard WebGL and Alembic.

Environment

Inspect the exact settings used prior to running a publish. This can be useful when debugging to find out exactly why a particular error occured.

File Browser

For files that can't be natively visualised by the front-end, there is the browser. Inspect file names, sizes and their location on the network.

Architecture

A centrally accessible computer is running a networked database, such as Firebase to which each publish communicates with during conform/integration.

When the database receives an event, the event is pushed to a Python Web Application server, such as Flask. Once flask receives the event, it produces an Event Widget that is then pushed live to each session, so that users get updates in real-time without having to refresh their browsers.

Data

Initially, the front-end merely displays the limited information available during each publish, such as who, when and what.

Additional plugins may then augment the supplied data with additional information, such as a stacktrace, logging or videos and images.

The web server then listens for particular data members of an event and produces widgets depending on what they are; e.g. a video link is drawn as a media player whereas a stack-trace is drawn with pygments-formatted source code.

Conclusion

The Web Front-End is a useful companion to the native UI used by artist when initiating a publish. The front-end facilitates distributed and background publishing and enables users to gain an overarching view of what is happening under the hood of their pipeline, along with facilitating collaborative review and feeback on a smaller scale.

Linked Events

As discussed in #113, one publish may trigger more publishes.

The font-end could provide the means to visualise these dependencies in the form of a node-graph; each node linking back to it's corresponding event in the list of events on the front-page and thus provide an equally detailed report of each publish whilst also visualising its dependencies to other events.

[mottosso]

We need web-developers for this! If you know anyone, do point them to this issue.

Best, Marcus

[mottosso]

Got suggested by @chadmv on the python-inside-maya mailing list to use:

• Bootstrap for front-end
• Django for back-end

As a non-web developer, looks to me like Bootstrap has some neat-looking widgets we could utilise, along with behaviours like animation and transitions. Django looks well-known and stable and could possibly be beneficial to use due to its large userbase. We should be able to get help on Stackoverflow and such.

I've only had (minimal) experience with Flask before and quite like it for its simplicity but can certainly see the benefits of going with something, what looks like, more established.

@ljkart, you mentioned having worked with Django before, what are your thoughts on this?

[ljkart]

Yeah that looks fine for me, good to start. On Sep 20, 2014 11:04 PM, "Marcus Ottosson" notifications@github.com wrote:

Got suggested by @chadmv https://github.com/chadmv on the python-inside-maya https://groups.google.com/forum/#!topic/python_inside_maya/KF2wYEdqe5g mailing list to use:

• Bootstrap http://getbootstrap.com/ for front-end
• Django https://www.djangoproject.com/ for back-end

As a non-web developer, looks to me like Bootstrap has some neat-looking widgets we could utilise, along with behaviours like animation and transitions. Django looks well-known and stable and could possibly be beneficial to use due to its large userbase. We should be able to get help on Stackoverflow and such.

I've only had (minimal) experience with Flask http://flask.pocoo.org/ before and quite like it for its simplicity but can certainly see the benefits of going with something, what looks like, more established.

@ljkart https://github.com/ljkart, you mentioned having worked with Django before, what are your thoughts on this?

— Reply to this email directly or view it on GitHub https://github.com/abstractfactory/pyblish/issues/99#issuecomment-56274504 .

[eoghancunneen]

Angular.js and Bootstrap might be a good start. Angular can play nicely with Firebase, apparently, and there's a lot of Angular/Bootstrap dev done as well.

[eoghancunneen]

Marcus, Shotgun manages a lot of this. Are you planning any Shotgun hooks?

[mottosso]

Thanks, @eoghancunneen, glad you could make it. I've only briefly glanced at angular.js, what part would it be suited for? Frontend or backend or both? Would it be an alternative to Django/Flask?

Shotgun manages a lot of this. Are you planning any Shotgun hooks?

I certainly am, but initially focusing on minimal, independent functionality useful for those without any existing intranet backends, such as smaller commercial houses.

Both this and a Shotgun integration will end up as standalone packages of Pyblish, like the Maya integration. We should grab a virtual coffee sometime and run through how to get that off the ground!

[mottosso]

@eoghancunneen Angular.js doesn't seem to involve Python. Considering that most everything else about the project is built using Python, would you still consider a Javascript library such as Angular.js optimal for this task, as opposed to a pure-Python framework, such as Django or Flask?

[eoghancunneen]

I’ll be back in London in a couple of weeks, so we can grab a real one!

On 20 Sep 2014, at 17:33, Marcus Ottosson notifications@github.com wrote:

Thanks, Eoghan, glad you could make it. I've only briefly glanced at angular.js, what part would it be suited for? Frontend or backend or both? Would it be an alternative to Django/Flask?

Shotgun manages a lot of this. Are you planning any Shotgun hooks?

I certainly am, but initially focusing on minimal, independent functionality useful for those without any existing intranet backends, such as smaller commercial houses.

Both this and a Shotgun integration will end up as standalone packages of Pyblish, like the Maya integration. We should grab a virtual coffee sometime and run through how to get that off the ground!

— Reply to this email directly or view it on GitHub.

[mottosso]

Sweeet, looking forward to it!

[Byron]

Angular.js is useful for interactive web applications, running in the browser directly. The backend would commonly be some sort of REST service, or/and something compatible to websockets to allow implementing your own protocol.

My approach would be to implement a python-powered REST interface to serve and possibly manipulate the pyblish data, alongside a simple python http server to serve front-end resources.As a second step, one can use a web-socket to distribute information about changes in real-time. That way, a change made by one user will be distributed to all others right away. In the front-end, angular.js is used together with angular-bootstrap for some more widgets. It's worth noting though that when using angular, you will be best off using angular-compatible widgets. Otherwise, you will have to write adapters to make some particular javascript library interact neatly with angular.

Using a big web backend, like django, is a dependency I wouldn't want to pull in, just because all the logic ends up in the frontend anyway when using angular.js .

This is my two cents about it, with the proposed usage I have made very good experiences already which is why I can recommend it without any limitation.

[mottosso]

Thanks, Sebastian!

As a second step, one can use a web-socket to distribute information about changes in real-time. That way, a change made by one user will be distributed to all others right away.

What do you think about users polling for updates instead?

My motivation lies in reducing complexity as Push notifications seem rather complex. At least more-so than polling every second or two for changes, as I could do that with a Javascript timer in the frontend.

Using a big web backend, like django, is a dependency I wouldn't want to pull in, just because all the logic ends up in the frontend anyway when using angular.js .

What do you think about providing it as a bundled vendor package instead?

This is my two cents about it, with the proposed usage I have made very good experiences already which is why I can recommend it without any limitation.

You've made something like this already? That's great! Have a demo?

[Byron]

What do you think about users polling for updates instead?

Polling will certainly do initially, especially if bandwidth and server-performance are not a concern. Implementing a notification system with websockets is astonishingly easy to do, and even though it's another layer of complexity, the layer itself is not a particularly complex one. Provided of course, there is a web-socket implementation in python already, which might end up being another dependency that you have to provide when shipping. All in all, polling is certainly the easiest, if more resource intensive way to implement it.

What do you think about providing it as a bundled vendor package instead?

Questions about distributing dependencies wouldn't really be my concern here, as it will have to be done the pyblish way in any case.

You've made something like this already? That's great! Have a demo?

Yes, it should be a simple as downloading the latest godi-web release and giving the binary a double-click. My suggestion is to ignore the downright fhantastic web-design, and open a few tabs possibly in different browsers and/or on different devices.

[mottosso]

All in all, polling is certainly the easiest, if more resource intensive way to implement it.

Mm, I hear ya, but I'm considering that this utility is to be run on an intranet and never reach more than say a hundred users at any single point in time. Although I'm sure polling is well suited for thousands of requests as well, though perhaps not millions.

Yes, it should be a simple as downloading the latest godi-web release and giving the binary a double-click. My suggestion is to ignore the downright fhantastic web-design, and open a few tabs possibly in different browsers and/or on different devices.

Will do! Do you think your code is reusable for this purpose? You're under LGPL too right?

[Byron]

Totally agree on that, and considering that shotgun doesn't do what we are talking about (real-time notifications in the web-frontend), it should even be alright to not have this feature at all in the first version.

Will do! Do you think your code is reusable for this purpose? You're under LGPL too right?

It might be a good real-world example for how it can be implemented, but the code certainly can't be copied line by line. License-wise, I see no issues, as it is LGPL indeed, as is pyblish.

[mottosso]

Added section on "Linked events" top top post.

[mottosso]

An update on this.

I've managed to get Flask up and running with Socket.io for real-time communication from server to client and added a minimal API for communicating via RESTful requests, GET and PUT.

• https://github.com/pyblish/pyblish-event

It displays data read from a JSON file. The same file is then augmented with whatever data is submitted via REST so that each event not only updates in real-time but also persists across sessions (i.e. crashes and restarts).

From here on

Transmitting data via Python is equally easy.

from requests import put
put('http://localhost:5000/event', data={'name': 'EventName02'})

Which means that for us to use this in Pyblish may look something like this.

from requests import put
import pyblish.api

class ConformCharacter(pyblish.api.Conformer):
    def process_instance(self, instance):
        # Emit all data available within instance
        put('http://localhost:5000/event', data=instance.data())

Which means that what remains from here on out is styling and interaction, for which I've had a look at Bootstrap for template graphics, seems really great, and AngularJS for interactivity.

If you're interested in helping out, have a look at

• style.sass for styling or
• templates for layout

Best, Marcus

[tokejepsen]

that is some great development. Nice work, Marcus!

[mottosso]

Thanks, Toke.

It's one of those things that was amazingly difficult to figure out, but so easy once you know it. It feels almost silly to take too much credit, I'm really just wiring different frameworks together.

[mottosso]

Big update on the back-end, with little to show for it. :)

• https://github.com/pyblish/pyblish-event

Remains the same

1. Still running Flask
2. Still running Socket.Io

What's new

1. AngularJS is now running the front-end, one tiny visible feature of this is the search box
2. Bootstrap is powering the visuals, as can be seen by the nice font and animated accordions.

Summary

AngularJS works fine for this task and will be responsible for each interactive element in the mock-up above; like visualising images and exploring files. Bootstrap is also very neat, relieving you from having to do much designing; similar to how PyQt ships with widgets that look and feel like the native OS, Bootstrap ships with widgets that look and feel like most websites, including GitHub which may even be using Bootstrap for all I know.

I'll get a demo up on Heroku tomorrow for you to play around with. I think this will also be a good experience for new users; sending publishes to a central demo-app so that everyone can have a go without actually installing anything.

See here for an example I put up the other day.

• http://af-contacts.herokuapp.com

[tokejepsen]

some very cool developments:) Nice work Marcus!

[mottosso]

Thanks Toke!

[mottosso]

An update on this.

Turns out, that to use AngularJS and build web-applications you also need to know HTML, CSS and Javascript. Who knew. So that's what's been happening during these past few days and I'm happy to finally be able to say that I'm now comfortable enough to produce designs of moderate complexity.

Such as this one.

For starters, there won't be very much to visualise, but as we start visualising I'm sure things will start to get clearer in regards to what exactly we are interested in knowing about previous publishes. Ultimately, I'd like this to be a minimal review platform in which artists and supervisors can witness, comment and annotate on media coming out of the various hosts we support; such as Maya, Nuke and Modo (which now have integrations, by the way. Thanks to @tokejepsen for integrating the latter two!).

As always, let me know what you think, and if there is anything else you could think to be visualised in the first round of implementing.

[tokejepsen]

very cool stuff! already looks very interesting, is it a difficult setup?

[mottosso]

I'm just getting into implementing this now, but the back-end should remain the same. The Flask server receives a dictionary with relevant information - such as instance name, date, comment and author. The rest is really up to the app itself; e.g. filtering and sorting.

[mottosso]

Oh, and I'm still thinking we should provide clickable events; to dig deeper into what happened/is happening. Such as a full stack-trace, logging and image viewers and such, as illustrated in the initial post. Though those things must come later, as I've got no plug-ins currently producing any of that.

Ultimately, I'd like for all publishable information to be "visualisable" right here.

[mottosso]

Bam!

• https://pyblish.herokuapp.com/

[tokejepsen]

Wow! Bam indeed:)

I wonder whether having what host the instance is coming from, might be useful?

[mottosso]

Yes! There's lots of room for cool things here. For one, the columns should all be configurable to display whatever data is coming from a publish. Some are defaulted, and then you should be able to select which you'd like to include.

On top of that, there should be a persistence editor in which a user may define his/her own defaults, so that the app can return to a custom view.

The sidebar should also contain some additional filters, possibly of the same values as are available as columns? Like users, hosts, time-ranges etc.

Then, most importantly, to have some introspection on each event, to visualise images, playblasts and such.

It's gonna be amazing! :)

So if you think of stuff, now is the time to throw out ideas!

[mottosso]

Okay, it's operational.

http://event.pyblish.com

• Got a MongoDB backend for persistent events
• Got socket.io for real-time communication
• Got a RESTful interface for communication with Pyblish, and custom programs.

See the API

From here

Next step is making a simple plug-in to encapsulate the API and emit whatever information is relevant to visualise in Event. From there, it should be pretty obvious what needs work and where we'd like to go next.

Try it out

Paste this in a terminal, and see Event update in real-time.

$ curl -H "Content-Type: application/json" -d '{"instance": "GitHub01", "author": "you", "comment": "Published by me, after seeing it on GitHub!", "family": "git.hub"}' http://event.pyblish.com/api

Or run it from Python, as seen in the API.

Let me know what you think!

Best, Marcus

[justinfx]

Do you think it would be good to use an actual restful uri design before getting deeper into building around this current one? Right now you have it hitting /api, but what if you want to provide other functionality in addition to getting and posting event objects?

Maybe your current url should be changed to:

/api/events /api/events/

This would more closely follow Restful uri styles and let you have more than just a generic api endpoint.

http://blog.2partsmagic.com/restful-uri-design/

[mottosso]

Hey Justin,

That's a good point. In that regard, does it make sense to remove the /api prefix altogether?

POST /events
GET /events/1
GET /events/2

Also, I'd expect singular as opposed to plural.

POST /event
GET /event/1
GET /event/2

Saving plural for posting and getting collections of events.

# Get all events
GET /events

# Post multiple events
POST /events

Thus leaving room for other operations at other addresses.

...

Though I can't honestly think of anything else at this point as I've never before used a web-based API. :s

http://blog.2partsmagic.com/restful-uri-design

Thanks for that, that was an interesting read, although he seems a little confused over the difference between a URI and URL and it's relation to, how I understand it, REST which is about HATEOAS and relationships between resources, as opposed to schemes as he seems to make it up to be.

Have you read REST in Practice? What do you think about their advice?

Best, Marcus

[justinfx]

I think keeping the /api namespace is up to you, if you want to be able to separate it from the purely html serving endpoints.

My suggestion would be to stick with one plural endpoint. "events" is where you can list all of them with a GET, or a single with events/123

You may not be able to think of any other functions now, but just like any other typical app you might figure out something layer. Then you will easily be able to create another namespace to serve.

On Sat, 1 Nov 2014 11:07 PM Marcus Ottosson notifications@github.com wrote:

Hey Justin,

That's a good point. In that regard, does it make sense to remove the /api prefix altogether?

POST /events GET /events/1 GET /events/2

Also, I'd expect singular as opposed to plural.

POST /event GET /event/1 GET /event/2

Saving plural for posting and getting collections of events.

Get all events

GET /events

Post multiple events

POST /events

Thus leaving room for other operations at other addresses.

...

Though I can't honestly think of anything else at this point as I've never before used a web-based API. :s

http://blog.2partsmagic.com/restful-uri-design

Thanks for that, that was an interesting read, although he seems a little confused over the difference between a URI and URL and it's relation to, how I understand it, REST which is about HATEOAS and relationships between resources, as opposed to schemes as he seems to make it up to be.

Have you read REST in Practice http://www.amazon.co.uk/REST-Practice-Hypermedia-Systems-Architecture/dp/0596805829? What do you think about their advice?

Best, Marcus

— Reply to this email directly or view it on GitHub https://github.com/abstractfactory/pyblish/issues/99#issuecomment-61363575 .

[mottosso]

My suggestion would be to stick with one plural endpoint. "events" is where you can list all of them with a GET, or a single with events/123

You're referring to the "hackable urls" from the article? Thought I can agree with his point of view, I can't quite get comfortable posting to a single event to a plural address, and then accessing single events from the same plural address.

Though these are surely details at this point.

You may not be able to think of any other functions now, but just like any other typical app you might figure out something layer.

Certainly, though ideally I'd keep it minimal and limited for as long as possible in an attempt not to overbuild.

I think keeping the /api namespace is up to you, if you want to be able to separate it from the purely html serving endpoints.

The way I imagined accessing individual events as html as via the same address as when accessed as json.

/event/1

But maybe that's not a good idea? I'd imagine distributing JSON when the headers says application/json and HTML otherwise. Though I can't think of any example where this is used, or whether or not it's expected from a user point of view. I would think so, as they point to the same resource, and it's format shouldn't be part of the URL?

[justinfx]

Its plural when you access /events GET to list events. It is singular when you call /events/123 to GET a specific by I'd. And it is still consistently one endpoint. But anything is better than /api to cover all communications. I just figured since you said it was a restful api that you were doing a restful api. Whatever works.

[mottosso]

Ah okay.

I suppose my use of "RESTful" at this point is more about me not knowing what else there is, and it's quite likely that a single point of access isn't even enough for it to be called RESTful at all. Appreciate you stepping in and clarifying things, there'll most likely be more things to clarify down the line, so thanks.