[Architecture] Decide on scheme for OF + Browser

[robotconscience]

@quinkennedy Starting convo here to keep track of prototypes!

Tested so far:

• CEF
  • Pros:
  • Full version of Chromium
  • Supports all modern browser stuff (WebGL, etc)
  • Stable
  • Cons
  • OF support is so-so, WIP
  • Mediocre framerate in OF, not sure why yet
  • Not the easiest to use
  • Notes
  • Maybe could work better as a more parallel-process situation, i.e. not trying to draw as texture in OF?
• WebView in OF (using this)
  • Pros
  • Simple, not hack-y way of OF + browser
  • Decent framerate (still not amazing)
  • Cons
  • Does not support WebGL
  • OS X only (probably fine for us)
• WebView via Interface Builder in OF (using ofxCocoaGLView)
  • Pros
  • Same as WebView, but with Interface Builder which is nice
  • Supports WebGL!
  • Cons
  • OS X only (probably fine for us)
  • Kind of a weird, non-typical way of building OF apps (pure cocoa ofApp, for example)
• Syphon + CamTwist
  • Pros
  • Simple, easy to use
  • Cons
  • Occasionally drops, which is unacceptable
  • TBD on future support of CamTwist
  • Latency is noticeable
• Websocket stream
  • Pros
  • Native JS supported
  • Cons
  • Have to implement some sort of frame queue–latency/quality relationship is not great

Not tested

• Awesomium / Berkelium
  • Are these project still supported?
• FFMpeg stream
  • Some infos out there, could be slightly better than websockets?

[robotconscience]

Update: Awesomium and Berkelium are RIP!

[quinkennedy]

FFMpeg Stream made me think of WebRTC. I remember seeing some native libraries for WebRTC support.

quickest of searches: https://github.com/roxlu/WebRTC

[robotconscience]

Cool, seems worth a look.

Looking today at just making a basic CEF browser with a Syphon overlay, as another route

[robotconscience]

Hm looking like overall stream into browser is still best bet!

I think we want to mix UI elements and the video in a way that'll be pretty tough to do otherwise. Have a few different things I've tried, but I think main options are: a) screw it, back to all OF b) stream away

Experimenting with WS streams as we speaketh.

[quinkennedy]

I can dig up the websocket video stream stuff from Paris if you need. The basic HTML version was pretty simple, I think if we need to re-size the incoming JPG then piping it to WebGL becomes faster... that's just from my memory, and probably changes considerably once you put DOM elements over the WebGL context.

Also, there are actually a few different communication options we have (I think these are all available on OSX):

1. network sockets
2. UNIX sockets
3. named pipes

The latter two would probably require a browser extension

[robotconscience]

Cool, that could be worth a shot!

Websocketpp now works as a header-only lib, so super easy to use with OF et al. Might be simple to start with a more barebones/customized thing instead of going to the more general / wild zone of the streamer.

Let's think on it, talk more next week!

[robotconscience]

@quinkennedy posted two viable options: 1 - WKWebView (apple built-in webview) with Cocoa window version of OF

• Pros:
  • Using very little code around the web view itself, leaning on Apple's built-in stuff
  • Pretty simple to mix OF textures and web views, as view has transparent bg
  • Solid 60fps with WebGL
  • Can pass JS between browser and OF (! through funky WebKit stuff)
• Cons
  • Kind of chunky CSS animations - need to explore further

2 - WebSocket streamer

• Pros
  • We've done this before
  • Higher framerate in browser
  • Decoupled apps
• Cons
  • Memory/buffer control a little out of our hands. Too risky?
  • Definitely introduces at least a little bit of latency OR image compression

[robotconscience]

Link to quick websocket++ streamer: https://github.com/robotconscience/ofx_websocketpp_examples