search

processing / p5.js-editor

Deprecated desktop editor for p5.js
https://p5js.org
MIT License
338 stars 91 forks source link

Support for real-time, collaborative, direct manipulation design process #189

Closed brylie closed 8 years ago

brylie commented 9 years ago

User story

As a student
I need 'a personal dynamic medium' to express my thoughts
so I gain immediate and intuitive understanding of complex ideas
As a person communicating through rich, interactive media
I would like to directly manipulate my narrative expression without switching context
so that I can create at the speed of thought without having to think, much, like a computer

Principle

Humans should human computers should computer and computers only exist so we can better human

Challenge

Even though the programming environment minimises time-to-comprehension by quickly rendering results when the play button is pressed, there is still significant separation between the visual output and the generative commands.

The challenge is to work together to build a toolkit that provides nearly real-time responsiveness, intuitive interaction, and feedback between system components while avoiding syntax errors.

There are several projects in the same vein, or design space, that are working separately on seemingly different problems. The opportunity here is to combine forces to fill out the picture of a cohesive, visual, direct manipulation interface for creating non-linear, hypermedia narratives.

Feature

Allow end-users to directly interact with geometric primitives, DOM elements, and other P5.js objects, and each other in real-time. People can set parameters on objects from one another, run arbitrary expressions in object parameters, drop 'block' elements on to the canvas, and compose interactive objects that change scene data flow.

Environment

The environment is the container. It can contain information about the state of the exploration, global events, etc. It is the master document into which sub-documents are added and linked, in arbitrary combination. The environment is

a general window that view[s] a large virtual world

-- Alan Kay describing the Sketchpad User Interface

The environment is, in effect, any modern web browser.

Stages / Scenes

Scenes are arranged within the environment, and within one another. They contain elements, both visible and invisible. Scenes and their elements can change over time.

Godot scene graph

Source: Godot engine scene diagram

Linking

Scenes can be linked to one another, in any order. Twine flow diagram interface

Actors / Elements

Elements are like the actors in the scene. They have properties and behaviours that make up their personalities. They interact, sometimes badly, with one another.

Shapes

Complex shapes can be made of simple shapes. People like to work with things directly, so may like to work directly with the shapes.

People can select basic shapes from a palette, drag them on to the stage, and modify their properties.

inkscript-shapes

Multimedia

There are also sound, image, video, and three dimensional elements, as well as lights and cameras.

Flow and rules

Ideas can move throughout the system very quickly. While data and events are moving around, they get modified and cause reactions. Stage objects, and stages themselves share and modify data streams.

Tensor Flow diagram

Source: TensorFlow

Programming

Within flow elements, step-wise programs can be built with visual blocks, or some other iconic representation. This visual/iconic grammar is accessible to adults and children alike:

the directly symbolic approach taken by [many languages] would be difficult for the kids to process since evidence existed that the symbolic stage (or mentality) was just starting to switch on. In fact, all of the educators that I admired (including Montessori, Holt, and Suzuki) all seemed to call for a more figurative, more iconic approach.

Alan C. Kay - The Early History of Smalltalk

PureData flow interface

Source: Wikipedia - PureData

Declarative language/reasoning

Humans are very good at declaring things. "I'm hungry", "put the groceries on the table", "I wish my flowers would grow"... Computers need to understand declarative statements, so that humans can tell them what to do.

With regards to a rich media environment, we need a declarative language to create multi-dimensional, interactive media.

[Declarative 3D for the web will] allow anyone to easily create, share, and experience interactive 3D graphics - with possibly wide ranging effects similar to those caused by the broad availability of video on the Web.

Source: Declarative 3D for the Web Architecture Community Group

Blocks

Programming elements can be arranged and snapped together, or connected in a flow graph:

MIT Scratch block interface

Source: Wikipedia - Scratch (programming language)

Containers

Programming logic is contained in, or associated with, canvas elements: squeak eToysUI

Encapsulation, Abstraction

Blocks can be combined into 'meta' blocks, which abstract the inner workings for visual simplicity. Visual programs may be encapsulated into self-contained elements. These elements may send and receive data streams through external interfaces:

Cameleon language

Source: Cameleon language: Features

Modularity

Creations can take external input and provide output that can be accessed by other creations. This modular approach allows the entire space of design possibilities to communicate and resonate.

Rendered output at each stage

Each programming element can display the current 'state' of processed data. This helps the user to understand changes in each step of the programming flow:

Blender 3D node-based editor

Source: Wikipedia - Working with nodes in Blender 3D

Rendered output can include visual, numeric, text, and audio representations.

Context

Elements interact with one another, communicate, and change their surroundings. This form of context-full interaction enables rich, and emergent behaviour.

With native methods, we operate on context ... we operate on objects — instantiate them, change their properties, and add them to canvas. You can see that these objects are first-class citizens...

Source: Introduction to Fabric

Collaboration

Delayed

People may wish to share their work with others. To enable sharing and remixing, allow makers to publish their work to a public repository.

Publishing and remixing has been demonstrated by many community projects, including Khan Academy, Scratch, D3 Building Blocks, and eToys.

Real-time

In addition to sharing work for later reuse, people may wish to collaborate in real-time. To this end, the tool should support concurrent interaction with the project space. It should also indicate presence and enable communication, so that collaborators can interact with one another as well as the project.

Massively multiplayer

Makers can share and remix resources in a virtual space, i.e 'metaverse'. Collaborators can make open-access or protected creations.

Voxelands photo

Source: Voxelands: the fun-focused Free Software voxel world game

Platform

The platform should be built of Web Platform technologies.

Output

The final output of a project will be standard Web Platform technologies. The output should primarily target web browsers, or other environments that render native Web Platform code.

WebGL

One output option is WebGL. WebGL has the following benefits:

  • works in browsers without installing additional software (plugins)
  • crossplatform, intended for all desktop and embedded systems
  • open standard, does not require licensing fees
  • supported by the leading participants of the IT market (Google, Apple, Microsoft, Nvidia, Samsung, Adobe and others)
  • based on OpenGL which is familiar to developers
  • can be integrated with other browser technologies

Source: Blend4Web User Manual - Advantages of WebGL

Reference

There are several emerging projects and tools that allow direct manipulation of graphic and program objects. This section contains a few demonstrations and tools.

Presentations

Dataflow
Dataflow connect[s] computational actors into stages (pipelines) that can execute concurrently. --Wikipedia: Dataflow
Metaverse
A collective virtual shared space... made up of persistent, shared, 3D virtual spaces linked into a perceived virtual universe. --Wikipedia: Metaverse
Operational transform
[Method to] support a range of collaboration functionalities in advanced collaborative software systems. --Wikipedia: Operational transformation
mauriciovieira commented 9 years ago

:+1:

antiboredom commented 8 years ago

This is super interesting - thanks for posting! I think this is a bit outside the scope of the p5 editor at the moment.

brylie commented 8 years ago

OK, thanks for your feedback. The main observation is that all of the projects have limited scope, but seem to be working on pieces of a bigger picture.