search

overtone / shadertone

A mix of www.shadertoy.com and Overtone
Other
419 stars 38 forks source link

readme 

                     __              __          __
               _____/ /_  ____ _____/ /__  _____/ /_____  ____  ___
              / ___/ __ \/ __ `/ __  / _ \/ ___/ __/ __ \/ __ \/ _ \
             (__  ) / / / /_/ / /_/ /  __/ /  / /_/ /_/ / / / /  __/
            /____/_/ /_/\__,_/\__,_/\___/_/   \__/\____/_/ /_/\___/
                                                                                 .

HeaderImage

A Clojure library designed to mix Musical Synthesis via Overtone and OpenGL shaders a la Shadertoy.

Overtone is a toolkit for designing sound synthesizers and coding musical performances. Based on Clojure and Supercollider, it is a powerful combination of a general-purpose, interactive, multi-threaded language built on the JVM with a state of the art, realtime sound synthesis server.

Shadertoy is a website that exhibits some amazing dynamic imagery--all procedurally generated via WebGL. It is a fantastic example of how limitations and constraints allow you to focus and create great artwork. All the imagery is created via a functional per-pixel program called a Fragment Shader that is provided only a few inputs like the pixel position and current time. The imagery is generated inside your Graphics Processing Unit (GPU) and offloads your CPU. Visuals are more powerful than words, here. Take a look at some fun examples: isolines, flower, and electron.

This library provides a way for you to create music and visuals that work together. The example inspiration for Shadertone was this live-coding video by h3xl3r. H3xl3r edits the fragment shader visual code live while the music plays. This library gives you the ability to hack on both the visual and audio code at the same time.

I've created a few screencast demos that you can watch:

Meta-eX has used Shadertone in live performance. Here are a few videos from their May, 2013 Kiev #hotcode conference performance that show how Shadertone can be used: after party 1, main conference, after party 1, after party 2. Latest news & videos can always be found at their website

Repl Electric has also used Shadertone in performance. Here is a wonderful recreation of "The Stars" livecoded on Vimeo.

If you're interested, here's the original announcement on the Overtone Google Group.

I hope this library allows you to create some happiness. Enjoy!

Usage

The library is just coming together, so expect change. There are two main ways to use the code. If you want the latest code or are just exploring what shadertone can do, you should clone the repo from github. But, if you have a project idea and want to use shadertone as part of that project, you can specify the current version in your Leiningen project.clj to download the library from clojars.

Option 1: clone this repository

  1. See top of https://github.com/overtone/shadertone for details on how to clone the repo.
  2. Bring up your favorite REPL and step through the demo files in the examples directory. There are examples of all of the various shadertone features in the demos.
  3. When you want to create your own code, make some files like examples/yourcode.clj and examples/yourshader.glsl. See below.
  4. Start with something basic in both files to be certain it works.
  5. You can edit both files "live". Shadertone will watch for edits to your active shader and load them when you save the file.

Option 2: download from clojars

In your project.clj, add Clojars Project to your :dependencies.

See https://github.com/rogerallen/sot for a simple example of using shadertone via Leiningen. Use the example directory in this repository for ideas.

Writing Fragment Shaders

Fragment Shaders are code that is executed for every pixel displayed in the window. The output is the color of that pixel, which you store in gl_FragColor. To create different imagery, you take the current pixel's position (from gl_FragCoord) and use that as input. To have Overtone affect the color, you'll want to use inputs like iOvertoneVolume. The syntax is C-like and given the math-processing power of modern GPUs, it is somewhat astonishing what you can do in 1/60th of second.

The code needs to be in a GLSL format with a main() routine that returns the gl_FragColor. Shadertoy.com has changed their syntax to make translation just a bit more work. You will need to translate Image shaders formatted like this:

void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
  ...
  fragColor = vec4(...);
}

To GLSL fragment shaders formatted like this:

void main(void) {
  ...
  gl_FragColor = vec4(...);
}

One of the creators of www.shadertoy.com, Iñigo Quílez, has a website that explains a variety of techniques to create interesting Fragment Shaders at http://www.iquilezles.org/www/index.htm. He also has some interesting live coding demos here http://www.iquilezles.org/live/index.htm.

Shader Inputs

Here are the inputs you have available to your fragment shader.

From www.shadertoy.com:

There are several implicit values that Shadertone provides, just like the website. You will have to define other inputs at the top of your shader.

uniform vec3      iResolution;     // viewport resolution (in pixels)
uniform float     iGlobalTime;     // shader playback time (in seconds)
uniform float     iChannelTime[4]; // channel playback time (in seconds)
uniform vec3      iChannelResolution[4]; // channel width, height, 1
uniform vec4      iMouse;          // mouse pixel coords. xy: current (if MLB down), zw: click
uniform samplerXX iChannel0..3;    // input channel. XX = 2D/Cube
uniform vec4      iDate;           // (year, month, day, time in seconds)

Overtone inputs:

There are three inputs from Overtone that you can use.

First is a tap on the output volume. It is sent to the variable iOvertoneVolume which you should define at the top of your shader if you start your window with shadertone.tone/start.

uniform float iOvertoneVolume; // tap of system volume

Next, are two special textures. First, if you use the keyword :overtone-audio as a texture name, Overtone will send the output sound's frequency spectrum (FFT) and audio waveform data to the iChannel texture you specify. This is similar to www.shadertoy.com's audio textures. The first row of the texture contains the FFT data. The second row of the texture contains the current waveform data.

Second, if you use the :previous-frame keyword as a texture name, Shadertone will capture the framebuffers you render and allow you to use this as input for the next frame.

As an example, if you want these textures data to go to the first two textures, add the argument :textures [:overtone-audio :previous-frame] to your shadertone.tone/start call. Then, in your glsl code use texture2D(iChannel0,uv).r to access the audio and texture2D(iChannel1,uv).rgb to access the previous frame.

But that isn't all. If you want to create your own inputs from Overtone to Shadertone, you can create atoms that hold either a floating-point number or a vector of 1-4 values.

For example, this clojure code

(def my-rgb (atom [0.3 0.1 0.5]))
(t/start "shaders/rgb.glsl" :user-data { "iRGB" my-rgb})

Will adjust this shader code live

uniform vec3 iRGB;
void main(void)
{
    gl_FragColor = vec4(iRGB, 1.0);
}

Via a call like this:

(swap! my-rgb (fn [x] [0.55 0.95 0.75]))
Synth Inputs

You can tap a synth value and easily communicate that value to your GLSL fragment shader. See the "vvv" synth example in the 00_demo_intro_tour.clj demo and also the core.clj usage.

In a nutshell, you define your synth and add a call to (tap "a" 60 a) where "a" is the name of your tap, it is updated 60 times per second and the a synth variable is some interesting value you want to communicate.

Then in your start call, the :user-data map should include a reference to that tap. Like :user-data { "iA" (atom {:synth v :tap "a"}) } to call the value iA as input to your GLSL shader, if your synth was instantiated as v.

Finally, in your GLSL fragment shader, define your new input value as uniform float iA; and use iA to control your shader in interesting ways.

Clojure API

To use Shadertone, the main routines you will use are in the shadertoy.tone namespace. Use start or start-fullscreen to bring up a window. Use stop to bring it down. Calling start while a window is active will stop it gracefully before starting up the new window.

start takes a few parameters

start-fullscreen takes just a subset of these, but the meanings are the same: :display-sync-hz, :textures, :user-data, and :user-fn

The base api is in shadertone.shader and does not depend on Overtone. This is useful when you don't need sound accompaniment and could be useful for other interactive ideas.

Lisp-like GLSL

You can program your GLSL in a lisp-like language. It is a simple, straight translation layer from lisp to a GLSL string. For those hoping for Clojure instead of lisp, I welcome your suggestions and help, but a Clojure-to-C compiler was way too much work for me at this time.

See https://github.com/overtone/shadertone/blob/master/examples/03demo_translate.clj for an example of how you can use this. In addition, there are tests in https://github.com/overtone/shadertone/blob/master/test/shadertone/translate_test.clj that could be informative.

To access this functionality, import the shadertone.translate namespace and use defshader to compile your lisp into a string for passing to the start function.

Special forms and how they relate to GLSL code

Reading Pixels from the Framebuffer

You can read back a pixel value from your frame either via (shadertone.shader/pixel) function or by accessing the @shadertone.shader/pixel-value atom directly. With this feature, your fragment-shader could control your synths! Thanks to Circu Virtu on the Google Group for this idea.

Enable this feature via (pixel-read-enable! x y) where x and y are valid locations within your window. Be careful as there is no error checking here.

Disable this access via (pixel-read-disable!)

Changes

Acknowledgements

Many thanks to those who have helped directly and indirectly to make Shadertone. Specifically, I'd like to thank:

License

Copyright © 2013-2014 Roger Allen and other contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

For full license information, see the LICENSE file.