ALPHA β Work in progress... Welcoming feedback! π
Please also see RFCs and open questions in the issue tracker. Thanks!
[!NOTE] tl;dr Jump to the Getting started section.
Over the past years, generative/computational art has experienced a surge in popularity because of the availability of online art platforms for publishing these works. The number of these platforms keeps on mushrooming, each one defining their own ad hoc solutions to deal with common aspects (e.g. handling of parameters to customize pieces/variations, generating previews, etc.). Often, this means artists have to either already decide on which platform to publish a new piece before they work on it and/or spend a considerable amount of time reworking key aspects (like parameter, screen resolution or time handling) when repurposing a piece for a different use (e.g. creating hires print versions, high-quality video assets, or displaying the piece in a gallery with different configurations or requirements). Since online platforms/startups usually only optimize for their own uses and neglecting the needs of artists regarding asset preparation, adaptation and re-use, we must take it upon ourselves to address these workflow issues more efficiently.
To improve this situation, this project proposes an API layer and a message protocol addressing recurring issues artists encounter when publishing and repurposing browser-based generative artworks β be it for diverse online platforms, environments (incl. offline, galleries, installations), but also aforementioned aspects of media production (for example helping to deal with realtime/non-realtime rendering when recording image sequences for video production, and do so in an unintrusive & externally controlled manner).
Schematic overview of the proposed architecture
The primary purpose of this API is to decouple key aspects commonly used for most generative/computational artworks, to deduplicate feature implementations and reduce the time & effort required for adapting browser-based artworks for different uses/environments. These benefits are not only in the interest of artists, but also simplify how online art platforms can use this API layer to reduce effort on their end, simplify providing customization features for such generative artworks (and even re-use tooling).
Another positive side effect of adapting the system proposed here: The emergence of secondary re-usable tooling to manage parameters and variations, for example: tooling to generate GUI controls for editing params, creating/storing/retrieving parameter presets/collections (aka variations), asset downloaders, transcoding tools. Even at this stage, some of these are existing already and actively being worked on...
In this document & repository we describe the approach, the proposed architecture and provide a TypeScript-based reference implementation, including fully documented interfaces & types, and some example test cases to demonstrate (and validate!) the approach and its benefits.
requestAnimationFrame()
-based animation) and/orThis API aims to remain as minimal as possible and will not directly support every feature of every platform, which would cause feature creep, bloat and unsustainable maintenance effort.
Artworks requiring or relying on advanced integrations with a specific platform are naturally highly dependent on those platform features, and so would not benefit from a more platform-independent approach in any way. Therefore, we consider these use cases as out-of-scope.
However, the proposed system is designed to enable platform-specific extensions in several ways, which can highly benefit all parties involved when adopted and can be supported unintrusively. For example, a platform/device can provide sensor input and expose it through a custom parameter type, allowing a mock implementation to be provided when running elsewhere.
The API implements a finite state machine with the following possible states:
init
: Initial state until the platform adapter, time
provider and the artwork's update function have been
registered or an error occurred during any init stepsready
: All init steps have been completed and playback of the artwork can
commenceplay
: API is currently running the animation loop, ruuning the artworkstop
: Animation loop has been paused/stoppederror
: API is in an unrecoverable error stateThe API also defines and uses a message protocol to communicate certain lifecycle events, state changes and requests to both internal & external participants.
genart:capture
genart:capturerequest
genart:paramchange
genart:paramerror
genart:randomizeparam
genart:frame
genart:resize
genart:resume
genart:setparams
genart:setparamvalue
genart:settraits
genart:start
genart:statechange
genart:stop
TODO β for now please see the following links for more information:
Full API docs are actively being worked on. For now, this readme, the source code and the example projects are the best reference.
Almost all generative artworks use parameters and randomization to produce numerous variations. Exposing some of these parameters to the outside world can allow people/agents to have more participation and direct control over the generative outcomes.
The API supports an extensible set of parameters types with a selection of commonly used types supplied as built-ins (described below).
Each parameter declared by the artwork is a simple vanilla JS object and any param value changes are being handled by the API and the param typeβs registered implementation (a set of functions dealing with validation, coercion, updating and reading param values). For convenience & type safety, the API provides factory functions for all built-in parameter types.
The following parameter types are available, but other custom ones can be registered, making the entire system extensible. Each declared param spec is a simple JS object, and the API provides factory functions for all built-in types, helping to fill in any default options.
IMPORTANT: All param declarations must include a desc
(brief description)
and have the option to define default
values for most parameter types. If no
default value is provided, a randomized value will be chosen within the
constraints of the param.
Generally speaking, parameters can be grouped into static & dynamic parameters with the former providing the same values each time they're read/evaluated and the latter being used for time-based values obtained from otherwise dynamic sources (e.g. sensors).
This section describes the set of static param types:
This enum-like param can only take on values from a fixed list of options. For simplicity values are always strings, but optionally can also define labels (only used for display purposes by external tooling).
// options
$genart.params.choice({
name: "Test param",
desc: "Shape size preset",
options: ["s", "m", "l"],
default: "m",
});
// ...or using options with labels
$genart.params.choice({
name: "Test param",
desc: "Shape size preset",
options: [
["s", "Small"],
["m", "Medium"],
["l", "Large"],
],
options: "m",
});
CSS hex color value (6 digits only). Other/newer color types (e.g. oklch()
)
might be supported later, but currently omitted due to lack of native browser
widgets for editing these colors...
$genart.params.color({
name: "Test param",
desc: "Background color",
default: "#aabbcc",
});
JS Date
value in precision of full days only (UTC midnight).
$genart.params.date({
name: "Test param",
desc: "Best before date",
default: new Date("2024-09-05"),
});
JS Date
value (in UTC).
$genart.params.datetime({
name: "Test param",
desc: "Date and time (in UTC)",
default: new Date("2024-09-05T12:34:56+02:00"),
});
Image parameters are used to provide spatially varied param values (e.g. gradient maps, masks etc.) by allowing the artwork to read pixel values from different points in the image.
An image param has a pixel array as value
, alongside width
, height
and
format
, which can be:
gray
(8 bits)rgb
(packed int, 24 bits, channels in MSB order: R,G,B)argb
(packed int, 32 bits, channels in MSB order: A,R,G,B)The format
is only used as hint for platform adapters and external tooling
(e.g. GUI param editors) to optimize their handling of the image data. I.e. an
image with mode gray
can be encoded as byte sequence vs. argb
requiring 4
bytes per pixel. Likewise a param editor allowing image uploads/customizations
is responsible to resize an image to the expected dimensions and provide its
data in the expected format.
[!IMPORTANT] Note to artists: Ensure to keep your image size requirements as low as possible! Storing image data in URL parameters (as done by most platforms) has hard browser-defined limits and this param type is not (yet?) supported by any of the major online art platforms.
TODO
$genart.params.numlist({
name: "test",
desc: "List of numbers",
default: [1, 2, 3],
});
$genart.params.strlist({
name: "test",
desc: "List of strings",
default: ["a", "b", "c"],
});
Numeric value from a closed range/interval (defined by min
/max
, defaulting
to [0, 100]). If step
is given, the value will be rounded to multiples of
step
(always clamped to min/max).
$genart.params.range({
name: "Test param",
desc: "Pick a number between 0-100",
min: 0,
max: 100,
step: 5,
});
Single or multi-line text, optionally with min
/max
length and/or regexp
pattern validation.
$genart.params.text({
name: "Test param",
desc: "Seed phrase",
max: 256
match: "^[a-z ]+$"
multiline: true
});
Time-of-day parameter with values as [hour, minute, second]
-tuples in 24h format.
On/off switch (boolean) parameter.
Similar to the Choice parameter type, but here each option also has an associated weight/importance. When randomizing this parameter, a new value is chosen based on the probability distribution defined by the relative weights given to each option.
Note: The options can be given in any order and their weights are always only
treated relative to each other. The $genart.params.weighted()
factory function
is automatically sorting these options by weight and pre-computes their total
weight.
If no default value is given, it will be picked randomly (using the weights, as described above).
// define color options with these probabilities:
// - black: 8/15th (53%)
// - cyan: 4/15th (27%)
// - magenta: 2/15th (13%)
// - yellow: 1/15th (7%)
$genart.params.weighted({
desc: "Controlled randomness",
options: [
// format: [weight, value]
[8, "black"],
[4, "cyan"],
[2, "magenta"],
[1, "yellow"],
],
});
// optionally, labels can be provided for each option
$genart.params.weighted({
desc: "With labels",
options: [
// format: [weight, value, label]
[8, "#000", "black"],
[4, "#0ff", "cyan"],
[2, "#f0f", "magenta"],
[1, "#ff0", "yellow"],
],
});
A 2D dimensional tuple which produces values in the [0,0] .. [1,1] range. Useful to control two co-dependent parameters using an XY controller/touchpad...
$genart.params.xy({
name: "Test param",
desc: "Bottom-left: [dark,dry] / Top-right: [bright,wet]",
default: [0.5, 0.5],
});
Dynamic parameters are either time-based (using a abstract notion of "time") or otherwise produce values which could change each time a parameter is being read/evaluated by the artwork (e.g. sensor inputs).
Unlike static param types, these parameters have no default value and their actual value is always computed.
A ramp defines a one-dimensional curve/gradient via a number of stops/keyframes, from which then an actual value will be derived by sampling the curve at a specific position. Ramp are different to other parameter types in that they're mostly intended for time-based parameters whose value should evolve over time (rather than being a static quantity).
Ramps support any number of stops/keyframes (though, at least 2 are required!) and by default the API supports the following interpolation modes:
linear
: linear interpolation (default)exp
: exponential interpolationsmooth
: smoothstepRecommended GUI widget:
$genart.params.ramp({
name: "Test param",
desc: "Brightness over time",
stops: [
[0, 0.1],
[0.9, 1],
[1, 0],
],
mode: "smooth",
});
The system supports registering custom parameter types and their implementation
via
$genart.registerParamType()
.
These can be useful to provide additional app-specific or platform-specific
parameters (e.g. values obtained from arbitrary hardware sensors to which an
artwork might respond dynamically).
The registerParamType()
allows overriding of existing param type
implementations, but doing so will print a warning. When an artwork or platform
registers its own types, it SHOULD consider namespacing the type name, e.g.
platformname:customtype
.
The following example shows how to implement, register and then use a sine wave oscillator parameter type, providing time-based values. A more advanced version is shown in the param-custom example.
// define a new param type with given name and implementation.
interface OscParam extends Param<number> {
type: "user:sinosc";
freq: number;
amp: number;
offset: number;
}
// this implementation does not allow any customizations or value randomization
// (the latter is common to all dynamic params, since the value is ALWAYS computed)
$genart.registerParamType("user:sinosc", {
// for brevity we decide this param cannot be customized (once defined)
valid: (spec, key, value) => false,
// the read function is called each time this param is evaluated,
// here to provide a time-based value
read: (spec, t) => {
const { freq, amp, offset } = <OscParam>spec;
return Math.sin(t * freq * Math.PI * 2) * amp + offset;
},
});
// register parameter
const param = $genart.setParams({
sine: {
type: "user:sinosc", // param type (should be unique)
freq: 0.25, // frequency in Hz
amp: 5, // amplitude
offset: 5, // center offset
default: 0, // ignored
},
});
$genart.setUpdate((t) => {
// current time in seconds
t *= 0.001;
// evaluate param
console.log(t, param("sine", t));
// keep on animating
return true;
});
Composite parameter types contain child parameters to allow the main param be
configured in more detailed/complex ways. Usually, a composite parameter type
will define a
.read()
method to dynamically produce values, based on the current configuration
of its child params.
An example of such a composite parameter type is an extended version of the oscillator described in the previous section.
TODO
Please see the param-custom example for reference...
Some platforms support the concept of artist-defined metadata, commonly called "traits" or "features". In most cases, this metadata is a key-value dictionary, where the keys are arbitrarily defined criteria the artist wants to expose, and usually, the values are derived from the current parameter configuration of a particular variation.
This API principally supports this use case via the $genart.setTraits()
function, but it's the platform adapter's responsibility to deal with this
information provided, i.e. declaring traits has zero impact on any other
workings or state of the API layer.
The value of a single trait can be a number, string or boolean.
// declare a single param
const param = await $genart.setParams({
bright: $genart.params.range({
name: "brightness",
desc: "overall brightness",
min: 0,
max: 100,
}),
});
// internal random param
const density = $genart.random.rnd();
// declare traits to outside world
$genart.setTraits({
brightness: param("bright") < 50 ? "dark" : "light",
density: Math.floor(density * 100) + "%",
});
TODO This section will describe the role(s) of adapters responsible for providing (deployment) platform-specific functionality and interop features.
TODO write docs
TODO write docs
Related issues/RFCs:
Platform adapters are responsible to provide a seedable and resettable,
deterministic pseudo-random number generator which the artwork can access via
$genart.random
.
Usually, the adapter just has to return the PRNG provided by the respective
platform.
For cases where a platform does not provide its own PRNG, this repo contains two implementations which can be used by an adapter:
Platform adapters are responsible to provide screen/canvas dimensions (and pixel
density information) to the artwork, and the latter MUST use the
$genart.screen
accessor to obtain this information (rather than directly querying
window.innerWidth
etc.). This accessor returns a ScreenConfig
object.
This approach gives the platform full control to define fixed dimensions, irrespective of actual window dimensions and also be in charge of resizing behavior.
Similar to the above, artwork which aims to be responsive to changing screen
configurations MUST NOT listen to window
resize events directly, but MUST add
a listener for genart:resize
messages, like so:
// resize events contain the new screen config
$genart.on("genart:resize", ({ screen }) => {
console.log("new screen info:", screen.width, screen.height, screen.dpr);
});
If an artwork is unable to dynamically respond to resize events, it should at least implement a handler which reloads the window on resize, like so:
$genart.on("genart:resize", () => location.reload());
TODO
A time provider is a further decoupling mechanism to allow artists (or platforms) to re-purpose an artwork in different media production or asset management contexts. Common scenarios include: an artist (or platform) might want to produce high-res outputs or high-quality video assets which can only be created by running the piece at a much lower frame rate (or a frame rate controlled via external events/triggers). In order to protect the artwork code from having to be adjusted for these use cases, the API uses pluggable timing providers which are merely responsible to schedule the next animation frame and provide a related timestamp and frame number.
The API defaults to using a requestAnimationFrame()
-based
provider, but
this one can be replaced by loading an alternative implementation via another
<script>
tag, like so:
<!-- main GenArtAPI script -->
<script src="https://github.com/thi-ng/genart-api/raw/main/lib/genart.js"></script>
<!--
optional: use custom time provider (e.g. for non-realtime rendering of image sequences)
configure API to use offline time (new frame every 250 ms, time base: 60 fps)
IMPORTANT: MUST be loaded AFTER the main genart script!
-->
<script>
$genart.setTimeProvider($genart.time.offline(250, 60));
</script>
requestAnimationFrame()
-based[!NOTE] The reference implementation of the API provided has no dependencies.
This repo contains several examples used for testing and evaluating the reference API implementation. These are all separate projects/packages located in the /examples directory. Please ensure you read their README instructions, since a certain build order must be used:
Project | Live demo w/ editor | Description |
---|---|---|
p5-basic | Demo | Basic p5.js example |
param-test | Demo | Minimal "art" example using various parameter types |
param-image | Demo | Example using an image map parameter |
param-custom | Demo | Custom & composite parameter type example |
param-editors | Example/reference editor implementations |
This repo contains an empty project template which can be used as starting point for new projects. The template uses TypeScript & Vite, but can be very easily adapted to other tooling (eg. there's hardly any code, so switching to JavaScript requires just renaming the source file).
git clone https://github.com/thi-ng/genart-api.git
# copy template to new directory
cp -R genart-api/project-template my-new-art-project
# install dependencies (genart api, typescript, vite)
cd my-new-art-project
yarn install
The @thi.ng/genart-api package
contains the pre-built release files & type declarations, but is not an ESM
module which can (or even should) be imported via import
syntax. The API is
always provided as singleton via the global $genart
variable.
Instead, the JS file(s) should be copied to your project's /lib
dir (or
similar) to be referenced by <script>
tag in your HTML wrapper (see below):
yarn add @thi.ng/genart-api
# create dest dir
mkdir -p lib
# copy files
cp node_modules/@thi.ng/genart-api/*.js lib
If you're a TypeScript user, you'll also want to add the package to your
tsconfig.json
types field OR add a declaration file with the following content
to your /src
directory:
tsconfig.json
{
"compilerOptions": {
"types": ["@thi.ng/genart-api"]
}
}
/src/types.d.ts:
/// <reference types="@thi.ng/genart-api" />
Then you should be ready to go to the next step...
The /dist directory
contains all pre-built release files and type declarations which you should copy
to your project directory (e.g. in a /lib
dir or similar).
If you're a TypeScript user, you'll also want to add a declaration file with the
following content to your /src
directory:
/src/types.d.ts:
/// <reference types="../lib/genart.d.ts" />
This section explains the basic approach to create a GenArtAPI
conformant art
project and assumes you have installed the API files/types via one of the ways
described above.
<script src="https://github.com/thi-ng/genart-api/raw/main/lib/genart.min.js"></script>
The repo provides a basic platform adapter for sourcing
parameters via URL query string (aka
URLSearchParams
).
This adapter can be useful during local development, or used as basis for other
use cases:
<script src="https://github.com/thi-ng/genart-api/raw/main/lib/adapter-urlparams.min.js"></script>
TODO for now see existing adapter implementations...
yarn install
# output files will be written to /dist
yarn build
# generate TypeScript type declarations (also written to /dist)
yarn build:types
The following command is used to create production builds of all examples:
# from the repo root
yarn build:examples
The repo contains two reference implementations of GUI parameter editors to
modify param values exposed by an artwork running in an embedded <iframe>
.
These editors can be used to customize parameters of the other bundled
examples (or any other project using GenArt API and the
supplied reference platform adapter). See the
param-editors
readme
for details.
The following table provides an overview of the current parameter types support in both editors:
Param type | thi.ng/rdom editor | thi.ng/imgui editor |
---|---|---|
Choice | β | β |
Color | β | β (1) |
Composite params | β | β (2) |
Date | β | β |
Datetime | β | β |
Image | β | β |
List | β | β |
Range | β | β |
Ramp | β | β |
Text | β | β (3) |
Time | β | β |
Toggle | β | β |
Weighted choice | β | β |
XY | β (1) | β |
© 2024 Karsten Schmidt and contributors // MIT License