Milkdrop supports drawing user-defined "sprites", which are simple rectangular images with an optional color key (for transparency in image formats which don't have an alpha channel). These sprites can optionally be animated via expression code, using a few input values like preset timers and the beat detection values.
projectM should also support rendering these sprites, as they're very useful for streamers and making videos with watermarks or additional animated logos on top to save post-processing work and sync the image movement exactly the the visuals.
Milkdrop Sprite Definition File Format
The sprite def file is very similar to the .milk preset format, as it also uses INI syntax. Each sprite is defined in its own section named [imgXX], where XX is a two-digit number. Each section can contain the following lines:
img = <filename>: The sprite texture image to load.
colorkey = 0xAABBGGRR: Transparency color key. Not exactly sure about the byte order, must investigate further!
init_N = <code>: Initialization expression, run once after loading the sprite. N is the line number, starting with 1. Numbering must be continuous. Each line of <code> is appended and executed as one program.
code_N = <code>: Per-frame expression, run once every time the sprite is drawn. N is the line number, starting with 1. Numbering must be continuous. Each line of <code> is appended and executed as one program.
The following expression variables are available in the code_N expressions, updated with the current frame data:
time: Time passed since program start. Also available in init code.
frame: Total frames rendered so far. Also available in init code.
fps: Current (or, if not available, target) frames per second value.
progress: Preset blending progress (only if blending).
bass: Bass frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
mid: Middle frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
treb: Treble frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
bass_att: More attenuated/smoothed value of bass.
mid_att: More attenuated/smoothed value of mid.
treb_att: More attenuated/smoothed value of treb.
The following output variables are used to draw the sprite:
done: If this becomes non-zero, the sprite is deleted. Default: 0.0
burn: If non-zero, the sprite will be "burned" into currently rendered presets when done is also true, effectively "dissolving" the sprite in the preset. Default: 1.0
x, y: Sprite x/y position (position of the image center). Range from -1000 to 1000. Default: 0.5
sx, sy: Sprite x/y scaling factor. Range from -1000 to 1000. Default: 1.0
rot: Sprite rotation in radians (2*PI equals one full rotation). Default: 0.0
flipx, flipy: If a flag is non-zero, the sprite is flipped in this axis. Default: 0.0
repeatx, repeaty: Repeat count of the image on the sprite quad. Fractional values allowed. Range from 0.01 to 100.0. Default: 1.0
r, g, b, a: Modulation color used in some blending modes. Default: 1.0
Default values are used if the expressions don't explicitly set a value. No q or t variables are available to sprite expressions. The regXX vars, gmegabuf and megabuf are individual per-sprite contents, not shared between sprites or with any running presets.
The blending modes are using this "effect" matrix:
Sprite positioning and aspect-correction calculations could be done in a sprite-specific vertex shader, as well as implementing the blending modes in the fragment shader. This should make the C++ code a bit cleaner.
When implementing user sprite support in projectM, these sprites must be drawn after the whole preset rendering process and any transition blending is done, effectively being the last step before returning the render_frame method. This makes sure that sprites are only drawn once and continue working even after introducing new preset formats besides Milkdrop.
Burning-in stuff into presets is currently not implemented. Could be implemented easily by adding a method to the Preset interface class, returning a reference to the framebuffer holding the current texture which is passed to the next frame, with this texture being the only color attachment. Instead of drawing into the final output framebuffer, the sprite can be drawn here. Note that during blending, this must be done with both running presets!
Put all sprite-related API functions into a separate header, e.g. sprites.h and include the header in projectm.h.
Adding sprites should be done via a new API function, which takes the sprite definition as text (const char*) and also could have an additional "type" parameter so we can later add additional sprite types which differ from the Milkdrop format. Sprite data should be formatted in the same way Milkdrop accepts them (see below).
In Milkdrop, sprites are "launched" or killed by the user entering a two-digit number. The sprites exists either until the sprite sets the done flag or the user actively kills the sprite. Since libprojectM doesn't handle any keyboard input, we need API methods to start or kill specific sprites.
There should also be a method to clear all loaded sprites, and optionally additional methods to enumerate and replace/unload specific sprites. Do not confuse this with the launch/kill methods above, which control whether a sprite is drawn or not.
libprojectM must not search for sprite definition files on its own - adding sprites is solely application-controlled.
Milkdrop Source References
Here's a list of relevant code parts in the Milkdrop sources related to sprite rendering:
Milkdrop supports drawing user-defined "sprites", which are simple rectangular images with an optional color key (for transparency in image formats which don't have an alpha channel). These sprites can optionally be animated via expression code, using a few input values like preset timers and the beat detection values.
projectM should also support rendering these sprites, as they're very useful for streamers and making videos with watermarks or additional animated logos on top to save post-processing work and sync the image movement exactly the the visuals.
Milkdrop Sprite Definition File Format
The sprite def file is very similar to the .milk preset format, as it also uses INI syntax. Each sprite is defined in its own section named
[imgXX], whereXXis a two-digit number. Each section can contain the following lines:img = <filename>: The sprite texture image to load.colorkey = 0xAABBGGRR: Transparency color key. Not exactly sure about the byte order, must investigate further!init_N = <code>: Initialization expression, run once after loading the sprite.Nis the line number, starting with 1. Numbering must be continuous. Each line of<code>is appended and executed as one program.code_N = <code>: Per-frame expression, run once every time the sprite is drawn.Nis the line number, starting with 1. Numbering must be continuous. Each line of<code>is appended and executed as one program.The following expression variables are available in the
code_Nexpressions, updated with the current frame data:time: Time passed since program start. Also available ininitcode.frame: Total frames rendered so far. Also available ininitcode.fps: Current (or, if not available, target) frames per second value.progress: Preset blending progress (only if blending).bass: Bass frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.mid: Middle frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.treb: Treble frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.bass_att: More attenuated/smoothed value ofbass.mid_att: More attenuated/smoothed value ofmid.treb_att: More attenuated/smoothed value oftreb.The following output variables are used to draw the sprite:
done: If this becomes non-zero, the sprite is deleted. Default: 0.0burn: If non-zero, the sprite will be "burned" into currently rendered presets whendoneis also true, effectively "dissolving" the sprite in the preset. Default: 1.0x,y: Sprite x/y position (position of the image center). Range from -1000 to 1000. Default: 0.5sx,sy: Sprite x/y scaling factor. Range from -1000 to 1000. Default: 1.0rot: Sprite rotation in radians (2*PI equals one full rotation). Default: 0.0flipx,flipy: If a flag is non-zero, the sprite is flipped in this axis. Default: 0.0repeatx,repeaty: Repeat count of the image on the sprite quad. Fractional values allowed. Range from 0.01 to 100.0. Default: 1.0blendmode: Image blending mode. 0 = Alpha blending (default), 1 = Decal mode (no transparency), 2 = Additive blending, 3 = Source color blending, 4 = Color key blending. Default: 0r,g,b,a: Modulation color used in some blending modes. Default: 1.0Default values are used if the expressions don't explicitly set a value. No
qortvariables are available to sprite expressions. TheregXXvars,gmegabufandmegabufare individual per-sprite contents, not shared between sprites or with any running presets.The blending modes are using this "effect" matrix:
Implementation Notes
Sprite positioning and aspect-correction calculations could be done in a sprite-specific vertex shader, as well as implementing the blending modes in the fragment shader. This should make the C++ code a bit cleaner.
When implementing user sprite support in projectM, these sprites must be drawn after the whole preset rendering process and any transition blending is done, effectively being the last step before returning the render_frame method. This makes sure that sprites are only drawn once and continue working even after introducing new preset formats besides Milkdrop.
Burning-in stuff into presets is currently not implemented. Could be implemented easily by adding a method to the
Presetinterface class, returning a reference to the framebuffer holding the current texture which is passed to the next frame, with this texture being the only color attachment. Instead of drawing into the final output framebuffer, the sprite can be drawn here. Note that during blending, this must be done with both running presets!Put all sprite-related API functions into a separate header, e.g.
sprites.hand include the header inprojectm.h.Adding sprites should be done via a new API function, which takes the sprite definition as text (
const char*) and also could have an additional "type" parameter so we can later add additional sprite types which differ from the Milkdrop format. Sprite data should be formatted in the same way Milkdrop accepts them (see below).In Milkdrop, sprites are "launched" or killed by the user entering a two-digit number. The sprites exists either until the sprite sets the
doneflag or the user actively kills the sprite. Since libprojectM doesn't handle any keyboard input, we need API methods to start or kill specific sprites.There should also be a method to clear all loaded sprites, and optionally additional methods to enumerate and replace/unload specific sprites. Do not confuse this with the launch/kill methods above, which control whether a sprite is drawn or not.
libprojectM must not search for sprite definition files on its own - adding sprites is solely application-controlled.
Milkdrop Source References
Here's a list of relevant code parts in the Milkdrop sources related to sprite rendering: