new backend-integration

[adriankastrati]

• [ ] New endpoints for split, prompt-edit and prompt-generation

  (Not decided if the endpoints will use a URL or BLOB) The new endpoints will for computing the graph will now handle one node at a time prompt-edit endpoint (sound_URL, prompt) prompt-generation endpoint(prompt) split endpoint(sound_URL)

• [x] Job logic in frontend

  Topological sorting will be required If a parent node changes, restart the current job of nodes sent to endpoint. Put a warning that all nodes will be re-computed

• [ ] merge will be done in frontend

  find audio-library that can handle the capabilities of overlaying the sounds, altering each sound independ before merge-nodes are sent to API, they need to be written to sound db for a link

• [ ] Audio persistance

  Upload sound for URL new endpoints for post-audio and get-audio

Continued on this instead: https://github.com/Skein-studio/front-end-webapp/issues/39

[denipersson]

For the audio library, how about Howler.js? https://howlerjs.com

[adriankastrati]

@jespermunkeby Is it not better to just have send the node to one endpoint and have the backend validate it. So just a fetch with a body similar to this { node: signalNodeData nodeType: "Signal }

[adriankastrati]

found some other libraries that we can use, the thing with them is that they have some more capabilities. Some notes on these, I know @jespermunkeby worked with Tone.js so we can check with him what he though about it but I think that Tone.js or Web Audio API is the way to go if we want to have more features:

Pizzicato.js

Overview

Pizzicato.js aims to make working with the Web Audio API simpler. It wraps the API in a more convenient interface and provides useful utilities for common tasks, like applying effects and filters.

Features

• Sound Generation: Generate sound from oscillators, audio files, or input streams.
• Effects: Pre-built effects like delay, distortion, reverb, etc.
• Filters: Frequency filters, low-pass, high-pass, etc.
• Groups: Create a group of sounds to manipulate them together.

Pros and Cons

• Pros:
  • Easy to get started for beginners.
  • Lightweight with no dependencies.
  • Well-suited for simple to moderately complex projects.
• Cons:
  • Not as feature-rich as the raw Web Audio API or Tone.js.
  • Less active community compared to other options.

Example: Playing a Sound with Reverb

var sound = new Pizzicato.Sound({
  source: 'wave',
  options: { type: 'sine', frequency: 440 }
}, function() {
    var reverb = new Pizzicato.Effects.Reverb({
        time: 0.8,
        decay: 0.5,
        reverse: false,
        mix: 0.5
    });

    sound.addEffect(reverb);
    sound.play();
});

Tone.js

Overview

Tone.js is a powerful library for creating music in the browser. It's built on the Web Audio API and offers abstractions to create instruments, effects, sequencing, and more.

Features

• Instruments: Pre-built instruments like Synth, Sampler, etc.
• Effects: A wide array of effects, including reverb, distortion, etc.
• Sequencing: Built-in sequencing and timing controls.
• Modulation: Support for amplitude and frequency modulation.

Pros and Cons

• Pros:
  • Rich set of features for advanced audio processing.
  • Good documentation and examples.
  • Well-suited for complex audio and music web apps.
• Cons:
  • Can be overwhelming for beginners.
  • Heavier than other libraries for simple tasks.

Example: Simple Tone.js Synthesizer

const synth = new Tone.Synth().toDestination();
const now = Tone.now();
synth.triggerAttackRelease("C4", "8n", now);
synth.triggerAttackRelease("E4", "8n", now + 0.5);
synth.triggerAttackRelease("G4", "8n", now + 1);

Web Audio API

Overview

The Web Audio API is a high-level API for processing and synthesizing audio in web applications. It provides granular control over audio rendering but can be complex for simple tasks.

Features

• Nodes: Basic building blocks that can generate, modify or output audio.
• Graph: Audio nodes can be connected in a graph-like structure.
• Effects and Filters: Built-in effects can be applied by chaining nodes.
• Custom Processing: ScriptProcessorNode allows custom audio processing using JavaScript.

Pros and Cons

• Pros:
  • Most powerful and flexible option.
  • Supported natively by modern web browsers.
• Cons:
  • Steeper learning curve.
  • More verbose code for simple tasks.

Example: Creating Oscillator

var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var oscillator = audioCtx.createOscillator();
oscillator.type = 'sine';
oscillator.frequency.setValueAtTime(440, audioCtx.currentTime);
oscillator.connect(audioCtx.destination);
oscillator.start();

[adriankastrati]

With Web Audio API we can implement some more cool stuff, as spatial sound, vibrato/tremolo and stuff like that to, lets say guitar and voice.

[denipersson]

Really hard for me to say

[jespermunkeby]

@jespermunkeby Is it not better to just have send the node to one endpoint and have the backend validate it. So just a fetch with a body similar to this

{

node: signalNodeData

nodeType: "Signal

}

No I don't think so. I think that signal/node-types should be frontend concepts only. If we decouple them this way the api makes less assumptions and becomes much more flexible in case we want to experiment with building other products. There is more to say here too about how we handle separation between what should be done on client and server. We can discuss this further in person!

[jespermunkeby]

Great job with the audio alternatives. My experience from webAudio was that it's a bit too low level. I used Tone.js for our original beatroot app, it was simple but not fantastic by any means. Let's maybe create som every simple POCs with each one of merge for example, and see which one we like the best? I suspect both Tone and Pistachio are built in top of webaudio, so we can probably tap into webaudio stuff if there are particular parts where we need more control.

[jespermunkeby]

I think this also depends on who will work with the audio stuff on the frontend mostly. Let's discuss it soon