Integrate Strudel with csound-wasm to facilitate cloud-music pieces

[gogins]

The idea is, that if I am committed to making always-on or cloud-music pieces, it makes sense to develop using a WebAssembly build of Csound, integrating with Tidal Cycles Strudel if possible. The only things missing here would be some of the C++ opcodes and VST hosting. These are significant but don't seem deal breaking.

The "sf" opcodes look at first glance like a workable replacement for the "fluid" opcodes.

The "chn*" opcodes are indeed a workable replacement for the signal flow graph opcodes, especially since I have realized that named channels can be specified as global string variables in the instrument "namespace" similar to the way I now use ftgen.

What would hurt is losing the Pianoteq and Organtec, basically.

[gogins]

VS Code can certainly open an npm application in a local Web server. This is a two step process, the "npm dev" task starts vite and then the user must jump to the URL that is being served.

Can't get VSCode to preview oblivion.html which runs just fine in other ways. I suspect the HTML root is wrong.

Of course I can just add a tool to SciTE for node run. But I don't think it's really that simple.

[gogins]

Does the Web-IDE or its Csound support plugin opcodes?

[gogins]

Strudel (according to the tutorial) makes sound in the following ways:

1. Directly using WebAudio and a built-in sampler (WebDirt).
2. Using MIDI but then your patterns have to be <pattern>.midi().
3. By sending SuperDirt style OSC messages but then your patterns have to be <pattern>.osc() and you have to run npm run osc to create a WebSockets/OSC broker.

None of these options seems to make it possible to simply inject Csound. The hap objects are pretty low-level.

[gogins]

Managing JavaScript dependencies with npm seems like a very good idea.

[gogins]

I am examining the details of the WebAudio/WebDirt interaction with Strudel. This happens in the webAudioOutput function, but then webAudioOutput is assigned to defaultOutput, so that acts as a layer of abstraction or hook. If the abstraction were simple enough then defaultOutput could be replaced with csoundOutput or something, but it's not that simple because webAudioOutput is always messing with the WebAudio signal flow graph, loading samples, etc.

But then csoundOutput could defer to webAudioOutput to continue supporting the way things work now.

I must try to hook up MIDI to see how that works.

[gogins]

I got MIDI hooked up. Yes, the pattern has to have midi called in order to send MIDI out. And you have to enable IAC Midi in macOS using the Audio MIDI Setup app, Window menu, Show MIDI Studio dialog, IAC Driver dialog, and enable the Device is online checkbox.

Then in GarageBand I enabled input from all MIDI in ports, and Strudel was playing GarageBand just fine.

Con: adding .midi() to note generating patterns is awkward.

Pro: You can mix .midi() patterns with the default WebDirt/WebAudio patterns with .osc() to SuperDirt patterns. So that gives one a bit more power.

The upshot is, if one of my always-on WASM pieces can get MIDI from the IAC Driver, this becomes doable.

[gogins]

As I had never tested WebMIDI input in csound-wasm, it had some bugs, which I'm pretty sure I can fix.

[gogins]

• [x] Verify that native Csound can handle MIDI events from a MIDI file. Yes it can.
• [x] Verify that native Csound can handle MIDI events from the IAC Driver. Yes it can, at least from Strudel.
• [x] Verify that csound-wasm can handle MIDI events from the IAC Driver. Yes it can, but only if the Strudel REPL has focus will messages actually go to Csound. There is a question of how to get Strudel to send to different MIDI channels. It's <pattern>.midi(port, channel) but MIDI does not chain like the other operators. Stil, that should make this integration usable, though channels would be nice. I don't think Strudel sends MIDI controllers.
• [x] Verify that csound-wasm can be included in a branch of the Strudel REPL. It's easier to embed, but the embedded iframe can't send MIDI to the containing page, so I have to run npm run repl locally, which permits MIDI.

[gogins]

Alex McLean and the others are very good programmers, and I will tell them so. There are things that need work:

• [ ] The music visualizer isn't picking up on Strudel's audio stream. It might not be possible to do this if csound-wasm and the Strudel repl are running in separate processes as they do in my tests so far.
• [x] I need to serve both Csound and Strudel from the same local vite server, so I can embed Strudel in cloud-music somehow without losing MIDI permissions. Works fine using a regular Web server, off a build of Strudel..
• [x] I should try hard to avoid using virtual MIDI, there must be some way to do this, perhaps just using WebMIDI. I created a "patternified" Csound output, two actually, one by Felix Roos named csound and one by me named csoundm using MIDI key and velocity for pitch and loudness.- [x] I need to be able to inject Silence code (i.e. the JavaScript interfaces to it) into Strudel patterns, though at present I have NO idea how to do this. But if I can introspect Strudel's loop, and find JavaScript objects instantiating patterns, then I CAN do this, though it might be hard from the standpoint of algorithms. _I figured this out by writing a csoundm output. I can put any kind of loaded JavaScript code, including the JavaScript CsoundAC interface, into the <Pattern>.<output>.onTriggered hook._
• [x] I need to find out what is the server infrastructure and code that supports embedding Strudel. I think it's just npm run start on a public domain in the Strudel repository. This is not actually relevant.

[gogins]

This means that if I do npm run start in strudel, and if I can then access one of my cloud-music pieces from the same domain, then the MIDI stuff will work and I can embed Strudel in the piece. Looks like I can do that!

[gogins]

@felixroos responded.

All Strudel packages are in NPM. The REPL is not. But the REPL has a crazy number of node_modules.

michaelgogins@Michaels-MacBook-Pro docs % npm search strudel.cycles
NAME                      | DESCRIPTION          | AUTHOR          | DATE       | VERSION  | KEYWORDS                                       
@strudel.cycles/core      | Port of Tidal...       | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/embed     | Embeddable Web...      | =yaxupaxo...      | 2022-11-06 | 0.2.0    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/eval      | Code evaluator for...  | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/midi      | Midi API for strudel | =yaxupaxo...      | 2022-11-17 | 0.4.1    | titdalcycles strudel pattern livecoding algorave
@strudel.cycles/mini      | Mini notation for...   | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/osc       | OSC messaging for...   | =yaxupaxo...      | 2022-11-17 | 0.3.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/react     | React components...    | =yaxupaxo...      | 2022-11-17 | 0.4.2    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/serial    | Webserial API for...   | =yaxupaxo...      | 2022-11-13 | 0.2.0    | titdalcycles strudel pattern livecoding algorave
@strudel.cycles/soundfont | Soundsfont support...  | =yaxupaxo...      | 2022-11-17 | 0.4.2    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/tonal     | Tonal functions for... | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/tone      | Tone.js API for...     | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/transpile | Transpiler for...      | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/webaudio  | Web Audio helpers...   | =yaxupaxo...      | 2022-11-17 | 0.4.2    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/webdirt   | WebDirt integration... | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave
@strudel.cycles/xen       | Xenharmonic API for... | =yaxupaxo...      | 2022-11-17 | 0.4.1    | tidalcycles strudel pattern livecoding algorave

The packages defined in Strudel are:

core
embed
eval
midi
mini
osc
react
serial
soundfonts
tonal
tone
transpiler
webaudio
webdirt
xen

[gogins]

This is how to get a development environment for Strudel up and running and, I suspect, also to create a production Web site with Strudel (simply replacing the vite host):

git clone https://github.com/tidalcycles/strudel.git && cd strudel
npm i # install at root to symlink packages
npx lerna bootstrap # install all dependencies in packages
cd repl && npm i # install repl dependencies
npm run start # start repl

If I can get cloud-music into scope of this I can proceed.

But this fails.

[gogins]

This does run the REPL:

npm run setup
cd repl
npm run dev

[gogins]

Examining the repl function that needs to call Csound. The hap (event) values are more or less specified in controls.mjs.

I only need to handle some of the possible values of the hap and only for the Csound output.

var csoundOutput = function(hap, deadline, duration) { // Save the original default output, we will still need it. // Switch on which output we want to dispatch. }

This is a bit of a mystery and using MIDI would be simpler.

[gogins]

I'll first try this:

• [x] Turn off the virtual MIDI routing and try to route MIDI output from Strudel to Csound's MIDI input. This might just conceivably work because Csound and Strudel are running in the same domain even though they are probably running in separate threads or processes. It looks like JavaScript cannot create MIDI inputs or outputs, only access ones that already have been opened. WebMIDI has been talking about this for almost 10 years.
• [x] Find where the "start" button in the REPL calls repl or whatever. Find out how to invoke that function programmatically and do that right after starting Csound. I can do this with the macOS IAC driver. Got it to work using jQuery and a better understanding of how IFrames work. This will have to be changed as what I did is fragile and will break if Strudel changes much.
• [x] Hook the haps some other way. I will add a new Csound output to the Pattern class. I should be able to model this very closely on the existing MIDI output. The only real question I see at this point is how to get the running instance of Csound into the scope of this function.
• [x] Finish integrating Strudel REPL IFrame into cloud-music.

[gogins]

I will prepare a pull request for Strudel from gogins/strudel's csound-integration branch after:

• [x] Add a rudimentary API (start, stop, update, getCode, setCode) to the embedded REPL. Works, but start and stop would be better than toggling.
• [x] Add a Csound output method to Pattern (and a scheduler.setCsound/getCsound?).
• [x] Enable csnd to be chained like other Patterns, e.g. to show piano roll displays.
• [x] Move csnd into its own package, can't seem to do it, missing some undocumented configuration or build step. Yup, just had to import csound into the repl/src/App.jsx file.
• [x] Add StrudelReplComponent to the embed package. _Nope, implemented strudel_embed.js and strudel_rep instead, which can be any place in the server filesystem._
• [ ] Add a chnset(name, value) output to send control channel values to Csound at k-rate (similar to Tidal controls). Later. Don't understand this well enough yet
• [x] Finish example piece.
• [x] Push this work into gogins.github.io/cloud-music.

[gogins]

There's a problem in that the cloud-music pages, code, and assets are hosted at https://gogins.github.io/cloud-music, the origin for that site is just https://gogins.github.io. But the embedded Strudel REPL must be hosted right in the origin. That's why everything works locally but not in GitHub pages. Fixed by changing URI for the REPL depending on whether it needs to be something more than just the origin.

[gogins]

Might need this:

<Router>
  <Routes>
    <Route path="/cloud-music" element={<Home />} />
  </Routes>
</Router>

[gogins]

I have changed the embedded REPL to issue requests to repl.html and not to index.html. We'll see if that works with GitHub Pages. Yes it does.

[gogins]

Possibly the final to dos:

• [x] Fix up Controls to match the orchestra.
• [x] Revert to README.md which is easier to maintain.
• [x] Change the names of my customized Strudel files fromembed.js and repl.htmlto strudel_embed.js and strudel_repl.html to avoid overwriting files when updating cloud-music with a new build of Strudel.
• [x] Fix up the Tidal code to use more than just one Csound instrument.
• [x] Sync the cloud-music and Tidal start buttons.
• [ ] Get rid of startup delay, or at least make it clearer what's going on.

[gogins]

The instruments I'm using are:

instr Guitar uses instrument number 1
instr FilteredSines uses instrument number 2
instr Blower uses instrument number 3
instr Sweeper uses instrument number 4
instr Plucked uses instrument number 5
instr FMWaterBell uses instrument number 6
instr YiString uses instrument number 7
instr Harpsichord uses instrument number 8
instr ZakianFlute uses instrument number 9
instr Kung4 uses instrument number 10
instr Kung2 uses instrument number 11
instr Bower uses instrument number 12

[gogins]

I have rebased my fork of Strudel and modified my code to act like Roos' code, including being "patternified." This does mean optional pfields are not sent. I will think more about using channels which, of course, should also be patternified.

[gogins]

In its current state it works very well with a rebased Strudel. If my PR does not go through, I will re-open this issue.