Discussion and development of web/wasm version

[falkTX]

For those unaware, https://cardinal.kx.studio/ is totally a thing now for some reason. Also https://minicardinal.kx.studio/ for a mini/stripped-down version that loads faster.

There are some issues and things still to do. Because this is for playground and testing and fun, and not all that serious, let's not open more than 1 ticket for web related things. Everything web/wasm related should be on this ticket.

Stuff to do or implement:

• [x] audio input
• [ ] touch events
• [x] clipboard (do fake internal one if not possible to expose in browser)
• [x] webmidi
• [ ] host time controls (on menu bar, providing fake but valid BBT values)
• [x] intro message explaining what it all is, what to expect etc
• [x] detect incompatible browser setups and refuse to even load (noscript, no simd, no webgl2)
• [x] build libsndfile and fftw for extra module features
• [ ] minimum gui size (CSS is enough?)
• [x] allow url parameters to tweak page behaviour, also load patches as base64 encoded string
• [x] test loading all modules and disable those with custom non-GLES2 compat code

Known issues:

• [x] Loading zst compressed project file fails to load patch.json
• [x] opening salamonis-mono-regen example patch triggers an exception
• [x] going full screen can mess up with the viewport size and coordinates
• [x] ctrl+a clears all text
• [x] audiofile player is missing regular audio support (only mp3 works)
• [x] audiofile and midifile inside ildaeil fail to show file dialogs
• [x] latest changes made performance in chrome worse?? firefox still fine..
• [x] compressed data assets > 50Mb
• [x] missing versioning to js, wasm and data downloads (so they are updated automatically when reopening the page as needed)

Stuff to fix or workaround:

• [x] apply high-dpi event offset
• [x] test and implement fractional scaling "high-dpi" (only supports 2x and 3x now)
• [x] prevent ctrl/cmd+ shortcut keys from going to the browser
• [x] prevent browser from zooming the page
• [x] oddities with mouse down and leaving browser window

Undecided:

• [ ] host parameters
• [ ] host cv
• [x] persistent storage (client-side browser DB)
• [x] bundle a few simple LV2 plugins (mda, caps, tap, etc)

[falkTX]

Leaving a note to myself, https://github.com/Dreamtonics/juce_emscripten/blob/master/modules/juce_audio_devices/native/emscripten/RtMidi.cpp has rough but working code for webmidi. Not that it is complicated to do, but helps to have something to compare against.

[falkTX]

high-dpi screen should work now. tested locally with a mac mini.

if you have such a screen, let me know how it works!

[john-the-bard]

The ping pong delay from MSC makes a ton of noise if you load it, this basic patch which should play a single plucked note every second into a short and feedback heavy delay.

NOISE WARNING the output is connected and it immediately clips.

NOISEWARNING_Pluck.tar.gz

[diimdeep]

Insanity. Wowzers. Would be cool to load patch from url encoded parameter, https://cardinal.kx.studio/?patch=encoded_blob like this

[falkTX]

Insanity. Wowzers. Would be cool to load patch from url encoded parameter, https://cardinal.kx.studio/?patch=encoded_blob like this

I thought about this, and while I typically dislike such approaches, but I think it makes sense here. A way to generate such link directly from the page would also be quite cool, so we can:

1. open cardinal web page
2. make a patch showing a specific thing
3. use file -> generate pasteable link (or whatever better name there is)
4. send that link to someone for quickly trying the patch

there are 2 things though that need to be done before we can tackle this:

1. fix opening zst compressed patches
2. allow for versioned caching of wasm data, so page reloads are instant (loading 100Mb of resources everytime to check new patches is just painful)

will add to the todo, thanks for the idea!

[pauldavisthefirst]

drag doesn't work to scroll boundaries of view area. click on output "socket", drag upward to reach module that is currently "off-screen" .. drag will break.

[pauldavisthefirst]

if scroll is used to zoom, easy to get to a point where cardinal does not occupy entire browser screen, at which point the coordinate determination seems to be messed up. drags will affect things not under the mouse.

[falkTX]

drag doesn't work to scroll boundaries of view area. click on output "socket", drag upward to reach module that is currently "off-screen" .. drag will break.

mentioned on the 1st post as

oddities with mouse down and leaving browser window

and

if scroll is used to zoom, easy to get to a point where cardinal does not occupy entire browser screen, at which point the coordinate determination seems to be messed up. drags will affect things not under the mouse.

known issue, I fixed it already locally but didnt deploy a build for it yet

[falkTX]

Small update: fixed up the ildaeil audiofile and midifile plugins, adjusted for more high-dpi screens, and updated view menu to follow official Rack 2.1.2 changes.

With the ildaeil fixes it is now possible to get some MIDI going. Follow these steps:

1. Add ildaeil module
2. Add ildaeil expander midi output, place it touching the right side of ildaeil
3. In ildaeil, switch to internal plugins (it is JSFX by default)
4. Load "MIDI File"
5. Set "Host sync" to off, "Repeat Mode" to on
6. Press "show custom gui" button
7. Pick a MIDI file from your PC

Right-click the expander module for polyphony options.

[falkTX]

Good news, the issue with loading compressed project files is fixed. (or better said, a workaround is in place) Somehow zstd extracts the tar with incorrect permissions, and then fails to load. We now go for 777 for any file to be open, and all is good. The "files" are all virtual anyway, so doesn't matter on the web context.

Also, you can now pass it a patchstorage "slug" to the URL as a way to load that patch on start. For example https://cardinal.kx.studio/?patchstorage=fern-sequence and https://cardinal.kx.studio/?patchstorage=example-patch-2 The URL parameter is the same that is used on patchstorage.com views, e.g. https://patchstorage.com/example-patch-2/

[falkTX]

Another update, this is fun :laughing:

Thanks to previous work done in https://github.com/DISTRHO/PawPaw I could just repurpose it to build generic opensource libs for wasm. For now that is libogg, libvorbis, flac, opus, sndfile, liblo, fftw and lv2 related libs. This helps a lot on building opensource plugins in wasm. Also these extra libs allow to get the extra/proper file support in the audiofile module and brings in the audio-to-cv-pitch module (which required fftw)

I did a few more tweaks and made Cardinal able to load audio plugins from the "filesystem" (JSFX dont really count, they are not binaries). I have compiled artyfx, die-plugins (imported ardour fx), fomp and mda LV2 plugins, and added them to the site/webapp. Just open up ildaeil and they will be there for the LV2 option.

[falkTX]

ok a couple more things now:

• fixed up event key input handling a bit, ctrl+a no longer clears all text
• keyboard is no longer propagated to browser, so stuff like ctrl + scroll works now (some key combos like ctrl+n and ctrl+t cannot be overridden)
• pointer lock is implemented and enabled by default (opt-out in view menu)
• added full screen menu option

[dromer]

Going out of full screen returns to the wrong size render (x/y mouse coordinates are still correct, but don't align with the view).

[falkTX]

Going out of full screen returns to the wrong size render (x/y mouse coordinates are still correct, but don't align with the view).

I have some ideas for this, the fullscreen mode typically changes the viewport to render in "pixel perfect" mode by default, while typically contents are scaled instead. When entering fullscreen, we can request how to deal with the viewport and keep things as-is. Will try later

[falkTX]

And more news for the web side, Web MIDI is implemented! Both MIDI input and MIDI output. Does not work on Firefox, as it is not implemented there. Only Chrome/ium-based browsers work for that. And if you are using an Ubuntu based system with snaps, alsa-midi access is disabled there so this wont work.

The web audio stuff was rewritten from scratch too, I was using SDL before but it was doing some weird stuff behind the scenes, specially for audio input. Oh yeah, I got audio input to work too, but did not enable yet as the permission request for it would make the page look too suspicious. ("Why does this random page want mic access?") A button on the menubar to request mic access might be the best, to handle later. I did try the web audio input, and quality is really awful, so we are not losing much.

For the curious ones, relevant code for this web audio + midi is at https://github.com/DISTRHO/DPF/blob/develop/distrho/src/jackbridge/WebBridge.hpp This is not using worklets yet, as that has special security considerations and just a pain to setup. With the current single-threaded approach we do not get the best performance, but in turn it is also possible to just run the html files directly in any compatible browser.

I will try to have the html/js/wasm files built as part of 22.07 release on CI. There is nothing special about it, just emscripten with USING_GLES=true build flag.

[falkTX]

Audio input is now enabled. And MIDI is disabled by default, as I try to optimize things a little big (chrome is behaviour worse after changing to custom written code vs old SDL2 usage, for some reason)

You can enable audio input and MIDI under the engine menu.

Audio input quality is really subpar here.. and forces itself as mono even if we request 2 channels. You can see the blocky-ness of it in a spectrum, really ugly :(

But, well, feature is enabled now.

[falkTX]

A bit more news on the input and processing.. For one, the issue I had with quality seems related to pulseaudio. using chrome with alsa the input quality is much better.

Chrome cannot do stereo input though, only Firefox does.

Also "fixed" the choppy audio in chrome after the latest updates, turns out it is just tweaking buffer size. SDL was likely buffering things, not using a real 512 samples as requested. Default now changed to be 2048 samples, but you can change this in the engine menu. Might help make the webapp work on slower machines.

[falkTX]

There is a mini version now at https://minicardinal.kx.studio/ Contains only core and fundamental modules, extras like carla/ildaeil are also not in there.

The mini version assets have a combined 15Mb size. It is small enough that Firefox is happy to cache the page, so that's nice :)

Feel free to use and share!

[falkTX]

A few more things:

touch events are now enabled, sorta.. because neither Rack or pugl can deal with touch events, they are converted to mouse events in hopefully a way that makes sense.

• only 1 finger used at the moment.
• quick press does a mouse click
• press and hold does mouse right click
• press and move does a mouse click + drag (so if you dont want to trigger a right click menu, move finger after pressing)

moving around the canvas is still an issue. I am considering mapping 2 finger movements to mouse wheel, will need to run some tests.. I managed to make it load on my phone, but since it is a > 7 year old one, its performance is quite terrible. otherwise works. testing and feedback is appreciated.

the hosted files are now gzip compressed, which makes them smaller and thus faster to load. we can gzip compress the files and host both compressed and uncompressed, then apply some apache magic to load compressed files if browser supports it. it is a nice reduction in size!

I will push the setup for this later.

A final thing added was detection of required browser features before trying to load the web assembly program. Old browsers cant load Cardinal web, but that is kinda expected, it is nice to have a list of unsupported features.

And speaking of unsupported browsers, with this we now know why it does not work on Safari - lack of wasm simd support. That means it wont work on any iOS device at the moment too.. Apple seems to be pushing in wasm territory though, so perhaps this will be coming soon. https://platform.uno/blog/the-state-of-webassembly-2021-and-2022/ seems to give a good overview of the current situation.

[falkTX]

Oh, and another small change I forgot to mention: when browser tab/window is not visible, cardinal wont spend time rendering the webgl gui anymore. which makes it use less cpu when running in the background.

[falkTX]

Good news, with more optimizations and compression, the wasm binary and data downloads are now below <50Mb.

This is great because, at least in Firefox, downloads with less than 50Mb are allowed to be cached.

Additionally I did some tests for clipboard support. While it is possible to support it, and I verified that it actually works, mapping it to current APIs as used by DPF and Rack requires us to use asyncify which complicates the build and makes it slower. in the end it seems not worth it to pursue. note that this is only for pasting from system clipboard into the site/webapp, the other way around does not have such strong limitations. we can even fake/force copying out of the webapp by creating a hidden html element, setting its text to the one we want and instruct the browser to copy from it.

so starting today we have a sorta fake clipboard in the webapp, with it having internal data for dealing with copy&paste and allowing to copy out of the webapp as an extra.

[falkTX]

We have URL parameters handling now!

Both passing full patch, or a link to some file online.

Use patch keyword for patch file in URL, alike https://cardinal.kx.studio/?patch=eyJtb2R1bGVzIjpbeyJpZCI6IDEsInBsdWdpbiI6IkNhcmRpbmFsIiwibW9kZWwiOiJUZXh0RWRpdG9yIiwiZGF0YSI6eyJldGV4dCI6IldoYXQgYSBoYWNrIHdvdyJ9fV0sImNhYmxlcyI6W119Cg==

Use patchurl for loading an online patch file, alike https://cardinal.kx.studio/?patchurl=https://patchstorage.com/wp-content/uploads/2022/07/JTB-wasm_house_toshare.vcv

The embed patch in the URL is kinda disappointing, even with zst compressed data we easily get into the limits of how long we can make the URL. It works, but it is very limited.. It always uses base64 encoded string, if there is a better way to pass binary data as string please tell me.

The remote file seems the best approach here. you can host a file on a pastebin like service, and pass the raw contents url as parameter to cardinal web app. Can be quite useful to quickly show some patch to someone.

Have fun!

[Ulfmusic]

Hi Could I please have some simple instructions on locally hosting the wasm version of Cardinal? Thanks for a great DAW

[falkTX]

Download the artifacts and serve them under a web server. There is no magic needed, these are just static files. Either grab the 22.07 release files, or the latest build from the actions tab.

[aolney]

In a word, amazing :partying_face:

I'm looking at incorporating this into a tutorial website/book I'm working on. Some questions with that in mind:

• Looks like computer keyboard for midi input was not ported over. Staying in-browser and ideally within Cardinal, is there an option besides the Impromptu 12-key module? This type of capability is really useful for beginner patches.
• Can menu-based settings be set from the outside? It would be great to
  • Detect Chrome and automatically turn off Lock cursor while dragging to avoid "Press esc to show your cursor" messages
  • Possibly change scrolling behavior for mobile phones (not getting a bottom scroll bar on Android currently)
  • Possibly change the buffer size for mobile phones to something larger than the default

[falkTX]

Cardinal does not do midi-like input via keyboard, never did. For such changes, you can fork the repo and do the changes as you wish.

[falkTX]

Small update: The main https://cardinal.kx.studio/ has been updated to the latest code. Sadly this brings the size of the compressed assets above 50Mb again, and thus they are not being cached by the browser anymore. Files are also not being automatically updated (their cache invalidated) so that is a small thing to fix too

[aolney]

Thanks - going back in the build history, it looks like wasm builds were commented out in build 1188.

The last successful build before that was build 1182. Would this build be the best build for wasm that is still cache friendly?

FWIW I currently have build 1153 deployed for testing/evaluation. I'm deploying through GitHub Pages, so caching is helping me avoid going over their bandwidth quota.

[falkTX]

wasm builds are enabled already. they were failing due to new modules having a lot more code which made the build machine go out of RAM. I have reverted voxglitch to the same version as used in 22.07 (with minor fixes), but we are at the limit of what the free github CI VMs can do..

[aolney]

I think we're missing a file - patchurl.php seems to be required to load a patch from patch storage.

[aolney]

Brotli compression seems to solve the caching problem. On build 1153, I get these results

File	Size on disk
CardinalNative.data	89.4
CardinalNative.data.zip	51.2
CardinalNative.data.br	35.8

The brotli file at level 11 compression is a 30% reduction over zip and 60% reduction over uncompressed. If browser cache entries must be < 50mb, then the original file could approach 125mb and still be cached using brotli level 11. I'll report back after I configure nginx for for brotli and live test.

[falkTX]

https://github.com/DISTRHO/Cardinal/blob/main/src/emscripten/patchstorage.php has one of the files with https://github.com/DISTRHO/Cardinal/blob/main/src/emscripten/htaccess being what I use in terms on apache setup.

The patchurl.php file I did not post yet because I need to make it better, this is stuff that ends up running server-side after all. So people more knowledgeable of php than me can potentially do bad things with it.

[aolney]

Thanks :) Brotli works great live as expected; I'm using nginx.

I get caching on Firefox for everything, but for some reason on Chrome CardinalNative.data will not cache. It looks like Chrome is refusing to send if-none-match in its request.

Did you ever get Chrome to cache CardinalNative.data?

[falkTX]

Good question, I never use chrome for daily stuff so I didnt check there. Could be it has a more strict caching requirement. You can try https://minicardinal.kx.studio/ as comparison, as that will have much lower file sizes. If it is possible to cache, the minicardinal will say.

[falkTX]

@aolney btw, because you mention incorporating into a website/book, be aware that due to licensing of many modules artwork and design, Cardinal is strictly a non-commercial project. You cannot involve anything commercial in regards to Cardinal (be it advertising for revenue, donations, merchandise, selling of services, etc, whatever can make something commercial).

If you want to have commercial services related to a Cardinal build, you need to strip out a considerable amount of modules and all the "component library" stuff from VCV.

[aolney]

@aolney btw, because you mention incorporating into a website/book, be aware that due to licensing of many modules artwork and design, Cardinal is strictly a non-commercial project. You cannot involve anything commercial in regards to Cardinal (be it advertising for revenue, donations, merchandise, selling of services, etc, whatever can make something commercial).

If you want to have commercial services related to a Cardinal build, you need to strip out a considerable amount of modules and all the "component library" stuff from VCV.

No worries, the book is CC-BY-NC-SA 4.0 ;)

BTW I think I've run this down in Chrome - Chrome is not caching .data files over some ??? size. That page also discusses some ideas about breaking data into chunks, and this page discusses manually caching to IndexedDB.

So although Brotli is nicely accelerating things, it does not reduce the size of the .data file enough for Chrome to cache it; I went back to build 1153, where the .data file is 35.7MB with brotli, and it's still too big to cache.

[falkTX]

IndexedDB is being phased out by browsers, not something to rely on

did you try the wasm-mini builds? there is a small chance chrome doesnt cache any wasm data at all

[aolney]

Yes sorry, the minicardinal files are completely cached, both the .wasm and the .data. So I know that Chrome is capable of caching with the headers I'm using, but it decides not to if the .data file is too big (it always caches the .wasm file in my tests).

[aolney]

Found a bug in Bogaudio Analyzer-XL. If you connect Noiz to it, it throws an exception and crashes the page.

BTW I made a repo for dockerized nginx deployment using brotli and php here.

[aolney]

Found a bug in Bogaudio Analyzer-XL. If you connect Noiz to it, it throws an exception and crashes the page.

It seems the Bogaudio analyzers crash with any input. I assume this is the reason for Sassy Scope? Unfortunately Sassy is a bit too noisy to get clear frequency peaks with all the settings I've tried, though it's fine for showing spectrum for many other uses.

[falkTX]

sassy was because we could, it is an imgui-based widget after all and can be quite useful. I suppose bogaudio doesnt crash on desktop? Does it crash on windows or linux 32bit perhaps? it could be a 32bit compat thing, as wasm target runs in 32bit mode as well.

[aolney]

Does it crash on windows or linux 32bit perhaps? it could be a 32bit compat thing, as wasm target runs in 32bit mode as well.

Apologies for the delayed response; it's crashing on 64bit linux in Cardinal but works fine on the desktop, also 64bit.

update 11-16-22: bug has disappeared; suspect firefox update was to blame BTW found a new strange bug today that is periodically appearing on Firefox when I try to load a patch from the file menu:

If I reload the page it goes away. There's no /tmp/Cardinal... on my disk, so I assume it's referring to a virtual filesystem?

[diimdeep]

Tip: FF users can increase cache limits in about:config e.g. browser.cache.disk.max_entry_size > 50mb and browser.cache.disk.capacity ~ 1gb

[aolney]

Tip: FF users can increase cache limits in about:config e.g. browser.cache.disk.max_entry_size > 50mb and browser.cache.disk.capacity ~ 1gb

Nice! If only Chrome had a similar option

[davybjj]

Hello, Thank you for your work. I don't know if this is the right place. If the following helps. I've been using the web version for a while in the office because I can't install it. Since the last build a few days ago, the app no longer wants to open on a Windows10 + Chrome Version 108.0.5359.95 (Official Build) (64 bit). It was working until last weekend. I also tried at home on ubuntu + chrome, that doesn't work either. And also with ubuntu 22.04+ firefox with alsa behind, which crashed the machine. For all practical purposes, here is the message in the html page: This is the windows10+chrome computer: I am not a programmer but the html-console displays error messages. cardinal.kx.studio-1670427286423.log

[falkTX]

no ideas there, logs are not useful. you tried to access it on work office/space? they might have blocked the domain?

I made the page versioned just a few moment ago, to fix the issue of updates being rolled out partially. now the data package and web assembly will always match

in any case, you can download the release wasm zip files and just open the html file inside, so you dont rely on an online version being there. that way you pick which version to use too.

[aolney]

@davybjj in your log file, the last line is this:

Échec du chargement de la carte source par les Outils de développement : Impossible de charger le contenu de chrome-extension://cfhdojbkjhnklbpkdaibdccddilifddb/browser-polyfill.js.map : Erreur système: net::ERR_FILE_NOT_FOUND

This makes me wonder if the problem is an extension you've installed. I googled this message, and it seems to be associated with Adblock Plus. However it could also be caused by other extensions.

[davybjj]

Thank you, I will try.

Le mer. 7 déc. 2022 à 18:27, Andrew M Olney @.***> a écrit :

@davybjj https://github.com/davybjj in your log file, the last line is this:

Échec du chargement de la carte source par les Outils de développement : Impossible de charger le contenu de chrome-extension://cfhdojbkjhnklbpkdaibdccddilifddb/browser-polyfill.js.map : Erreur système: net::ERR_FILE_NOT_FOUND

This makes me wonder if the problem is an extension you've installed. I googled this message, and it seems to be associated with Adblock Plus. However it could also be caused by other extensions.

— Reply to this email directly, view it on GitHub https://github.com/DISTRHO/Cardinal/issues/287#issuecomment-1341317978, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3R42ZOZZK4OD43BBV3YUZ3WMDCJTANCNFSM522BT5BQ . You are receiving this because you were mentioned.Message ID: @.***>

[davybjj]

Hello, I come back to you for my feedback. I cannot run the WASM version. As soon as the html link is pasted, there is an error message: Exception thrown:Script error. In Browser Inspector: Browser does not accept CORS origin from local c:/. After research no NO-CORS plugin could work to circumvent it. Only one server will apparently be able to function. I can't try on this workstation. Disabling browser add-ons did not allow me to run the classic web version. On the other hand, I can access the mini web version as before. So it shouldn't be a domain issue. It even started on my modest android mobile. It's great, she makes the sound. I guess it's in development. Thanks again for your work, I follow you.

Le jeu. 8 déc. 2022 à 14:23, Davy Voiturier @.***> a écrit :

Thank you, I will try.

Le mer. 7 déc. 2022 à 18:27, Andrew M Olney @.***> a écrit :

@davybjj https://github.com/davybjj in your log file, the last line is this:

Échec du chargement de la carte source par les Outils de développement : Impossible de charger le contenu de chrome-extension://cfhdojbkjhnklbpkdaibdccddilifddb/browser-polyfill.js.map : Erreur système: net::ERR_FILE_NOT_FOUND

This makes me wonder if the problem is an extension you've installed. I googled this message, and it seems to be associated with Adblock Plus. However it could also be caused by other extensions.

— Reply to this email directly, view it on GitHub https://github.com/DISTRHO/Cardinal/issues/287#issuecomment-1341317978, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3R42ZOZZK4OD43BBV3YUZ3WMDCJTANCNFSM522BT5BQ . You are receiving this because you were mentioned.Message ID: @.***>

[falkTX]

Some small bit of news.

I investigated the pointer lock issue on chrome, seems to be bug in emscripten as reported in https://github.com/emscripten-core/emscripten/issues/9681 I added a workaround for now, so default knob behaviour works under chrome-based browsers now.

Some input keys were not being accepted before, like - = ^ ~ ? characters, should now be corrected.

Updated the web app again just now, not sure I mentioned it in previous posts, but we got stoermelder-packone, MindMeld PatchMaster, Sapphire and SurgeXT modules now.

Not from Cardinal/DISTRHO side, but related to the discussion, the just released Firefox 108 now supports WebMIDI :rocket: You get this weird scary popup warning, but afterwards it appears to work.