Adds Preset Discovery

[baconpaul]

Curious why the preset stuff isn’t in an extension and is in the top clap directory?

[abique]

Curious why the preset stuff isn’t in an extension and is in the top clap directory?

Because it is not part of the plugin instance. You don't need a plugin instance to index the presets. The preset-load extension is the plugin extension which is used to load the preset from a file which have been indexed using preset discovery.

On the other hand, yes it could go in ext/draft, but somehow as it doesn't extend directly the plugin/host it feels to me that it shouldn't go there.

[defiantnerd]

Because it is not part of the plugin instance. You don't need a plugin instance to index the presets.

I beg to differ :) Presets are different in connected hardwares. At least, the plugin instance should probably provide additional per instance presets.

[abique]

Because it is not part of the plugin instance. You don't need a plugin instance to index the presets.

I beg to differ :) Presets are different in connected hardwares. At least, the plugin instance should probably provide additional per instance presets.

Then we'd need a preset bank extension right?

But in a way it isn't trivial to index because it means:

• it needs a connected hardware to be indexed
• it needs a connected hardware to be loaded

So somehow we'd need to model "connectable" or "dynamically available" content into our browser. I think it opens many questions which aren't trivial. Also if the content is specific to one plugin instance and not to the hardware itself, then what's the point to index it?

[defiantnerd]

Then we'd need a preset bank extension right?

This would be a similar, but different extension for the plugin, I guess. And it is an edge case, indeed.

[baconpaul]

Curious why the preset stuff isn’t in an extension and is in the top clap directory?

Because it is not part of the plugin instance. You don't need a plugin instance to index the presets. The preset-load extension is the plugin extension which is used to load the preset from a file which have been indexed using preset discovery.

On the other hand, yes it could go in ext/draft, but somehow as it doesn't extend directly the plugin/host it feels to me that it shouldn't go there.

So one of my synths has entirely in-memory presets. How would you index those without an instance? I guess I would have a clap which makes a factor which makes and discards an instance?

Also the JUCE Preset API is entirely instance based. I am not sure if I will be able to adapt it to this (other than doing the create-and-discard trick). And there's no requirement JUCE presets are on file system.

So I guess: Are we implicitly requiring the idea of "one physical on disk file per preset" here?

[baconpaul]

Or another way to ask it: is there any requirement at all that the contents of the path is a file you can stat?

[baconpaul]

(what I'm thinking is: if path becomes clap_preset_unique_name then the in-memory thing is easy. I just advertise 50 names and load them from memory when you hit me up. If it is actually something the daw will do fs::is_file on, then that won't work obviously).

[abique]

Curious why the preset stuff isn’t in an extension and is in the top clap directory?

Because it is not part of the plugin instance. You don't need a plugin instance to index the presets. The preset-load extension is the plugin extension which is used to load the preset from a file which have been indexed using preset discovery. On the other hand, yes it could go in ext/draft, but somehow as it doesn't extend directly the plugin/host it feels to me that it shouldn't go there.

So one of my synths has entirely in-memory presets. How would you index those without an instance? I guess I would have a clap which makes a factor which makes and discards an instance?

Also the JUCE Preset API is entirely instance based. I am not sure if I will be able to adapt it to this (other than doing the create-and-discard trick). And there's no requirement JUCE presets are on file system.

So I guess: Are we implicitly requiring the idea of "one physical on disk file per preset" here?

If the preset are in-memory and per-instance we don't index them, what would be the point? Yes we're talking about preset which are on disk.

[abique]

Or another way to ask it: is there any requirement at all that the contents of the path is a file you can stat?

Yes, because we list the filesystem and then do find the preset files by extensions. You can always create a dummy file as a "gateway" to something.

[baconpaul]

If the preset are in-memory and per-instance we don't index them, what would be the point?

Well so users could load them!

Maybe I mis understood. Why are you indexing presets at all? I thought it was so you could show a user a browser of all the presets for all their instruments in a unified place kinda like the arturia analog lab plugin. In that case the storage mechanism doesn’t seem to matter.

If it is a file management thing like sample collection that’s different of course

[baconpaul]

Or another way to ask it: is there any requirement at all that the contents of the path is a file you can stat?

Yes, because we list the filesystem and then do find the preset files by extensions. You can always create a dummy file as a "gateway" to something.

Right so this proposal is opinionated that presets are files on disk you can traverse with find by extension?

extension isn’t unique per plugin though. All of surge pianoteq and serum use .fxp files. Will you crack the file to find the correct synth?

[abique]

If the preset are in-memory and per-instance we don't index them, what would be the point?

Well so users could load them!

I understand that you'd want to load them, but what would be the point for the host to index them and show them in the content browser? When you open the content browser, you don't necessarily have a plugin instance.

I believe that what you're asking for is what @defiantnerd already asked for, which is a way to expose your banks to the host. See https://github.com/free-audio/clap/pull/265#issuecomment-1367572326

Or another way to ask it: is there any requirement at all that the contents of the path is a file you can stat?

Yes, because we list the filesystem and then do find the preset files by extensions. You can always create a dummy file as a "gateway" to something.

Right so this proposal is opinionated that presets are files on disk you can traverse with find by extension?

Yes, we assume that preset is content on the disk. Unless you're downloading your preset from internet, in which case it'd be very hard for us to index with a simple API and maybe not even desirable; your preset if available offline must be stored somewhere on your disk.

extension isn’t unique per plugin though. All of surge pianoteq and serum use .fxp files. Will you crack the file to find the correct synth?

Bitwig can already index and use .fxp files I think or most standard preset container file format. This extension shines with preset which are stored using a custom file format, which the plug-in will parse for the indexer.

At the same time, if the plug-in want to provide the metadata for .fxp files in a given location, it can very well do that an the host will prioritize the plug-in's output or merge it.

I believe that plug-ins which ends up with large preset collections, will have to use files because they are easy to manage, organize and share. And you'll have from 1 to 3 folders to scan and monitor:

1. factory content directory
2. eventually an extra directory where the additional sound packs are installed
3. user directory

[baconpaul]

I understand that you'd want to load them, but what would be the point for the host to index them and show them in the content browser?

So my thinking is the content browser shows all presets for all plugins. When you double click one it creates a track with then plug with that preset loaded. Roughly. Is that right?

if so I think you would certainly want to insert non file presets into the database.

I guess this could all be solved if path was a uri. So file:/// or plugin:// type name.

I’m not that invested but I think we are missing a trick by requiring files for presets as opposed to thinking of this as a database of Uris the clap shares with us and then can respond to.

[abique]

I understand that you'd want to load them, but what would be the point for the host to index them and show them in the content browser?

So my thinking is the content browser shows all presets for all plugins. When you double click one it creates a track with then plug with that preset loaded. Roughly. Is that right?

if so I think you would certainly want to insert non file presets into the database.

I guess this could all be solved if path was a uri. So file:/// or plugin:// type name.

I’m not that invested but I think we are missing a trick by requiring files for presets as opposed to thinking of this as a database of Uris the clap shares with us and then can respond to.

I'm not sure we need to solve problems for things which doesn't matter.

By design, if your preset aren't file on the disk then your approach isn't scalable, you can't share preset etc... So your collection is small.

If the preset are stored within the plugin, then the plugin file itself is a preset container and it'll work with the current API.

If the preset belongs to the plugin instance, then there is no point in indexing them.

The interface as it is today is quite simple, adding support for another approach would require a very different interface, and I'm not sure that the introduced complexity would be worth it.

[baconpaul]

OK!

[Bremmers]

If I understand correctly Paul is saying JUCE based plugins can't (easily) know their own built-in presets without creating a plugin instance. That would be a bit of a problem I think? From a user point of view you'd want to see all presets, regardless of where they are located.

[baconpaul]

If I understand correctly Paul is saying JUCE based plugins can't (easily) know their own built-in presets without creating a plugin instance. That would be a bit of a problem I think? From a user point of view you'd want to see all presets, regardless of where they are located.

yes that is correct. Also those plugins are just index and name. So there's no obvious implementation of this in CJE for people who have the juce preset mechanism implemented. (Surge does not but tuning workbench synth does).

(I was also suggesting that we add a file:// to paths to make them URIs in case it does turn out that we want to index non-filesystem content).

[baconpaul]

I guess in the case of JUCE we could

1. Make every JUCE CLAP itself a preset container
2. Implement the container-get API as making an instance of the plugin
3. Implement the container-contents preset id as juce://14:My Preset Name as returned by the instance
4. Implement preset load to parse that special URI

And then the clap itself shows up as a preset container? Is that the intent?

[Bremmers]

So where do the JUCE presets come from? Do people define them in code in the plugin constructor or something like that?

[baconpaul]

So where do the JUCE presets come from? Do people define them in code in the plugin constructor or something like that?

Lots of them are just references to files. But that file nature is opaque to juce. But the requirement is really an indexed list and load by index so they can be from anywhere including compiled in content or even generated at runtime

the api juce gives is “get num programs” “get program name(I)” and “load program(I)” basicalky

[baconpaul]

(Surge doesn’t use that mechanism by the way. And Alex api is easy for surge to support but we will support it “outside” of juce. A juce dev who follows the juce api is the case I am struggling with)

[baconpaul]

So for more context on JUCE: As you know LV2 includes all the presets inside the LV2 / Manifest / TTL bundle. Juce currently handles this at build time by

1. Building the juce plugin up-to-but-not-including the manifest and bundle
2. Creating an instance of that object
3. Iterating over the programs and
4. Dumping out the ttl

https://github.com/juce-framework/JUCE/blob/bbd6ccbc863edec3150e1e024e4fa3f636802596/modules/juce_audio_plugin_client/LV2/juce_LV2_Client.cpp#L864

It gives the presets an indexed URI name

https://github.com/juce-framework/JUCE/blob/bbd6ccbc863edec3150e1e024e4fa3f636802596/modules/juce_audio_plugin_client/LV2/juce_LV2_Client.cpp#L818

A separate part of the LV2 build chain actually ejects those preset files from a running instance which I can't find right now.

I suppose the idea is we would do the same thing here at runtime rather than at build time when hit up as a container for a juce plugin unless it implemented some cje extension?

[abique]

To me this discussions points out technical issues within JUCE, not within this extension itself.

[Bremmers]

Thanks for the info! So even the JUCE people themselves couldn't think of an easy way to get to the preset info without creating an instance. That's probably a no-go then...

Other frameworks may have similar limitations.

So now we have a nice API to fetch presets without creating plugin instances, but in reality half of the plugins is going to create an instance anyway.

[baconpaul]

https://github.com/iPlug2/iPlug2/blob/49f2de5334f4307496d50beaa20ff57e4c3a6a7b/IPlug/IPlugPluginBase.h#L173

iPlug2 has basically the identical API fwiw.

[baconpaul]

DISTRHO is slightly different but not much. It has a program count internally and then two methods: get the name of a program by index and load a program by index.

You can see its LV2 export does something similar to JUCE

https://github.com/DISTRHO/DPF/blob/22413340a6d8ef2ffbf38ce841fb44c448a1a84a/distrho/src/DistrhoPluginLV2export.cpp#L319

[baconpaul]

(so my initial conclusion - but i may be wrong - is JUCE, iPlug and DISTRHO would all create instances to support this API; and would all be containers; and would all return non-file based parameters. Or would have to extend their APIs)

[baconpaul]

(this also explains why I was initially confused that this wasn't an extension on the plugin. It's not how any of the plugins I've worked on work :) )

Anyway food for thought. Hope it helps. Even if you don't change anything I think making all paths into URIs is a cheap 7 byte protection.

[Bremmers]

I guess this two-phase build approach could work for CLAP too: the first build is used to generate the info we need, and the second build adds this info to the binary.

Probably difficult without official support from JUCE etc.

[baconpaul]

Doing a build time thing won't pick up user presets.

[baconpaul]

also: i really hate having to build load and run my plugin in my build pipeline. It would be a bummer if that was the solution.

[Bremmers]

Doing a build time thing won't pick up user presets.

User presets are always files I think? Perhaps that can be supported easily without using any plugin framework features?

[baconpaul]

Few other observations from some software which may help!

I have the Arturia vintage collection. They have their presets in /Library/Arturia/Presets as a file per, but they don't have any extensions.

paul /Library/Arturia/Presets/Jup-8 V4/Factory/Factory  % ls /Library/Arturia/Presets/Jup-8\ V4/Factory/Factory/Simplicity 
/Library/Arturia/Presets/Jup-8 V4/Factory/Factory/Simplicity

I have the KORG collection. Like Surge they store a bunch of fxp files in /Library/Application Support so that would work!

Pianoteq seems to put user presets in ~/Library/Application Supprot/Modartt but I can't find their factory presets on my hard drive. I wonder if they are embedded in the models? I can only find the nks files. There is a very large file called 'presources.dat' which ships with it but it seems to be encrypted, although the executable clearly references it. Maybe we should ask them, but it looks like they have factory presets in 'single file which the plugin cracks' maybe? But hard to tell.

[baconpaul]

Doing a build time thing won't pick up user presets.

User presets are always files I think? Perhaps that can be supported easily without using any plugin framework features?

sure? you can't store them in a sqlite db that requires code to extract? and you can know the path to them without starting any part of your plugin? neither of those things are improbably are they? In fact, that's exactly what u-he synths do.

[baconpaul]

In fact, that's exactly what u-he synths do.

Actually this is wrong. u-he synths update a sqlite database but do indeed store the user preset in an file (just like surge does)

[abique]

Requiring a plugin instance to extract a preset meta-data is a plugin implementation detail.

I'm sure you can find a solution to be able to extract the metadata stored into a preset file, even if under the hood you need to use a hidden/temporary plugin instance.

For surge, in worst case, you can lazily create a plugin instance which you keep around for loading the preset and extracting the metadata. This is dirty but it'll work for you until you figure a better way to extract meta-data out of your preset files.

[baconpaul]

For surge it is way easier since we don't actually use that JUCE API ourselves and I can instead just give you a list of fxp files relatively easily with a small bit of code that requires only a subset of the synth there's; just no way to write that code for a generic framework plugin.

Given the generic juce, iplug and distrho api, you will have the plugin act like a container which traverses the index; either those frameworks will need to change their API (and everyone adapt), will need to not support this, or will need to create instance at scan time. If that's OK, then that's OK!

[abique]

I have the Arturia vintage collection. They have their presets in /Library/Arturia/Presets as a file per, but they don't have any extensions.

In such case we could say that instead of having just the extension we have a regex or a globing pattern maybe...

[abique]

In fact, that's exactly what u-he synths do.

Actually this is wrong. u-he synths update a sqlite database but do indeed store the user preset in an file (just like surge does)

The sqlite db is just their own index, but the u-he implementation of preset discovery doesn't use it.

[abique]

If I were writing a plugin framework, I'd store the presets as follow:

<preset>
 <metadata>...</metadata>
 <state>BLOB</state>
</preset>

[baconpaul]

In fact, that's exactly what u-he synths do.

Actually this is wrong. u-he synths update a sqlite database but do indeed store the user preset in an file (just like surge does)

The sqlite db is just their own index, but the u-he implementation of preset discovery doesn't use it.

Yeah surge is exactly the same.

[baconpaul]

If I were writing a plugin framework, I'd store the presets as follow:

<preset>
 <metadata>...</metadata>
 <state>BLOB</state>
</preset>

if i was writing a plugin framework there's lots of stuff I would do differently too :)

But the plugin frameworks our users are using have a getNumPrograms/getProgramName/loadProgram type API, and are silent on whether those programs map to files or not.

If we just want to say "OK you don't get to participate unless you write custom code outside your framework", or if we want to say "OK we will have to make an instance of your plugin at scan time", then this isn't a problem per se. But there's no third thing we can say!

[abique]

If I were writing a plugin framework, I'd store the presets as follow:

<preset>
 <metadata>...</metadata>
 <state>BLOB</state>
</preset>

if i was writing a plugin framework there's lots of stuff I would do differently too :)

But the plugin frameworks our users are using have a getNumPrograms/getProgramName/loadProgram type API, and are silent on whether those programs map to files or not.

If we just want to say "OK you don't get to participate unless you write custom code outside your framework", or if we want to say "OK we will have to make an instance of your plugin at scan time", then this isn't a problem per se. But there's no third thing we can say!

You can still have an instance under the hood for loading the preset and extracting the metadata, what's wrong with that? You can also start discussing with JUCE so they improve their preset file format and do a v2 where you can extract metadata without using a plugin instance?

[baconpaul]

You can still have an instance under the hood for loading the preset and extracting the metadata, what's wrong with that? You can also start discussing with JUCE so they improve their preset file format and do a v2 where you can extract metadata without using a plugin instance?

JUCE doesn't have a preset format. They just have a named program index you can load. Just like iPlug2.

But yes if I make instances I can at least give you a list as a container. I thought we were trying to avoid having a plugin instance at index time but if we aren't then there's no problem!

[abique]

But the plugin frameworks our users are using have a getNumPrograms/getProgramName/loadProgram type API, and are silent on whether those programs map to files or not.

Those are supposed to be for MIDI program change, to mimic the way in the 80s to manage multiple presets inside a hardware synth. And there are multiple interpretation for what program change should do, one of them is to be able to switch preset within one sample, hence requiring to have memory and so on pre-allocated and ready to be used at any time. (MIDI Program Change is received on the audio thread)

[baconpaul]

But the plugin frameworks our users are using have a getNumPrograms/getProgramName/loadProgram type API, and are silent on whether those programs map to files or not.

Those are supposed to be for MIDI program change, to mimic the way in the 80s to manage multiple presets inside a hardware synth. And there are multiple interpretation for what program change should do, one of them is to be able to switch preset within one sample, hence requiring to have memory and so on pre-allocated and ready to be used at any time.

I feel like we are talking past each other. And this is nothing to do with surge. Surge can support the API as written just by side-stepping JUCE like we did for all the other features of CLAP we wanted.

Right now if you write a juce iplug2 etc plugin and support getNumPrograms / getProgramName / loadProgram then your presets show up in logic and reaper and so on in the plugin preset list if you make an AU or a VST3. That's kinda all you have to do and it just kind of works. That is how the plugins defacto expose their factory presets.

So first question: Do we want a plugin which coded to that mechanism to also expose those presets to CLAP using this mechanism. I had assumed "yes" but if the answer is "no" then everything is fine.

Now if the answer is "yes" lets think about how that happens. You need a plugin instance and there is no preset file. There's just a dll which can respond to those three API points. How do you do that with the API as proposed then? Well I think you make '.clap' files in the install directory of your plugin preset containers, but that seems like it would scan every clap in /Library/Audio/Plug-Ins/CLAP and so on. So I'm actually not sure how I advertise a single dll which contains the preset access.

But I don't see a way with the API as posed, if the answer is "yes", to write it super easily given the API. Or maybe not even at all. I think you could fix it by having a 'container file' as opposed to 'container path and extension' api which the plugin can then crack.

The alternate thing is to tell all plugin writers "you must use this alternate API to expose your files". Like I said that's about 10 minutes of work for surge. I just have no idea how to do it for clap juce wrappers for generic plugs right now.

See what I mean?

[baconpaul]

I propose adding adding a string constant for the location that indicates 'this CLAP file', like $PLUGIN or %PLUGIN%. Or URIs as previously mentioned, where file:///path/to/files refers to the /path/to/files directory, plugin:/// refers to the .clap file, and plugin:///foo could refer to a foo bank inside of the plugin's file (if that's useful to support).

I think the file:// and plugin:// URI solution or some way to refer to internal presets is a very good idea also.

[abique]

Right now if you write a juce iplug2 etc plugin and support getNumPrograms / getProgramName / loadProgram then your presets show up in logic and reaper and so on in the plugin preset list if you make an AU or a VST3. That's kinda all you have to do and it just kind of works. That is how the plugins defacto expose their factory presets.

So first question: Do we want a plugin which coded to that mechanism to also expose those presets to CLAP using this mechanism. I had assumed "yes" but if the answer is "no" then everything is fine.

Programs are out of the scope for this extension. In Bitwig we also show vst2 and vst3 programs.

CLAP doesn't have a program/bank management extension yet.

[baconpaul]

OK then I mis-understood the extension. And no JUCE, iPlug, DISTRHO etc plugin will generate a scannable preset list using their in-framework API for programs. Gotcha.