search

lukasberbuer / rt-vamp-plugin-sdk

Real-time Vamp plugin SDK for C++20
https://lukasberbuer.github.io/rt-vamp-plugin-sdk/
MIT License
10 stars 0 forks source link

Compatibility (not an issue) #1

Open gesellkammer opened 6 months ago

gesellkammer commented 6 months ago

Hi, nice project. I maintain vamphost (https://github.com/gesellkammer/vamphost), a vamp host in python, and some vamp plugins as well.

If I understand correctly, plugins compiled with this SDK will not be compatible with hosts using the original SDK, right? Have you tried submitting these changes upstream? My own experience in this regard has been somewhat frustrating, so curious to know if this is a trend.

lukasberbuer commented 6 months ago

Hi @gesellkammer,

I had to made those restrictions to avoid memory allocations during processing and to use vamp plugins in performance-critical applications.

Chris Cannam (@cannam), the author of the original SDK ,and I were in contact during the development of rtvamp. I guess my restrictions are to strict to be made available upstream. But in my opinion, there are quite a lot of optimizations possibilities in the C++ SDK which should be made. Sadly, https://github.com/c4dm/vamp-plugin-sdk seem to be maintained poorly. I tried to add CMake integration 2 years ago and never got a response.

What are your goals with vamp and your project vamphost?

cannam commented 6 months ago

Thank you for tagging me Lukas.

The Vamp-related projects under c4dm "namespace", and those at code.soundsoftware, effectively became unmaintained when I left c4dm in mid-2020. I continued to do some maintenance of Sonic Visualiser and a few related things where possible, but time and energy were limited and it seemed appropriate to de-prioritise those projects whose organisation I didn't feel I truly had authority to represent any more anyway. Unfortunately there were quite a lot of projects at c4dm that I was really the only maintainer of, so the situation wasn't ideal.

But things may be looking up. I am in fact doing some work at c4dm again now, including on Sonic Visualiser and related code, and I would also like to get some of those projects into better shape for broader and more collaborative support. It would probably make a big difference simply to move the reference Vamp SDK projects to a Vamp plugins organisation on Github, for a start - ideally one of which I am not the only admin! I've just created a vamp-plugins organisation to this purpose.

gesellkammer commented 6 months ago

plugins created with rtvamp should be compatible with all other vamp hosts plugins created with the original SDK will be compatible with the rtvamp host if the fulfill following requirements: https://github.com/lukasberbuer/rt-vamp-plugin-sdk?tab=readme-ov-file#plugin-restrictions

This is great. Where do you see the greatest speed-up, on the host or on the plugin?

I use vamp plugins for feature extraction for speech/music analysis and machine learning mostly within this project: https://maelzel.readthedocs.io/en/latest/index.html. When analyzing hours of audio any speed up is helpful.

lukasberbuer commented 6 months ago

I see two major points of speedup:

  1. Less heap-allocations and creation of temporary objects with every process call. This happens 50% on the plugin side (conversion to C API) and 50% on the host side (conversion back to C++ API). This plugin/host SDK removes some overhead for each call to the process member function. If the computations inside the process function are cheap, the overhead is relatively high. If tried to capture this with the RMS benchmark. You see, that the difference between the SDKs is higher for smaller block sizes (= less computations). At some point, the performance of the SDKs converges, because the absolute overhead becomes relatively small. RMS benchmark
  2. Better multi-threading performance of the plugins due to the internal std::shared_mutex. Shared mutexes are usually used in situations when multiple readers can access the same resource at the same time without causing data races, but only one writer can do so. This causes some measurable compared to the std::mutex used in the official SDK. Multi-threading benchmark
lukasberbuer commented 6 months ago

Thank you for tagging me Lukas.

The Vamp-related projects under c4dm "namespace", and those at code.soundsoftware, effectively became unmaintained when I left c4dm in mid-2020. I continued to do some maintenance of Sonic Visualiser and a few related things where possible, but time and energy were limited and it seemed appropriate to de-prioritise those projects whose organisation I didn't feel I truly had authority to represent any more anyway. Unfortunately there were quite a lot of projects at c4dm that I was really the only maintainer of, so the situation wasn't ideal.

But things may be looking up. I am in fact doing some work at c4dm again now, including on Sonic Visualiser and related code, and I would also like to get some of those projects into better shape for broader and more collaborative support. It would probably make a big difference simply to move the reference Vamp SDK projects to a Vamp plugins organisation on Github, for a start - ideally one of which I am not the only admin! I've just created a vamp-plugins organisation to this purpose.

This are great news @cannam! You created an amazing ecosystem for analysis of sound-like data with the Vamp plugins and Sonic Visualiser. Please let me know if I can support you in the maintenance and maybe modernization of the vamp-plugin-sdk. I would like to contribute.

cannam commented 6 months ago

The design of the original SDK certainly leaves some big performance holes.

That was intentional at the time, in that I wanted the simplest concept for the plugin writer who maybe wasn't very familiar with C++. Building structures and returning them by value seemed easy to understand in comparison to alternatives that involved references to internal data.

The perceived typical use for the original SDK was for someone who was not necessarily a self-identified C++ programmer to adapt into C++ a method perhaps originally written in something like MATLAB, that involved very significant amounts of processing per block of audio and returned only small numbers of features afterwards. The canonical example would be a beat tracker plugin. So for example the SDK has this extra support for doing frequency-domain conversion on the host side and that is partly for performance reasons and partly just to make it easier on the plugin side, but it doesn't really care about the overhead of allocating a few vectors or a small map per block. (Of course, this rationale is a little bit undermined by the example plugins found in the SDK itself, which include some for which the copying actually is a significant overhead.)

Also on a slightly fuzzier level, I really didn't want to insist that plugin authors do feature-extraction in a RT-safe way, and it seemed potentially helpful to make clear that there was nothing real-time about the SDK at all, right from the beginning.

I'm totally happy to see more than one SDK, so long as they interoperate. It would be great to be able to direct developers to one SDK or the other depending on their particular requirements, and I don't see why the "official" Vamp organisation/site shouldn't do that. I'm also thinking to try writing a plugin using the RT SDK myself to see how I get on.

It would be good for a host to be able to distinguish RT-safe from non-RT-safe plugins - I can't remember whether we talked about that @lukasberbuer? It's the sort of thing that could go in the RDF description that many plugins have, although the RDF stuff is confusing enough to deal with that I think very few plugin authors actually do anything with it themselves. It would be particularly interesting in the case where an update to a plugin switches it from one SDK to the other - I can well imagine reworking a few more data-intensive plugins into RT ones.

lukasberbuer commented 5 months ago

Hi Chris (@cannam),

the RT use-case is definitely a very special one and I don't expect that too many people need this. My main focus wasn't about processing of audio data but noise & vibration data (usually of machines) also in the ultrasonic frequency range. The algorithms are quite similar and the Vamp plugin ecosystem was quite a good fit. We at @vallen-systems process ultrasonic data with sampling rates of up to 10 MHz in real time and want to provide users an interface to embed their own feature extraction. Instead of inventing our own interface, we went with your standard trying to avoid this:

image ― https://xkcd.com/927/

Regarding interoperability: We talked about the issue, that the current rtvamp host does not support all plugins which might generate more outputs per process call. I couldn't find a good way to extend the current API without sacrificing performance or introducing heap allocations for the target use case (OneSamplePerStep and hasFixedBinCount == true). Maybe you have a idea or recommendation?

Btw, thank you for inviting me to the @vamp-plugins org. I was quite busy the last weeks but looking forward to work with you on Vamp plugins ecosystem 😊