[FEATURE] GuitarML's SmartPedal feature.

[38github]

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like GuitarML's SmartPedal feature so that it is possible to load json files and not be limited to just Tubescreamer and Blues Jr.

Additional information Thank you so much for your work! Where can I donate some caffeine money?

[GuitarML]

@jatinchowdhury18 This would be really great actually, there's alot of functionality in BYOD that would take a significant amount of effort for me to implement in my plugins (samplerate conversion, oversampling, preset manager, wet/dry mix, etc.). My new default model going forward is the LSTM with hidden size 40, and input size either 1 or 2 (conditioned models). Might be better to be flexible with model sizes and input parameters though. Anyway if this interests you let me know and we can do some planning to ensure the models I create remain compatible.

[jatinchowdhury18]

@GuitarML Yeah, that would be super cool! Right now the GuitarML module in BYOD is using the LSTM with hidden size of 40, which is compatible with some of your earlier plugins, but I can definitely update that so that the module is compatible with either layer size, as well as with conditioned models. If there's any models you'd like to be "built in" to the BYOD module, please attach them to this issue, and I can add them in!

As far as thinking about future compatibility with other network architectures, that might be a bit more difficult. Because of the module is using RTNeural's compile-time API, we'd have to decide ahead of time which network architectures will be supported. If you already have an idea of which architectures you're planning to work with in the future, we could try to get those setup, but it would be difficult to test if we don't already have some models using that architecture to experiment with. Additionally, I wouldn't want you to feel restricted to the architectures that we've planned for when developing new models for your plugins.

Supporting future models that might have more parameters wouldn't be as difficult, I'd just have to work out a way to have multiple "knobs" in the module editor if there's more parameters in model.

[GuitarML]

@jatinchowdhury18 That makes sense, then let's go ahead and settle on the LSTM 40, with at least the capability for no control parameters or 1 control parameter, but the ability to handle more inputs would be great. Here's the conditioned Blues Jr. and TS9 models, and a new one from a Mesa Rectifier 25 watt, on the "modern" channel.

With your permission I'd also like to do a youtube video about it, when it's done!

models_for_byod.zip

[jatinchowdhury18]

@GuitarML Sweet! I'll work on getting the new models implemented today. I should also ask what was the sample rate that these networks were trained at? I wonder if that's something we might want to encode in the json file?

Yeah, doing a YouTube video would be very cool :)

[GuitarML]

Those are at 44.1kHz, yeah that's probably a good idea to add an extra field for samplerate at some point. The training code could assign it automatically when it reads the wav files.

[jatinchowdhury18]

Cool, so I've gone ahead and implemented the LSTM40 models (both conditioned and non-conditioned) in the plugin, on this branch: https://github.com/Chowdhury-DSP/BYOD/compare/guitarml-update

At the moment I've been using the Mesa model for testing, so I haven't added that one to the built-in models just yet. Everything seems to be working, but it would probably good to get in a little ore testing as well. In particular, it would be nice if someone has a non-conditioned model to test with, since all the models I've been trying to use so far have been conditioned.

A few other questions for @GuitarML:

• We need some way to tell folks that only the LSTM40 models are compatible with the BYOD module. What would be the right text to use for that? (I'm thinking something like: "This module is compatible with trained models from Guitar ML plugins X, Y, and Z")
• For now I'm using the "sample_rate" json key, from the "model_data" part of the JSON config to detect the model's training sample rate, with a fallback of 44.1 kHz for models that don't have that field. Will that work for you in how you're planning to add the sample rate to the exported model? I can change it pretty easily on my end, if you'd rather do it a different way.
• Would you like to do anything different with the colours currently in BYOD's GuitarML module?

[38github]

I think it is great to see you two collaborating on this!

[GuitarML]

@jatinchowdhury18 Awesome, I'll test it out soon, if you want some non conditioned models you can get them here (except the ones labelled "Knob"): https://guitarml.com/tonelibrary/tonelib-pro.html

Right now only the Proteus plugin uses LSTM size 40, so you could say it's compatible with any Proteus models / models trained with the Proteus Capture Utility. There's probably enough DIY'ers that know what's going on to mention models trained with Automated-GuitarAmpModelling LSTM size 40 will be compatible, but I'll let you decide how much info you want to put and how to word it.

Yes, the sample_rate key is good, I'll probably only update my code if I deviate from the 44.1kHz, so that's great to use 44.1kHz as the default. The colors for the GuitarML module are good with me.

Can't wait to test it out! There's so much stuff in BYOD, I really want to find some time to explore routing the audio in unusual ways through all the effects you have in there.

[GuitarML]

@jatinchowdhury18 Tested it out, works well for both conditioned and snapshot models, and with multiple GuitarML modules. For the conditioned models, are you also applying the -18 to +18 gain?

I'm not sure how much work this would be, but is it possible to have the model name saved in the dropdown? Might make it easier when returning to saved sessions to remember what you loaded. I guess you'd need to keep the "Custom" in addition to the loaded model if the user wants to pick a new custom. Does that make sense?

[jatinchowdhury18]

@GuitarML

Sweet, great to hear that both types of models are working. I'm going to try testing with some random JSON files as well to make sure that doesn't crash the plugin :).

For the conditioned models, I was not applying the input gain, and was instead scaling the "gain" parameter into a 0-1 parameter which was then being input to the neural net. I think I'm going to switch that up so that when using a snapshot model, the user sees the "Gain" parameter like we see now, but when using a conditioned model, the user sees a 0-100% parameter. My questions then would be

1. Is the value of the conditioned parameter expected to be in the range [0, 1]?
2. What should we name the conditioned parameter? We could do something simple like "Parameter" or "Condition", or maybe "Drive" since that seems to be the most common thing to condition the model on? What could be even cooler is of the training script could allow the user to name the parameter that the model is being conditioned on, but I'm not sure how easy that would be.

For having the model name in the drop-down menu, would this be having the combo box show the custom model name that it's selected (and restore this name when re-loading the plugin state)? Or have the model name continue appearing in the menu so the user can immediately access it in all future sessions? The first would be pretty simple... the second would be a bit more difficult, plus I imagine it could be a little bit fragile, for example what happens to that entry in the menu if the user moves or renames the model file on their computer?

---

EDIT: I've updated the branch to cover most of the things that we've talked about here, in case you want to check it out. By the way, I have noticed that the MesaRectifier model is significantly louder than most of the other models. I wonder if there's some way we could do a bit of volume normalization either in the module of somewhere in the training process.

[GuitarML]

@jatinchowdhury18 I tried out the new build and that's exactly what I was thinking. Thanks for linking my website and tone library in the info! Yes that mesa model in particular is very loud, right now in the training I normalize both the input and output to 1, which seems to do well for most models, but that Mesa amp on the modern channel with the gain at 10 is especially hot. If you think the volume difference it too much I'll attach my Blackstar HT40 model to use instead.

I think in Proteus I check whether the file exists or not before loading, which if it got moved or renamed it would just do nothing. I should probably have some kind of user notification. For BYOD I think what you did here is good.

Edit: Somewhat related, in using your cmake build system, will these be mac M1 compatible if compiled on a non-M1 system? I think I'm building a universal binary because the mac builds are double the windows/linux size, but getting reports that my plugins "run slow" on M1's as if being emulated with Rosetta. I don't have an M1 on hand to confirm at the moment. I guess the fact they run at all means it's working, but I'm curious if there's something I can do to make it work better on M1.

[jatinchowdhury18]

@GuitarML great! I'll work on getting that branch merged in. I'll keep thinking about how to do volume normalization... I think that's something that can be done while loading the model.

For CMake, I've found that this set of arguments seems to work for compiling for both Mac CPU architectures. I think this argument is the most important: -D"CMAKE_OSX_ARCHITECTURES=arm64;x86_64".

[GuitarML]

Nice, so that will show up in the nightly build? Thanks for implementing this! I'll start mentioning it as an option for guitarml models, there's so much in BYOD to try out, I don't know how you have time to code it all.

For volume normalization, I guess you could run a pre-made sample through it, get the average volume level and apply a constant gain based on that.

[38github]

I am impressed! The BYOD implementation uses less than half of the CPU usage compared to Proteus. If you also implement SmartPedal I will donate to both GuitarML and Chowdhury-DSP. I am also spreading the word about both your work to people I jam with online.

[jatinchowdhury18]

Nice, so that will show up in the nightly build? Thanks for implementing this! I'll start mentioning it as an option for guitarml models, there's so much in BYOD to try out, I don't know how you have time to code it all.

Yes, the updates we've been talking about here are all now in the Nightly Builds!

For volume normalization, I guess you could run a pre-made sample through it, get the average volume level and apply a constant gain based on that.

Yeah, I tried doing something like this yesterday, but was getting inconsistent results... I'll come back to it at some point.

I am impressed! The BYOD implementation uses less than half of the CPU usage compared to Proteus. If you also implement SmartPedal I will donate to both GuitarML and Chowdhury-DSP. I am also spreading the word about both your work to people I jam with online.

Thanks! Major credit to Keith for ow great these models sound :). The SmartPedal network architecture is a good bit different, so I'm not sure I'm going to attempt it at the moment, but I'll definitely give it a try at some point.

[GuitarML]

What black magic are you doing to get half the cpu usage? Maybe how I'm handling stereo.. anyway that doesn't surprise me, Jatin is a coding genius.

Edit: The cpu usage difference is probably from the different samplerate converters used

I have been curious if there would be any cpu improvement with a RTNeural Wavenet, but yeah that's a much bigger endeavor than adding Proteus models. If you do look more into wavenet I might be able to help, I've been able to get conditioned wavenet models for SmartPedal but the training process is not very user friendly yet. I was also playing around with the dilation sizes, you can use less layers if you dilate them at a larger rate, but accuracy suffers.

[jatinchowdhury18]

@GuitarML I'm actually not 100% sure what might be causing the CPU differences... As I understand it we're both using RTNeural with the XSIMD backend. The resampler change will definitely make some difference, but from my previous testing I've found that the resampling CPU usage is only 10-20% of the total process CPU usage. One difference might be that BYOD processes the input in mono instead of stereo by default, so if the Proteus plugin is processing in stereo by default, then that could be part of it. That said, BYOD also oversamples by 2x by default, so to get a fair comparison, that option should probably be brought down to 1x. Anyway, I'll try doing some side-by-side testing later today and see what shows up!

I've actually been working with another fellow to try to improve the performance of convolution layers in RTNeural (particularly for larger dilation rates), so I'd love to take another shot at doing the WaveNet-style models in RTNeural once we've got that worked out. Unfortunately, the resampling trick that helps RTNeural do sample-rate agnostic processing for recurrent networks can't be directly applied to the WaveNet architecture, so we'd probably still need to do some external resampling for those networks.

[GuitarML]

@jatinchowdhury18 That's a really interesting paper, if I'm reading those graphs correctly it looks like there could be about a 6x improvement from the WaveNetVA code to the TCN models from the paper. I'm looking specifically at the TCN2 because it's most similar to what I'm using in the current SmartPedal code. Lots of exciting things happening in this field!

I'm good with closing this feature request btw, successfully implemented.

[jatinchowdhury18]

@GuitarML we actually merged a new version of the Conv1D layer yesterday, in case you want to try it out! At the moment only the Eigen backend has the new improvements, but we'll get those changes over to the other backends in time. I may have some time this weekend to try using the new layer with your pedal network, so I can let you know how that goes.

Anyway, I guess we should probably move the conversation over to that repo, so I'll close this issue for now!