search

VCVRack / library

Database for the VCV Library
https://library.vcvrack.com/
375 stars 81 forks source link

SurgeRack #565

Closed baconpaul closed 6 months ago

baconpaul commented 5 years ago

Hi.

I would love to get the SurgeRack beta into the v1 library builds. Per instructions, I'm making a new issue with the data needed to start the process.

SurgeRack takes the digital synth, Surge and makes it into a collection of rack modules.

I have quite a lot of work to do on the modules still, but at this point they all work, most have non-generic faceplates, I have done some rudimentary profiling, and I would like to make them available as a beta for testing.

Here is the release point for the beta1 release.

I have checked the following

One thing worth mentioning: Surge ships with >1300 presets and >300 wavetables, all released under GPL3. These modules allow users to use those presets and wavetables in a variety of ways, as such, the Makefile constructs a directory with this relatively large set of assets for the plugins. The makefile rules I used to do this work on all platforms, but you should expect a large zip file to result from the make dist. On my mac, the 1.beta1.0 zip file was about 40MB.

The plugin.json links to a manual which is incomplete. I'll work on that over the next few days.

I look forward to any feedback you may have.

Thank you for your efforts!

baconpaul commented 5 years ago

I have updated the transaction above to fix a big reported by a user found when running some modules on systems which hadn't also installed the VST.

cschol commented 5 years ago

Thank you for the submission, Paul.

@AndrewBelt Sanity check on the repo size here: after submodule clone the repo size on disk is 500 MB. Original repo size is about 12 MB. Any concerns?

AndrewBelt commented 5 years ago

Since this won't possibly be able to cross-build, I'd say don't add it to repos/. I'll check it out and build manually when I have a bare-metal build system set up later.

cschol commented 5 years ago

Sounds good.

One issue found, compilation fails on Windows: https://github.com/surge-synthesizer/surge-rack/issues/218

Once everything passes my tests on all platforms, I will move it to "Needs build update" without integrating it in the library repo.

baconpaul commented 5 years ago

Why can’t it cross build out of curiosity?

cschol commented 5 years ago

I think the cross build process checks out the source everytime from scratch. The repo is just too big for that. Andrew will build it manually.

baconpaul commented 5 years ago

Ok - most of the submodule aren’t used by the tack build. the size mostky comes from vst3sdk. I could branch surge to not include that (that branch would never build the vst of course) and make the submodule a branch

It’s a bit of a pain but not impossible. Would that help or is the approach you outline fine for you?

Also: azure builds binary assets which worknfine automatically Linux win and Mac. Would it be easier to just point at those assets I build instead?

Happy to do whatever is easiest for you that puts this in the plugin manager!

And will apply @cschol changes to let it build in win x-compile as well as win tomorrow

Thanks

baconpaul commented 5 years ago

I have applied @cschol cross compile case changes to surge and surge-rack. I have confirmed it builds win mac and lin still, on windows mac and linux images. I haven't confirmed the cross compile but did apply the patch offered.

Thanks!

baconpaul commented 5 years ago

I took a look at the repo size issue. The repo size comes from a the surge submodule which itself contains a couple of large collections of things I don’t need in rack (mostly: the surge vst UI assets which are in the surge repo and the vst3sdk sub-module, which amount to about 70% of the size). And then some things I do (the wavetables and patches).

If repo size is a constraint for you requiring a manual process, though, like I said, I could have a submodule of surge-rack be a surge branch which lacked the assets (annoyingish) and submodule (easy). Bit of a pain to keep in sync for me but nothing one or two smart scripts wouldn’t automate-ish away.

But is the concern size or checkout time? I build surge and surge-rack on clean empty hosts for every push to master. I took a look and submodule checkout time averages about 30 seconds on mac and linux and about 45 on windows from github to the azure open source vms (windows is also proportionally slower to build; a build takes 50 seconds on mac; 90 seconds on linux and about 150seconds on windows). So the checkout time is proportionally big but not that big in absolute. But vst3sdk is a big part of that checkout time, so branching that and the vstgui dependency out would speed things up a lot (surge itself when it builds takes about 22 seconds to resolve its submodules so a surge without vst3sdk should submodule update in more like 10 rather than 30 seconds) and shrink the repo.

Let me know if you want me to proceed with any changes.

Thanks!

AndrewBelt commented 5 years ago

No changes needed, I'll just manually build whenever I get around to it.

cschol commented 5 years ago

Plugin now passes all of my tests on all platforms and is ready for building. I will not integrate it in the library repo, but move it to "Needs build update" state.

To help you keeping track of manual builds we could add a Github tag for those. The tag would show up in the Project Board.

baconpaul commented 5 years ago

Ok thanks folks!

AndrewBelt commented 5 years ago

we could add a Github tag for those

No need, I can just see what's left after running an automated buid.

baconpaul commented 5 years ago

Hi all! Two quick things

1: i am without a laptop for about 2.5 weeks starting Saturday so if you don’t get to this before then let me also as @tank-trax to the thread here so he can check it out in the plugin manager. Zero pressure or rush of course!

2: it bums me out that you have to do manual work for a variety of reasons including: you have better things to do, and I hate being an exception which breaks an automated process. When I’m back I will do the branch to shrink the repo which I want for another reason anyway. Is there a repo size I should aim for that would let you integrate this like a “normal” module?

Thanks so much, and congrats on shipping 1.0 again. I look forward to surge-rack being part of it!!

baconpaul commented 5 years ago

Hello

I have a new point release

This version fixes the problem that triggers an assert in rack 1.2, so I would very much like to have it be released before Rack 1.2 ships. (That assert causes problems in rack 1.1 but means there's a clean crash with these modules in 1.2). It also fixes some other reported bugs and adds a noise module.

I also shrunk the repo some, in case that makes you want to add it to your auto process rather than your manual process. It's still pretty big - we have lots of versioned binaries and haven't done a gc - but the on disk image is 100mb or so smaller.

Thank you as always for your efforts.

cschol commented 5 years ago

Are the submodule changes committed?

$ git submodule sync
Synchronizing submodule url for 'surge'
$ git submodule update --init --recursive
error: Server does not allow request for unadvertised object 10287bc37bbb34324b8097369e2ec4eb73a51d65
Fetched in submodule path 'surge/vst3sdk', but it did not contain 10287bc37bbb34324b8097369e2ec4eb73a51d65. Direct fetching of that commit failed.
Failed to recurse into submodule path 'surge'
baconpaul commented 5 years ago

yes you need to either do a clean checkout (which sorry I thought you did or I would have mentioned this) or do 

git submodule sync --recursive
git submodule update --init --recursive

there are changes to submodules of submodules and if you don't recursive sync it will not work.

cschol commented 5 years ago

Ah, sync --recursive. Now it works. Thanks.

cschol commented 5 years ago

Cross-compile issue on Windows: https://github.com/surge-synthesizer/surge-rack/issues/244

baconpaul commented 5 years ago

Yup I will fix.

Like I said over there: I build every commit all 3 platforms. Would love to know how to run in your environment or, if you prefer, am happy to help you set up library to build in the ci environment we use.

cschol commented 5 years ago

Pointed you to my build script in the issue above.

tank-trax commented 5 years ago

I noticed the same. Good catch and thanks for reporting it @cschol

baconpaul commented 5 years ago

OK thanks for the find. I've updated with the downcase library

baconpaul commented 5 years ago

Hey Folks

I got a report of a memory error in the chorus and was able to fix it. Since you haven't updated the build yet, could I please use a pointer one commit ahead?

Thank you very much, as always!

AndrewBelt commented 5 years ago

These builds take a lot of my time so I'm not able to do them frequently. @baconpaul Can we treat these the same way as proprietary builds? Email contact@vcvrack.com if so.

baconpaul commented 5 years ago

Will do!

AndrewBelt commented 4 years ago

I've made some changes to my cross-build Docker container that allows this repo to be built if added to repos/ as a submodule. Next update, simply go through here and we'll re-integrate.

baconpaul commented 4 years ago

Fantastic news thank you so much! And greatly appreciate the work you do - it's a hard world out here in open source music land being a developer some days, so just want to make sure to say thanks!

baconpaul commented 4 years ago

Hi all.

Had a bit of activity on surge-rack. I updated it to the 1.6.3 engine which adds some expanded features for wavetables and also added a skinning mechanism. As discussed above, if you could add it to repos and move it along, or if there's any problem, let me know so I can address it, that would be great.

I confirmed it builds mac, linux and windows all three. Appreciate the help. Thanks!

baconpaul commented 4 years ago

Hi!

1.beta1.2 had an error with a nan suppression in a filter. I got it fixed. I try not to do releases too often, but this one is a bit annoying, so hope you don't mind.

baconpaul commented 4 years ago

Hi

I'm really sorry to keep revving this, but there's been a bit of an apparent uptick in usage; and that's exposed some fundamental bugs in the beta. Turns out I was mis-handling sample rate resets with some of the internal surge state and in a few cases that would lead to crashes.

So I'd like to do another revision

Sorry for the three releases in quick succession; each fixes a real crashing-style bug users have found as they push the modules more.

baconpaul commented 4 years ago

Hi!

We are releasing Surge 1.7.0 this week and have upgrades to our rack modules to match. With this release we decided to remove the beta label and have surge-rack versions follow rack parent versions so new version number here.

Thank you very much

cschol commented 4 years ago

@baconpaul Can you doublecheck that SHA? It does not exist after a pull on the repo.

baconpaul commented 3 years ago

Oh I'm so sorry I didn't push it to origin. My mistake.

https://github.com/surge-synthesizer/surge-rack/commit/33db4615a5922b174dd1fe1aca7893ea9d441b90

It's there now. Apologies!

baconpaul commented 3 years ago

Hi! No testing like the field. We released 1.7.1 this weekend and along with it, realized we missed some features from our rack plugins. If we could update again at your convenience? Thanks!

baconpaul commented 3 years ago

Hi!

Have a new point revision with a few fixes and feature changes as people used 1.7.1.

Versioning now so the first three digits match the surge release. Thanks as always

cschol commented 3 years ago

@AndrewBelt Did this get missed in the last update?

baconpaul commented 3 years ago

Hi!

Sorry, another small bug fix, this time with a streaming handler. Thank you as always.

AndrewBelt commented 3 years ago

Why is this plugin package so large? (58MB) Do we need the build/ directory? Does this plugin even make use of any of those files?

baconpaul commented 3 years ago

The build directory contains the build of surge (which is linked at compile time into the dylib) and a copy of the associated resources so they can be packaged coherently into the dist target.

It is created at build time so when we build the dist it has the necessary resources. You can see the Make target for build/surge-data which copies the subset of the surge content needed.

The plugin makes use of all these files. They are the wavetables and patches which the wavetable oscillator uses and the patch player plays.

baconpaul commented 3 years ago

Oh and also: only build/surge-data is included in dist, not all of build (although build/surge-data is the biggest chunk of data).

AndrewBelt commented 3 years ago

Okay, as long as it's necessary it's fine. Total bandwidth has been about 2TB which is only like $10, but of course more updates means more bandwidth. If it gets into the 20TB range, it'd be best to work on the filesize through lossy compression of samples or offering most of the data as an optional external download.

baconpaul commented 3 years ago

Ahh right bandwidth cost.

So yeah I could make the resources a separate release on our GitHub and have them be loaded first time round with a GH pull. Does the rack SDK have a portable way for me to address and download an SSL addressible zip from a URL other than library I could use? Happy to modify it that way.

Also if not, and if bandwidth cost is going up just drop me a note and I'm happy to subsidize it for these plugs. Like, if you just want me to venmo you a couple of hundred bucks or something I would happily do that. I understand the free service you are providing is useful to all of us and I'm happy to chip in to keep it going.

baconpaul commented 3 years ago

Alternately: Many of the modules will work without this. I could modify the code so that the two impacted modules (the wt osc and the patch player) look in the same place as a complete surge VI installation and then tell the users when they load those plugins "Surge WTOSC and Patch Player require you to also install the Surge Virtual Instrument from (url)". An easy code change.

I purposefully didn't do it that way in case people run surge VI versions different than surge rack versions. But doable. Although there was something super unpleasing about a set of rack plugins which didn't work unless you installed (some other software) first.

Any advice welcomed.

baconpaul commented 3 years ago

More specifically: is 'network.hpp' / 'requestDownload' a stable public api point I can use?

If so for 1.7.2.* later this year I can modify the rack plugins to lazy load this content from our GitHub and make our pipelines do releases there if that would help you. Let me know answer to both. Thanks.

baconpaul commented 3 years ago

Hi @AndrewBelt

just a quick ping here. Planning our fall dev work in the surge space and wonder if you could answer these questions about 'late loads'. Happy to restructure so the resources so they are late downloaded for our 1.8 family releases later this year / early next (which would greatly reduce the size of the bundle here) just if I know a couple of things

  1. do you object to plugins loading resources themselves from remote URLs and putting them in the user data folder?
  2. Is the network.hpp/requestDownload api a public stable one?

if the answer to those is 'yes' and 'yes' i can figure it out from there.

Appreciated.

AndrewBelt commented 3 years ago

I said "If it gets into the 20TB range", which is 10x the current bandwidth.

baconpaul commented 3 years ago

OK cool I'll leave it as is then. Let me know if it becomes a problem. There are a couple of alternate strategies as outlined above.

AndrewBelt commented 3 years ago

do you object to plugins loading resources themselves from remote URLs and putting them in the user data folder?

Yeah, that's kind of bad. Especially without the user's consent.

Is the network.hpp/requestDownload api a public stable one?

Yes, it's stable.

baconpaul commented 3 years ago

do you object to plugins loading resources themselves from remote URLs and putting them in the user data folder?

Yeah, that's kind of bad. Especially without the user's consent.

Consent would of course be important!

My rough idea was: Ship the plugins wit 5 wavetables and 5 patches; and then have the 2 impacted modules detect you are in that diminished state and show a UI element like 'get extra content' which prompts for consent then downloads the resources from GitHub.

But I don't need to write that extra code so I won't yet! But if it becomes a problem I am happy to.