Closed baconpaul closed 8 months ago
I have updated the transaction above to fix a big reported by a user found when running some modules on systems which hadn't also installed the VST.
Thank you for the submission, Paul.
@AndrewBelt Sanity check on the repo size here: after submodule clone the repo size on disk is 500 MB. Original repo size is about 12 MB. Any concerns?
Since this won't possibly be able to cross-build, I'd say don't add it to repos/
. I'll check it out and build manually when I have a bare-metal build system set up later.
Sounds good.
One issue found, compilation fails on Windows: https://github.com/surge-synthesizer/surge-rack/issues/218
Once everything passes my tests on all platforms, I will move it to "Needs build update" without integrating it in the library repo.
Why can’t it cross build out of curiosity?
I think the cross build process checks out the source everytime from scratch. The repo is just too big for that. Andrew will build it manually.
Ok - most of the submodule aren’t used by the tack build. the size mostky comes from vst3sdk. I could branch surge to not include that (that branch would never build the vst of course) and make the submodule a branch
It’s a bit of a pain but not impossible. Would that help or is the approach you outline fine for you?
Also: azure builds binary assets which worknfine automatically Linux win and Mac. Would it be easier to just point at those assets I build instead?
Happy to do whatever is easiest for you that puts this in the plugin manager!
And will apply @cschol changes to let it build in win x-compile as well as win tomorrow
Thanks
I have applied @cschol cross compile case changes to surge and surge-rack. I have confirmed it builds win mac and lin still, on windows mac and linux images. I haven't confirmed the cross compile but did apply the patch offered.
Thanks!
I took a look at the repo size issue. The repo size comes from a the surge submodule which itself contains a couple of large collections of things I don’t need in rack (mostly: the surge vst UI assets which are in the surge repo and the vst3sdk sub-module, which amount to about 70% of the size). And then some things I do (the wavetables and patches).
If repo size is a constraint for you requiring a manual process, though, like I said, I could have a submodule of surge-rack be a surge branch which lacked the assets (annoyingish) and submodule (easy). Bit of a pain to keep in sync for me but nothing one or two smart scripts wouldn’t automate-ish away.
But is the concern size or checkout time? I build surge and surge-rack on clean empty hosts for every push to master. I took a look and submodule checkout time averages about 30 seconds on mac and linux and about 45 on windows from github to the azure open source vms (windows is also proportionally slower to build; a build takes 50 seconds on mac; 90 seconds on linux and about 150seconds on windows). So the checkout time is proportionally big but not that big in absolute. But vst3sdk is a big part of that checkout time, so branching that and the vstgui dependency out would speed things up a lot (surge itself when it builds takes about 22 seconds to resolve its submodules so a surge without vst3sdk should submodule update in more like 10 rather than 30 seconds) and shrink the repo.
Let me know if you want me to proceed with any changes.
Thanks!
No changes needed, I'll just manually build whenever I get around to it.
Plugin now passes all of my tests on all platforms and is ready for building. I will not integrate it in the library repo, but move it to "Needs build update" state.
To help you keeping track of manual builds we could add a Github tag for those. The tag would show up in the Project Board.
Ok thanks folks!
we could add a Github tag for those
No need, I can just see what's left after running an automated buid.
Hi all! Two quick things
1: i am without a laptop for about 2.5 weeks starting Saturday so if you don’t get to this before then let me also as @tank-trax to the thread here so he can check it out in the plugin manager. Zero pressure or rush of course!
2: it bums me out that you have to do manual work for a variety of reasons including: you have better things to do, and I hate being an exception which breaks an automated process. When I’m back I will do the branch to shrink the repo which I want for another reason anyway. Is there a repo size I should aim for that would let you integrate this like a “normal” module?
Thanks so much, and congrats on shipping 1.0 again. I look forward to surge-rack being part of it!!
Hello
I have a new point release
This version fixes the problem that triggers an assert in rack 1.2, so I would very much like to have it be released before Rack 1.2 ships. (That assert causes problems in rack 1.1 but means there's a clean crash with these modules in 1.2). It also fixes some other reported bugs and adds a noise module.
I also shrunk the repo some, in case that makes you want to add it to your auto process rather than your manual process. It's still pretty big - we have lots of versioned binaries and haven't done a gc - but the on disk image is 100mb or so smaller.
Thank you as always for your efforts.
Are the submodule changes committed?
$ git submodule sync
Synchronizing submodule url for 'surge'
$ git submodule update --init --recursive
error: Server does not allow request for unadvertised object 10287bc37bbb34324b8097369e2ec4eb73a51d65
Fetched in submodule path 'surge/vst3sdk', but it did not contain 10287bc37bbb34324b8097369e2ec4eb73a51d65. Direct fetching of that commit failed.
Failed to recurse into submodule path 'surge'
yes you need to either do a clean checkout (which sorry I thought you did or I would have mentioned this) or do
git submodule sync --recursive
git submodule update --init --recursive
there are changes to submodules of submodules and if you don't recursive sync it will not work.
Ah, sync --recursive
. Now it works. Thanks.
Cross-compile issue on Windows: https://github.com/surge-synthesizer/surge-rack/issues/244
Yup I will fix.
Like I said over there: I build every commit all 3 platforms. Would love to know how to run in your environment or, if you prefer, am happy to help you set up library to build in the ci environment we use.
Pointed you to my build script in the issue above.
I noticed the same. Good catch and thanks for reporting it @cschol
OK thanks for the find. I've updated with the downcase library
Hey Folks
I got a report of a memory error in the chorus and was able to fix it. Since you haven't updated the build yet, could I please use a pointer one commit ahead?
Thank you very much, as always!
These builds take a lot of my time so I'm not able to do them frequently. @baconpaul Can we treat these the same way as proprietary builds? Email contact@vcvrack.com if so.
Will do!
I've made some changes to my cross-build Docker container that allows this repo to be built if added to repos/
as a submodule. Next update, simply go through here and we'll re-integrate.
Fantastic news thank you so much! And greatly appreciate the work you do - it's a hard world out here in open source music land being a developer some days, so just want to make sure to say thanks!
Hi all.
Had a bit of activity on surge-rack. I updated it to the 1.6.3 engine which adds some expanded features for wavetables and also added a skinning mechanism. As discussed above, if you could add it to repos and move it along, or if there's any problem, let me know so I can address it, that would be great.
I confirmed it builds mac, linux and windows all three. Appreciate the help. Thanks!
Hi!
1.beta1.2 had an error with a nan suppression in a filter. I got it fixed. I try not to do releases too often, but this one is a bit annoying, so hope you don't mind.
Hi
I'm really sorry to keep revving this, but there's been a bit of an apparent uptick in usage; and that's exposed some fundamental bugs in the beta. Turns out I was mis-handling sample rate resets with some of the internal surge state and in a few cases that would lead to crashes.
So I'd like to do another revision
Sorry for the three releases in quick succession; each fixes a real crashing-style bug users have found as they push the modules more.
Hi!
We are releasing Surge 1.7.0 this week and have upgrades to our rack modules to match. With this release we decided to remove the beta label and have surge-rack versions follow rack parent versions so new version number here.
Thank you very much
@baconpaul Can you doublecheck that SHA? It does not exist after a pull on the repo.
Oh I'm so sorry I didn't push it to origin. My mistake.
https://github.com/surge-synthesizer/surge-rack/commit/33db4615a5922b174dd1fe1aca7893ea9d441b90
It's there now. Apologies!
Hi! No testing like the field. We released 1.7.1 this weekend and along with it, realized we missed some features from our rack plugins. If we could update again at your convenience? Thanks!
Hi!
Have a new point revision with a few fixes and feature changes as people used 1.7.1.
Versioning now so the first three digits match the surge release. Thanks as always
@AndrewBelt Did this get missed in the last update?
Hi!
Sorry, another small bug fix, this time with a streaming handler. Thank you as always.
Why is this plugin package so large? (58MB) Do we need the build/
directory? Does this plugin even make use of any of those files?
The build directory contains the build of surge (which is linked at compile time into the dylib) and a copy of the associated resources so they can be packaged coherently into the dist target.
It is created at build time so when we build the dist it has the necessary resources. You can see the Make target for build/surge-data which copies the subset of the surge content needed.
The plugin makes use of all these files. They are the wavetables and patches which the wavetable oscillator uses and the patch player plays.
Oh and also: only build/surge-data is included in dist, not all of build (although build/surge-data is the biggest chunk of data).
Okay, as long as it's necessary it's fine. Total bandwidth has been about 2TB which is only like $10, but of course more updates means more bandwidth. If it gets into the 20TB range, it'd be best to work on the filesize through lossy compression of samples or offering most of the data as an optional external download.
Ahh right bandwidth cost.
So yeah I could make the resources a separate release on our GitHub and have them be loaded first time round with a GH pull. Does the rack SDK have a portable way for me to address and download an SSL addressible zip from a URL other than library I could use? Happy to modify it that way.
Also if not, and if bandwidth cost is going up just drop me a note and I'm happy to subsidize it for these plugs. Like, if you just want me to venmo you a couple of hundred bucks or something I would happily do that. I understand the free service you are providing is useful to all of us and I'm happy to chip in to keep it going.
Alternately: Many of the modules will work without this. I could modify the code so that the two impacted modules (the wt osc and the patch player) look in the same place as a complete surge VI installation and then tell the users when they load those plugins "Surge WTOSC and Patch Player require you to also install the Surge Virtual Instrument from (url)". An easy code change.
I purposefully didn't do it that way in case people run surge VI versions different than surge rack versions. But doable. Although there was something super unpleasing about a set of rack plugins which didn't work unless you installed (some other software) first.
Any advice welcomed.
More specifically: is 'network.hpp' / 'requestDownload' a stable public api point I can use?
If so for 1.7.2.* later this year I can modify the rack plugins to lazy load this content from our GitHub and make our pipelines do releases there if that would help you. Let me know answer to both. Thanks.
Hi @AndrewBelt
just a quick ping here. Planning our fall dev work in the surge space and wonder if you could answer these questions about 'late loads'. Happy to restructure so the resources so they are late downloaded for our 1.8 family releases later this year / early next (which would greatly reduce the size of the bundle here) just if I know a couple of things
if the answer to those is 'yes' and 'yes' i can figure it out from there.
Appreciated.
I said "If it gets into the 20TB range", which is 10x the current bandwidth.
OK cool I'll leave it as is then. Let me know if it becomes a problem. There are a couple of alternate strategies as outlined above.
do you object to plugins loading resources themselves from remote URLs and putting them in the user data folder?
Yeah, that's kind of bad. Especially without the user's consent.
Is the network.hpp/requestDownload api a public stable one?
Yes, it's stable.
do you object to plugins loading resources themselves from remote URLs and putting them in the user data folder?
Yeah, that's kind of bad. Especially without the user's consent.
Consent would of course be important!
My rough idea was: Ship the plugins wit 5 wavetables and 5 patches; and then have the 2 impacted modules detect you are in that diminished state and show a UI element like 'get extra content' which prompts for consent then downloads the resources from GitHub.
But I don't need to write that extra code so I won't yet! But if it becomes a problem I am happy to.
Hi.
I would love to get the SurgeRack beta into the v1 library builds. Per instructions, I'm making a new issue with the data needed to start the process.
SurgeRack takes the digital synth, Surge and makes it into a collection of rack modules.
I have quite a lot of work to do on the modules still, but at this point they all work, most have non-generic faceplates, I have done some rudimentary profiling, and I would like to make them available as a beta for testing.
Here is the release point for the beta1 release.
I have checked the following
One thing worth mentioning: Surge ships with >1300 presets and >300 wavetables, all released under GPL3. These modules allow users to use those presets and wavetables in a variety of ways, as such, the Makefile constructs a directory with this relatively large set of assets for the plugins. The makefile rules I used to do this work on all platforms, but you should expect a large zip file to result from the make dist. On my mac, the 1.beta1.0 zip file was about 40MB.
The plugin.json links to a manual which is incomplete. I'll work on that over the next few days.
I look forward to any feedback you may have.
Thank you for your efforts!