Automate builds with a cloud service

[kzantow]

I'm investigating automating builds using a cloud service -- Azure Pipelines apparently allows for both Windows and macOS for free for open source projects; I almost created a working pipeline for macOS, it would be nice to get 'nightly' builds at least for now, and have some way of automating release. Unless someone else has a better idea.

I started looking around after I noticed Mac Stadium allowed free macOS compute time for open source projects: https://www.macstadium.com/opensource

If anyone knows any other services offering free compute time for Windows + Visual Studio build tools and/or macOS + XCode build tools, please do post them here!

[asimilon]

I think Azure Pipelines is going to be the winner tbh, given it's free for OSS projects, additionally should also be possible to build Linux there too (Ubuntu images are available afaik). When getting to the point, it's possible to install InnoSetup using Chocolately to build the installers on Windows. I have created Packages based installer for macOS (see #98), but I'm not sure how easy will be to install packagesbuild, although I think that Packages is more or less a wrapper for the built in package building tools, so perhaps they should be migrated.

[kurasu]

What about doing a disk-image based installer instead for macOS? (with symlinks to application support and the plug-in directories with a background image instructing the user what to drag where)

That's what I did for the previous installers, and I could dig up the source files i used to do it if needed..

[baconpaul]

@kurasu that would be super handy. Long discussion on this in #85 also.

[kzantow]

FYI - I got all the macOS builds all automated. This is good news, since it validates that on a completely clean machine, all 3 builds work. I built them and just published them to my own GH for now so people could validate they work properly. These are also, of course, still each including the resources. Obviously, it would be ideal to publish them to some more official project site. I'm happy to share what I've done if someone else feels like setting up an Azure Pipeline for Open Source.

@baconpaul @kurasu ^

[esaruoho]

Can has vst2/3/AU esaruoho@gmail.com ? Also will you be working on PC deployment?

[baconpaul]

This is fantastic. Could we have it pump out nighties to some site? What do well run projects do in this regard?

I have no direct experience running public cloud ci pipelines so really appreciate you doing this!

[asimilon]

VST2 is not going to be possible, since we would need the SDK in the repo, and Steinberg have been actively going after repos that contain those files. Also I'm not sure about the legality of distributing the binaries. I have VST2 and VST3 developer licenses, but I don't know if that means I could distribute binaries that are not something my "company" is producing. I certainly don't want to just "chance it" and end up losing those licenses. License stuff I find a complete nightmare to know for sure the rights and wrongs of it all.

[baconpaul]

I completely agree that license risk is bad risk and we don’t want to take any.

I’ve been an au only person for a long time so am not familiar with these vst version issues. Appreciate the clarification.

[kzantow]

@asimilon you don't need to have the SDK in the repo, just available to the build machine (which is how I currently have the automated builds set up). I agree the VST 2 licensing situation is rubbish. from what I understand, anyone with a "signed developer agreement" prior to October 2018 should be able to continue to distribute VST 2 plugins. I'm convinced I did this before. I almost don't even care about supporting VST 2, it's just that many hosts still don't support VST 3, including Live.

[kzantow]

having a widespread API controlled by a company like this is just insanity. there really should be some standards body that comes up with open apis for this stuff

[asimilon]

you don't need to have the SDK in the repo, just available to the build machine

This precludes using Azure Pipelines then, since the VMs are ephemeral.

edit: perhaps it might be possible to do this with having it in a private repo? Otherwise it's tricky, all the Apple based "cloud" operations I looked into are quite expensive.

I understand, anyone with a "signed developer agreement" prior to October 2018 should be able to continue to distribute VST 2 plugins

that's correct, also to distribute VST3 binaries you need the separate VST3 agreement. Where I'm sketchy in knowledge (not being a lawyer and all) is do those agreements allow me (or anyone else with them) to distribute the binaries, given that they are not a product of the company with whom the agreement is.

I really hate the legalese in these things. Like what the hell is going on in Section 2 here. It's an agreement about using the Audio Units logo, but then starts going on about individual products having to pass a test, and links to a 404'd document!

having a widespread API controlled by a company like this is just insanity. there really should be some standards body that comes up with open apis for this stuff

Unfortunately that's just the way it panned out, you have the 3 de facto "standards" being proprietary APIs.

[kzantow]

This precludes using Azure Pipelines then, since the VMs are ephemeral.

No, the VMs can download things from various sources, e.g. a private git repo with a legally obtained VST2 SDK.

Edit: just saw your edit. haha. yes, as long as it's not publicly available.

[jarkkojs]

Proposal for synchronisation. I contribute the first CMake infrastructure without removing premake build files so that I don't break this. Then I'll open a ticket for migrating this to CMake. Once that is done we can safely remove premake.

[esaruoho]

@kzantow overall, how is this progressing?

[kzantow]

@esaruoho Almost done with an initial stab at a Mac/Win automated build script; just need to solve a sed issue on windows or rewrite some small regex replacement bit using Python (of course 2.x on Mac and 3.x on Windows... sigh but shouldn't be an issue). I've been busy, but might be able to get this done tonight.

[kzantow]

@esaruoho got 64-bit builds for osx and windows automated. the releases here are built on azure pipelines: https://kzantow.github.io/surge/ (please remember this is just a temporary spot for testing the automated builds...) any verification that the builds work properly would be great ;)

[esaruoho]

@kzantow can check in about 40min

[esaruoho]

@kzantow any chance you could also bake a 32-bit vst2 of this for macos?

[esaruoho]

confirmed, was able to get macVST2 loading with Renoise, and macAU running with Logic Pro X. those are the only apps i have access to, as of right now.

[esaruoho]

@kzantow the installer does not install macAU into /Library/Audio/Plug-Ins/Components or ~/Library/Audio/Plug-Ins/Components either. not sure what's going on here @kzantow

[kzantow]

I haven't mucked with the installer at all, just calling the install script that builds it.

@esaruoho speaking of installer... I think it needs to allow selectable binaries to install; I'll have a look at what it's doing and see what's going on with the AU. It's definitely building I can see from the logs.

[asimilon]

@kzantow the installer does not install macAU into /Library/Audio/Plug-Ins/Components or ~/Library/Audio/Plug-Ins/Components either. not sure what's going on here @kzantow

Works fine here.

[kzantow]

@asimilon @esaruoho @baconpaul I made some tweaks to the Mac installer, the code changes are #180 and I rebuilt the installers using the modified automated build scripts for testing here: https://kzantow.github.io/surge/ the main change is that it shows a selection for each plugin format now instead of having that hidden behind a 'configure' button

[baconpaul]

Nice feature thanks!

But I can’t actually test it since I’m traveling without my laptop until the after the new year, sorry!

[esaruoho]

copied from #115 which i closed - the suggestion was to also get 32-bit builds going for AudioUnit,VST2,VST3 on macOS and for Windows (VST2,VST3) too, if possible. @kzantow what's your analysis as to the possibility of this happening?

[esaruoho]

@kzantow i can confirm that the installer pre-packages VST2,VST3,AudioUnit 64-bit and installs them.

there have, however, been issues where people say that after running the installer, they don't have the data/resources files in ~/Library - that instead they are /Library - and that the "configuration.xml is missing" issue still occurs. what should we do about that?

[kzantow]

@esaruoho I'm happy to build 32-bit plugs, too, I just don't know the commands to run to do it. If someone could tell me, I'll add to the installer right away!

As for ~/Library the plugin is being installed globally so I made a change in the PR to also install the resources globally. It doesn't make sense to have a plugin global but then per-user resources, IMO.

[esaruoho]

@kzantow cool! let's try and figure this one out. i know @getdunne was doing something relating to this at some point in time.

let's see - here are excerpts from a previously closed ticket:

Use VS2017 Configuration Manager to add the 
Win32 platform, then build the 32-bit VST2.

https://github.com/kurasu/surge/files/2696528/surge-vst32proj.zip

I don't have time to go through the pull-request process, 
but here is my .vcxproj for the VST2 build and instructions 
for how to use it, assuming you have access to the necessary
files from the old VST2 SDK.

[esaruoho]

@kzantow the mentioned instructions on using the content are as follows:

You will need a copy of the VST 2.4 SDK, or at least the two folders from it that Steinberg used to include with the VST3 SDK.

Copy public.sdk/source/vst2.x folder from old VST2 SDK into vst3sdk/public.sdk/source aeffeditor.h audioeffect.cpp audioeffect.h audioeffectx.cpp audioeffectx.h vstplugmain.cpp

Copy pluginterfaces/vst2.x folder from old VST2 SDK into vst3sdk/pluginterfaces aeffect.h aeffect.h vstfxstore.h

Create the VS2017 solution using premake5, as described in the main surge README on GitHub.

Open Surge.sln in VST2017, choose "Add project", and add my surge-vst2.vcxproj.

Choose "x86" from the Visual Studio "solution platforms" drop-down menu, and build the VST2 project.

Open a 32-bit VST2 host and select surge/target/vst2_32/Debug/Surge-Debug.dll (or, if you do a Release build, surge/target/vst2_32/Release/Surge.dll).

[esaruoho]

@kzantow is this in any way useful? i don't really have proper access to windows so i'm not sure what else to do.

[esaruoho]

switching gears to macOS 32-bit builds:

The macOS 10.14 SDK no longer contains support for 
compiling 32-bit applications. If developers need to 
compile for i386, Xcode 9.4 or earlier is required. (39858111)

so in essence, to push out 32-bit stuff, one needs to have Xcode 9.4 commandline tools installed.

@kzantow is this like a big issue to do with those uhh computer instances or instance templates or something somethings you're using with your automated scripts?

[baconpaul]

I just don’t think 32 bit Mac is worth much effort but I think 32 bit audio unit is worth zero! I’m not sure what host would want it that would run the code

[asimilon]

I just don’t think 32 bit Mac is worth much effort but I think 32 bit audio unit is worth zero! I’m not sure what host would want it that would run the code

Some studios refuse to upgrade, because it all works and they don't see the need to fix what isn't broken. Having said that, I tend to agree about not bothering to support 32 bit under OSX. Which was the last version that still supported it?

[baconpaul]

Some studios refuse to upgrade, because it all works and they don't see the need to fix what isn't broken.

I totally get that. I just wonder which studio would not upgrade to software at least 7 years old and also download some open source synth from a github page into their production environment?

Logic 10 has supported 64 and 32 since outset and has been 64 only for a long time if I recall. This is really a logic 9 problem for au.

I don’t know the vst community well enough to offer comment!

[kzantow]

@esaruoho I don't think it's a huge issue, I just don't know what to change to make the builds work!

[esaruoho]

@kzantow well, what happens if you run xcode9 on the build script? 32bit success?-)

[kzantow]

@esaruoho Im running XCode 9.4.1. there needs to be another architecture specified for 32-bit. I just don't know what to do for windows, mostly

[baconpaul]

Hey @kzantow did you ever get surge to build with that free MSFT VM? I downloaded it and got visual studio but I immediately run into errors like "time.h" missing and stuff. I haven't used windows in forever so I am sure there is just some pre-req I need but if you can give me a hint and I figure it out, I'll update the readme.

[kzantow]

@baconpaul yeah, it's building for macOS and Windows. you can check the installers here: https://kzantow.github.io/surge there isn't much special you need to do, just run premake5 to get a fresh VS project, then msbuild it

[baconpaul]

right; when I fire up that vm, msbuild isn't in my path and it seems to be missing a whole bunch of headers. I think there's some "super rudimentary visual studio setup" i need to do which I haven't. Sort of the equivalent of "install Xcode command line tools" or "apt-get something".

Last time I compiled windows software was, what, one of the NTs. Maybe windows 7. I'm hopelessly out of date but would love a dev env so I can test.

[baconpaul]

like "can't find malloc.h" type errors even when I specify the build path completely. Hmm. I'll leave this be for now but it would be super to have directions from "virtual box on mac + that free windows vm" -> "built surge" eventually!

[kzantow]

ah yes. well, the Windows VM on Azure have VS 2017 installed and msbuild isn't on the path, but you can just run an MSBuild task in the Azure Pipeline. I just installed VS Community C++ on a local VM and opened the GUI to build it

[baconpaul]

I have literally no idea how to do that azure thing! But that's OK. I'll debug it some other time.

Over the last couple of days we've pushed quite a few bug fixes and cleanups. Any chance you can do a run of the build and stage new installer images at the kzantow/surge site?

[asimilon]

The VS projects need to be set up so that there are two "Configurations" for Win32 and x64, then on Azure just a matter of having two "Build Solutions" (this is using msbuild behind the scenes), with the appropriate "Configuration" and the "Platform" set to "Win32" or "x64" to correspond.

edit: sorry I misunderstood what this convo was about :) @kzantow probably worked that all out already

[baconpaul]

The convo is about a couple of things. Mostly about @kzantow automating builds which works great. But also a little about how I could use a free VM which @kzantow mentioned so I could do windows builds in virtual box, and that didn’t work for me. The equivalent of “run premake open Xcode go” didn’t work there. And I haven’t used windows for anything in too long to be useful, so that’s less important. And the final thing was me asking @kzantow if he could run his builds again so the installers are up to date since we fixed quite a few bugs in the last few days, including the about box having a build date and time which could help @esaruoho as he continues to help users who ask questions.

[kzantow]

@baconpaul sorry, I've been a bit busy the past few days. if you have an account on azure pipelines, I could probably give you access to run the builds yourself whenever you want. currently I haven't automated PR testing, although if we could get a build script merged in to the main repo, I think that would be fairly easy to do.

[baconpaul]

@kzantow please don’t apologize! This is volunteer work for everyone and I appreciate that you have made it build!

I know nothing about azure - I come from an ancient world before devops in the cloud was a sentence which parsed! I will try and look at your pr this morning

What needs to be merged to allow a regular build? @kurasu has a pull request to add a yml file which I didn’t understand but I think you might. If we merge that and yours are we starting to get there?

[baconpaul]

Oh also @kzantow apologies ahead of Time I pushed back on a VERSION file but the @jsakkine made an orthogonal and also compelling case for it so it may actually get swept. If I have time to look at prs today

[kurasu]

@kzantow @baconpaul Since I connected the repo with an azure pipelines account already, I think all you need to do is to open a pull request to master with a modified yml file and we should see if it builds

[baconpaul]

So we should seeep your pr and then merge and sweep @kzantow one and see what happens?