search

Chowdhury-DSP / releases

Release pipeline for ChowDSP plugins
12 stars 4 forks source link

Collection repository #1

Open jatinchowdhury18 opened 2 years ago

jatinchowdhury18 commented 2 years ago

Cloned from jatinchowdhury18/AnalogTapeModel#239:

@Bluezen wrote:

Hi, this is not specific to this plugin but didn't know where to ask.

Some plugin developers who have multiple plugins put them together into a collection. This makes it easier to package all the plugins as a bundle / one package. So there could be one repository called something like "Chow Plugins", which contains all of your plugins as submodules. As an example, take a look at x42-plugins: https://github.com/x42/x42-plugins

Is there something like this planned?

@jatinchowdhury18 wrote:

Having a "collection" repo is not really something I had thought about, but I definitely see the benefits that it could have from a "packaging" standpoint.

Currently, there is a "releases" repo, which I use for making nightly builds of each plugin. The repository only contains a GitHub Actions pipeline, build scripts for each plugin, and a little bit of metadata for each plugin... I wonder if that process could be simplified by using submodules instead. Then from there maybe we could add extra scripts for building/packaging multiple plugins at a time.

Anyway, this is a cool idea... I'll have to think about it a little more, and I'm definitely open to suggestions for how this sort of thing could be structured. I'd also love suggestions for renaming the "releases" repo, since it may now be used for more than just running nightly builds. For now, I'll also move this issue into that repo.

Bleuzen commented 2 years ago

I just started testing around if this is easy / possible at all: https://github.com/Bleuzen/Chowdhury-Plugins Sure I was missing something (never used cmake before)... with the simple setup with all Plugins as submodules there is the problem that submodules of the plugins get cloned multiple times and build target conflicts. Would be nice if all plugins could use just one shared JUCE, chowdsp_utils, ... submodule. Do you think this is possible with your plugins or do they need all their own JUCE, chowdsp_utils, ... (submodule) versions?

I'd also love suggestions for renaming the "releases"

I think "releases" is not that bad. But another suggestion: rename it to "Chowdhury-Plugins" since this sounds like a plugin collection of you and basically it contains all your plugins so would make sense.

Also, an idea for the releases: Every time there is a new Plugin update (or multiple ones in a short time/same day) update the submodule(s) in this repo and create a new release/tag, named just after the date ("v2022.01.17"), since plugins have their own individual versions.

jatinchowdhury18 commented 2 years ago

Ooh, thanks for taking a look at this!

Yeah, the submodule thing is kind of tricky, since some of the plugins are on older versions of JUCE, that may not be immediately compatible with the plugin code. For using the repository in a build pipeline, I bet it might be okay, since we could only clone the submodules for the plugin we want to build in that run of the pipeline. Another idea could be to use a different package manager like CPM, perhaps in tandem with git submodules.

That name suggestion sounds good as well, although I think I would prefer /ChowDSP-Plugins or /Chowdhury-DSP-Plugin (although maybe that's a bit too long).

Updating submodules in the repo should be pretty straightforward. I'm not sure how easy it is to automate creating GitHub releases, but I imagine that should be pretty do-able as well.

I should have some time on Thursday, so I can take a closer look then.

Thanks, Jatin

jatinchowdhury18 commented 2 years ago

Okay, so I was able to do some testing in the v2 branch, with just the BYOD plugin for now. Basically, an approach that can work is to clone the repo without submodules, then use the bash scripts within each plugin's directory to build that plugin (including fetching the relevant submodules). This does mean that some submodules will be re-cloned multiple times (if you're building multiple plugins), but I think that's okay because 1) most of the submodules (except for JUCE) are pretty small in size, and 2) it makes it less likely to have version discrepancies between two instances of the same submodule.

So from here, I think the process will look like this:

Bleuzen commented 2 years ago

Basically, an approach that can work is to clone the repo without submodules, then use the bash scripts within each plugin's directory to build that plugin (including fetching the relevant submodules).

Some linux distributions don't allow internet access during build time. However when building all plugins anyway cloning with all submodules in the first / downloading step should be fine.

This does mean that some submodules will be re-cloned multiple times (if you're building multiple plugins), but I think that's okay because 1) most of the submodules (except for JUCE) are pretty small in size, and 2) it makes it less likely to have version discrepancies between two instances of the same submodule.

This is also okay. Just asked about it in case it could be optimized. Maybe one day, but I don't think this is a dealbreaker.

So I had a look at your v2 branch and see you currently build using bash scripts. Would recommend to stay with a standardized (and the current cmake) build system. This is what most linux build tools support ootb, have their settings for preconfigured, can automatically handle and doesn't force the dependency on bash. Would be nice to have all the plugins integrated into one cmake project and only set the build targets if not all plugins should be built.

Something like, in the meta repos base dir:

cmake -DCMAKE_BUILD_TYPE=Release
make

to build all targets/plugins. And:

cmake -DCMAKE_BUILD_TYPE=Release
make CHOWTapeModel_VST3 CHOWTapeModel_LV2 ChowMatrix_VST3 ChowMatrix_LV2

to build only CHOWTapeModel and ChowMatrix without the standalone.

Do you think this could be possible this way, using only cmake / no bash?

I tried that in my repo linked above. Used cmakes add_subdirectory for that but this resulted in build target conflicts, since multiple plugins have the same submodules/targets with same name. Maybe renaming the plugins submodules / buildtargets for dependencies to be prefix with the plugins name could work (while ugly) or maybe there is another cmake option doing something similar as add_subdirectory but does handle the plugins separately, only exposing their top level build targets and not care about their sub-targets names.

jatinchowdhury18 commented 2 years ago

Cool, yeah I think we've got a good path forward starting from here.

I think I would like to keep the bash script interface, since I think it integrates better with GitHub Actions, and makes it easier to generate installers for Windows/Mac, but having a CMake interface would certainly be nicer for building locally, and probably for most packaging systems as well.

I'll have to do some testing (might not be able to until later this week), but I think the top-level CMakeLists.txt should look something like this:

cmake_minimum_required(VERSION 3.15)
project(ChowDSP-Plugins VERSION 2.0.0)

# This should fix the build target conflicts
set(ALLOW_DUPLICATE_CUSTOM_TARGETS TRUE)

# This plugin will be configured by default.
option(BUILD_CHOWTAPE "Build the ChowTape plugin" ON)
if(BUILD_CHOWTAPE)
    message(STATUS "Configuring build for ChowTape")
    add_subdirectory(plugins/ChowTape)
endif()

# This plugin hasn't been released yet, so it will NOT be configured by default.
option(BUILD_BYOD "Build the BYOD plugin" OFF)
if(BUILD_BYOD)
    message(STATUS "Configuring build for BYOD")
    add_subdirectory(plugins/BYOD)
endif()

Does that seem reasonable?

Bleuzen commented 2 years ago

I pushed your suggested CMakeLists.txt changes to my repo now and tried building that. It seems to work when only building one plugin at a time but with multiple ones there are still conflicts, here for example when I try to configure build of CHOWTAPE and CHOWCENTAUR: https://pastebin.com/raw/1tteZwCX

jatinchowdhury18 commented 2 years ago

Ah yeah, it looks like the JUCE targets are conflicting... I wonder if calling add_subdirectory() with EXCLUDE_FROM_ALL would help? Just looking at the docs at the moment.

By the way, sorry I've been pretty swamped this week, but I will definitely have some time to dig into this more fully on the weekend. Thanks!

Bleuzen commented 2 years ago

I wonder if calling add_subdirectory() with EXCLUDE_FROM_ALL would help?

Unless I didn't use it correctly, doesn't seem to help :/ Pushed to my repo so you can review. Output: https://pastebin.com/raw/Gtpq0RxN

By the way, sorry I've been pretty swamped this week, but I will definitely have some time to dig into this more fully on the weekend.

No problem. Thank you :)

jatinchowdhury18 commented 2 years ago

Hey! So I was able to spend some time on this today. It looks like what we've been looking for in the CMake world is not add_subdirectory(), but ExternalProject_Add(). I've been able to get a couple of plugins to build using this approach on the v2 branch. Here's the CMakeLists.txt. I think this is the right approach, and the majority of work from here will be in figuring out how to pass the relevant arguments through the CMake config.

That said, with CMake external projects, it might be a bit redundant to be using Git submodules, since the external projects can just as easily download the source using the branch/commit specified in the metadata file. Anyway, just something to think about...

Bleuzen commented 2 years ago

It looks like what we've been looking for in the CMake world is not add_subdirectory(), but ExternalProject_Add().

Hi, yeah this looks good, will test it later.

I think this is the right approach, and the majority of work from here will be in figuring out how to pass the relevant arguments through the CMake config.

Yes, building only plugins or only standalone should be possible.

That said, with CMake external projects, it might be a bit redundant to be using Git submodules, since the external projects can just as easily download the source using the branch/commit specified in the metadata file.

Remember: Some build systems don't allow internet access during build time. Not sure how they handle this ExternalProject_Add in other cases. Just a thought. Having cmake downloading the plugins at build time might break things. So if it has no downsides I think keeping git submodules could still make sense.

Bleuzen commented 2 years ago

Hm tested this with only building chow tape

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_CHOWMATRIX=OFF -DBUILD_CHOWKICK=OFF -DBUILD_CHOWCENTAUR=OFF -DBUILD_CHOWTAPE=ON
make

and it currently fails while trying to install JUCE.

[...]
Saving preset 21/22... done!
[ 99%] Built target CHOWTapeModel_LV2
[100%] Linking CXX shared module CHOWTapeModel_artefacts/VST3/CHOWTapeModel.vst3/Contents/x86_64-linux/CHOWTapeModel.so
[100%] Built target CHOWTapeModel_VST3
[ 87%] Performing install step for 'ChowTape_Package'
Consolidate compiler generated dependencies of target CHOWTapeModel_LV2_lv2_ttl_generator
Consolidate compiler generated dependencies of target RTNeural
Consolidate compiler generated dependencies of target BinaryData
[  1%] Built target CHOWTapeModel_LV2_lv2_ttl_generator
[ 33%] Built target BinaryData
[ 34%] Built target RTNeural
Consolidate compiler generated dependencies of target juce_plugin_modules
[ 48%] Built target juce_plugin_modules
Consolidate compiler generated dependencies of target CHOWTapeModel
[ 73%] Built target CHOWTapeModel
Consolidate compiler generated dependencies of target CHOWTapeModel_Standalone
Consolidate compiler generated dependencies of target CHOWTapeModel_VST3
Consolidate compiler generated dependencies of target CHOWTapeModel_LV2
[ 95%] Built target CHOWTapeModel_Standalone
[ 97%] Built target CHOWTapeModel_LV2
[100%] Built target CHOWTapeModel_VST3
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/include/JUCE-6.1.2/modules/juce_analytics
CMake Error at modules/juce/modules/cmake_install.cmake:46 (file):
  file INSTALL cannot make directory
  "/usr/local/include/JUCE-6.1.2/modules/juce_analytics": No such file or
  directory.
Call Stack (most recent call first):
  modules/juce/cmake_install.cmake:47 (include)
  modules/cmake_install.cmake:47 (include)
  cmake_install.cmake:47 (include)

make[3]: *** [Makefile:100: install] Fehler 1
make[2]: *** [CMakeFiles/ChowTape_Package.dir/build.make:102: ChowTape_Package-prefix/src/ChowTape_Package-stamp/ChowTape_Package-install] Fehler 2
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/ChowTape_Package.dir/all] Fehler 2
make: *** [Makefile:91: all] Fehler 2

Still have to disable this install step I guess.

Bleuzen commented 2 years ago

I think I found a way around that by using STEP_TARGETS. See my updated CMakeLists.txt

This works now, can build all plugins or only selected ones. Nice :) One option is still missing: build all (default), plugins only, standalone only

jatinchowdhury18 commented 2 years ago

Nice! Yeah, STEP_TARGETS looks like the right thing here.

For only building only standalone targets, I was able to start on this CMake config today. Haven't been able to test it much, plus it's a lot more repetitive that I'd like, but maybe it's a start:

option(BUILD_STANDALONE_TARGETS_ONLY "Build only standalone targets" OFF)

option(BUILD_BYOD "Build the BYOD plugin" ON)
if(BUILD_BYOD)
    message(STATUS "Configuring build for BYOD")
    set(BYOD_DIR "${CMAKE_SOURCE_DIR}/plugins/BYOD/BYOD")

    if(BUILD_STANDALONE_TARGETS_ONLY)
        ExternalProject_Add(BYOD_Standalone
            SOURCE_DIR "${BYOD_DIR}"
            CMAKE_ARGS "-DBUILD_RELEASE=ON"
            BUILD_COMMAND ${CMAKE_COMMAND} --build . --config Release --target BYOD_Standalone
            STEP_TARGETS build
        )
    else()
        ExternalProject_Add(BYOD_All
            SOURCE_DIR "${BYOD_DIR}"
            CMAKE_ARGS "-DBUILD_RELEASE=ON"
            BUILD_COMMAND ${CMAKE_COMMAND} --build . --config Release --target BYOD_All
            STEP_TARGETS build
        )
    endif()
endif()
Bleuzen commented 2 years ago

I just made test builds to compare: building all targets (default) vs. building only LV2+VST3.

Maybe it is not even worth it, it makes a much smaller difference in build time than I thought it would. Building all targets with the Standalone and deleting the Standalone after build to only get the plugins seems a viable option here since it only barely makes a difference compared to building only the plugins in the first place.

Can still keep this in the back of our head and think about it later for optimization, but I wouldn't see this as a dealbreaker and think it would be okay to not have this option for the initial release.

So I think it is fine now from my side. When you are ready too you can merge it with your github workflows stuff and make a first release.

jatinchowdhury18 commented 2 years ago

Yeah, that's a good point: most of the plugin code is built as a static library, which is then shared between the different plugin format wrappers, so adding/subtracting a plugin format won't affect the built time too much.

I think the next steps from here are:

I should have some time this weekend to hopefully get through the first three steps.

jatinchowdhury18 commented 2 years ago

Alright! It looks like the nightly build script is now updating the submodules in this repo automatically! I'll try to get a few more tests in over the next couple of days, but it looks like the basic workflow is in place.

Bleuzen commented 2 years ago

I tried building that with flatpak and it fails at cloning this repo. End of the log:

Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/BYOD/modules/chowdsp_juce_dsp'...
Submodule path 'modules/chowdsp_juce_dsp': checked out '0e8f18b4be33bdf38e07dd537a22e85d764ec4a4'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/BYOD/modules/sst-cpputils'...
Submodule path 'modules/sst-cpputils': checked out '916c8cdf516ec13a11145b8efa6b7ee98146bdca'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser'...
Submodule path 'ChowPhaser': checked out 'f738a7f400891233f4772c26f14403c940e17744'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser/modules/JUCE'...
Submodule path 'modules/JUCE': checked out 'f00a420bc348d79c4688b33b2b905b8ca0f25a3a'
Cloning into '/home/elias/_/build/flathub/.flatpak-builder/build/ChowDSP-Plugins-5/ChowPhaser/modules/foleys_gui_magic'...
fatal: git upload-pack: not our ref 9cab1a439d86cbf7942981b24ab59c60a55b7dcd
fatal: remote error: upload-pack: not our ref 9cab1a439d86cbf7942981b24ab59c60a55b7dcd
fatal: Fetched in submodule path 'modules/foleys_gui_magic', but it did not contain 9cab1a439d86cbf7942981b24ab59c60a55b7dcd. Direct fetching of that commit failed.
fatal: 
Error: module ChowDSP-Plugins: Der Kindprozess wurde mit Status 128 beendet

Edit: Created a new issue for that here: https://github.com/jatinchowdhury18/ChowPhaser/issues/18