search

VCVRack / library

Database for the VCV Library
https://library.vcvrack.com/
385 stars 82 forks source link

Migration to v1 #553

Closed AndrewBelt closed 1 year ago

AndrewBelt commented 5 years ago

Time to start discussing the migration to Rack v1 and the changes required in this repo!

  1. When the v1 API is stabilized, we should make a new branch v1 in this repo. Yes, there's already a v1 branch, but it was foolishly named years ago, so we'll just remove it.
  2. Instead of writing/editing manifest files ourselves, we should only approve them from plugin developers. We should help them out if needed. JSON is not a skill that everyone has, and we have certain guidelines for some of the properties. Luckily, testing your plugin with Rack already checks most of the plugin.json like invalid JSON or some bad properties. When approved, we'll simply add their repo to repos/, run a script that copies repos/MyPlugin/plugin.json to manifests/MyPlugin.json, and commit.
  3. Later, after Rack has a command line flag to dump module screenshots, we should add them to this repo in screenshots/. Err, maybe not. An envelope calculation says that it'll be over 100MB if screenshots are at 200% scale (~100KB/screenshot * 1000 modules). This might still be okay though.
  4. Because library manifests will be identical to repo manifests, the following properties need to be rethought.
    • repoVersion: We can just read repos/MyPlugin/plugin.json version for this, no longer needed
    • latestVersion: This will just be version since it's in the plugin.json format.
    • productId: This will be handled via slugs by the server. It can be removed.
    • buildTimestamp: Not sure where we should put this. Ooo, I should just generate it on the server by reading the package file timestamp.
    • status: Can be computed by the server as well.
    • Okay, so all of these are handled. Fantastic.
  5. The server API will handle requests for both v1 and v0.6 plugins (at least for a while), via a new API path or query.

All other operations will probably be handled the same way. Can anyone think of anything I'm missing?

cschol commented 5 years ago

So the plugin developer submits a PR for their updated manifest, which we will merge if it's approved? Do I still update the repo to the correct SHA? It would actually be nice if that SHA was in the manifest and could be approved/used for update of repo that way. Some developers provide the SHA they want in the issue, but sometimes they don't and I just assume it's the HEAD.

AndrewBelt commented 5 years ago

PRs are too complicated. "Plugin threads" have worked well IMO. Plugin developers should announce an update as usual in their plugin saying something like "MyPlugin v1.0.0 is available at abc123" or "the HEAD of my master branch", and we can update the their submodule as usual. The only additional step is to review their plugin.json and move it to manifests/MyPlugin.json (with a script). We don't to edit the manifest ourselves anymore.

Should I go ahead and create a v1 branch here?

cschol commented 5 years ago

OK. That makes sense and I agree has worked well. I don't see any reason why not for the v1 branch.

AndrewBelt commented 5 years ago

Okay, added https://github.com/VCVRack/library/tree/v1. Run python3 scripts/collect_manifests.py to copy all plugin.json files to manifest files. Currently only Fundamental is updated (and its version is a lie, Fundamental v1.0.0 isn't released yet). I left all the repos intact because it doesn't harm anything. Since they don't have a plugin.json file, they won't be scanned with my build script. You can use the intact repos to update to v1 versions.

AndrewBelt commented 5 years ago

Solved the screenshot issue. https://community.vcvrack.com/t/rack-v1-development-blog/1149/466

AndrewBelt commented 5 years ago

Made announcement in https://community.vcvrack.com/t/adding-your-v1-plugins-to-vcv-plugin-manager/3231

cschol commented 5 years ago

Here are a few thoughts on the overall migration process:

Having 1 issue per plugin would allow progress tracking of the migration with GitHub Project Boards for a v1 migration project, which would be a lot easier than what I did last time by manually editing an issue to keep track of plugin migration status to v0.6. The Library team would drive the overall migration with the Project Board. The Repair team assists with PRs for individual modules that need it.

I am envisioning 3 states per plugin:

  1. To be migrated to v1 branch (all open source plugins currently in master)
  2. Migrated to v1 branch (has plugin.json and v1 compatible branch)
  3. Done (builds against Rack 1.0 API and runs without major defect, e.g. crash on load)

Example scenario: modular80 is in state 1 and has a v1 branch with a plugin.json available. The author notifies in the appropriate plugin issue. I will move it to state 2 by integrating it in the v1 branch (copy plugin.json, update to v1 compatible submodule branch). It builds successfully against the current Rack 1.0 API and loads. I move it to state 3. Now, a late breaking Rack API change is required (happened last time), which will move it back to state 2. The author fixes the code (or the Repair Team provides a PR to the author; it gets integrated) and we move back to 3.

What do you think?

MarcBoule commented 5 years ago

I would actually be ready to start my requests to add our plugins to the v1 library, but I was just waiting so as to avoid as much as possible oscillating between states 2 and 3 until API stability, to keep things simpler for you guys; but if ever you would like some more plugins to be added to v1 library for testing workflows, or whatever other reason, just let me know and I'll trigger my issues to get our things into v1 since I'm at state 3 now :-). Cheers!

AndrewBelt commented 5 years ago

The Library Team is only migrating open-source plugins. Correct?

Yes. I'll handle updates to freeware and commercial modules through email.

GitHub Project Boards

I've enabled Projects. Feel free to use this feature. Your 3 states look fine, can't think of any more states.

State 3 should also require that screenshots with ./Rack -p works and that the plugin can be displayed in the Module Browser without crashing.

cschol commented 5 years ago

Thank you!

Project board for v1 migration is available at: https://github.com/VCVRack/library/projects/1

If a plugin is available for v1 should we begin the migration now or hold off for a while until the API is more finalized?

AndrewBelt commented 5 years ago

If you're asking about migrating plugin source code itself, wait until after v1 API is stable. If you're asking about adding current "early" migrated v1 plugins to the v1 library branch, go ahead.

cschol commented 5 years ago

Yeah, I was referring to early migrated v1 plugins.

I'll keep the issues v0.6 vs v1 straight in the project boards. You only need to process issues in state "Plugin build update" in the v0.6 library maintenance project board, as usual. The automation will handle the state in that board.

A v1 update in an issue is not automated right now and after integration in the v1 branch, I will close the issue again. These will never move to the "Plugin build update" state in the v0.6 board.

AndrewBelt commented 5 years ago

Changed state names of projects to describe what each state needs.

cschol commented 5 years ago

As I am starting to integrate v1 plugins that have been updated, I will close the issue after moving to the Needs Build Update state. I assume there will be no official v1 builds until right before the launch date. This will clean up the state for v0.6 plugins on the other project board. Once we officially phase out v0.6 building the issue will stay open until you have rebuilt the plugin with the build system. Does that make sense?

AndrewBelt commented 5 years ago

Makes sense. No v0.6 builds will be made after Rack v1 release, and no v1 builds will be made before.

squinkylabs commented 5 years ago

For a plugin dev like myself, what needs to be done to get v1 plugins in the mangers? Do I just add to my existing (0.6) issue and say "hey, build me some v1 plugins from sha xyz"? Do I start a new issue for v1, and continue to use my old one for 0.6?

cschol commented 5 years ago

@squinkylabs Exactly. Use the same, existing issue for both (for right now).

AndrewBelt commented 5 years ago

The v1 plugin server API is ready. I'm just a couple hours away from calling the client/server stable and working on v1 builds.

Effective now, all plugins distributed through the VCV Plugin Manager must follow the VCV Plugin Ethics Guidelines. Existing plugins are grandfathered in.

I'm also considering renaming the VCV Plugin Manager to the "VCV Library". I might do it when Rack 1.0 releases, or I might do it when the new Plugin Manager releases (with module pages, screenshots, etc).

cschol commented 5 years ago

@AndrewBelt When you start to build plugins for v1, can you check your build process for plugins that build libsamplerate as a dependency, e.g. Fundamental?

I have trouble crosscompiling it on Linux for Windows. This is also an issue with some open-source plugins, which use libsamplerate, e.g. FrozenWasteland and I want to ensure that there are no issues in the plugin itself.

make dep fails when I execute it with the mingw32 compiler:

checking for C compiler default output file name... a.exe
checking for suffix of executables... .exe
checking whether we are cross compiling... configure: error: in `/home/cschol/src/library-v1/repos/Fundamental/dep/libsamplerate-0.1.9':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
Makefile:16: recipe for target 'dep/lib/libsamplerate.a' failed
make: *** [dep/lib/libsamplerate.a] Error 1

Does this work for you?

Another thing I noticed is that if a make dep is required, on Linux you can just execute make and it will do a make dep first automatically. This does not seem to work on Mac or Windows platforms and an explicit make dep would be required first.

AndrewBelt commented 5 years ago

I never learned how to use ./configure's cross-compiling ability, but that could be an option, or I could set up VMs to actually build on each OS rather than cross-compilers in Docker.

cschol commented 5 years ago

The libsamplerate build dependency is not new for the Fundamental plugin, right? Are you building the plugins on their respective platforms (e.g. on Windows) or cross-compiling for the plugin manager in Docker on Linux?

AndrewBelt commented 5 years ago

I custom-build a lot of plugins. Fundamental, Host, VCV Recorder are some that have too many system deps or need to run code during the build process.

cschol commented 5 years ago

I took a look at the crosscompilation. Adding the "--host" option and a reference to mingw32's ar to configure in the Fundamental Makefile will allow successful cross-compilation of Fundamental on Linux:

Relevant Makefile excerpt:

$(libsamplerate):
        cd dep && $(WGET) http://www.mega-nerd.com/SRC/libsamplerate-0.1.9.tar.gz
        cd dep && $(UNTAR) libsamplerate-0.1.9.tar.gz
        cd dep/libsamplerate-0.1.9 && AR=x86_64-w64-mingw32-ar $(CONFIGURE) --host=x86_64-w64-mingw32
        cd dep/libsamplerate-0.1.9/src && $(MAKE)
        cd dep/libsamplerate-0.1.9/src && $(MAKE) install

This would resolve a huge headache since more and more plugins are compiling libsamplerate like Fundamental does (e.g. FrozenWasteland).

This is just a proof of concept. The additional configure options/definitions should probably be moved to a "Rack-level" .mk file.

Mac platform has the same problem. If this looks reasonable I will take a look at that one, too.

cschol commented 5 years ago

Successful Mac crosscompile of Fundamental on Linux. Similar change:

 $(libsamplerate):
        cd dep && $(WGET) http://www.mega-nerd.com/SRC/libsamplerate-0.1.9.tar.gz
        cd dep && $(UNTAR) libsamplerate-0.1.9.tar.gz
-       cd dep/libsamplerate-0.1.9 && $(CONFIGURE)
+       cd dep/libsamplerate-0.1.9 && RANLIB=/home/cschol/src/osxcross/target/bin/x86_64-apple-darwin17-ranlib $(CONFIGURE) --host=x86_64-apple-darwin
        cd dep/libsamplerate-0.1.9/src && $(MAKE)
        cd dep/libsamplerate-0.1.9/src && $(MAKE) install
AndrewBelt commented 5 years ago

I'll look into --host in a bit.

I will write the build system today. The build process will claim the responsibility of copying repos/<plugin slug>/plugin.json to manifests/<plugin slug>.json, since it will use the manifests folder as its reference for what repos need to be built. In other words, if a plugin manifest for a certain version exists in manifests/, then the build exists in the VCV Plugin Manager server.

cschol commented 5 years ago

OK. So I won't run the collect manifest script anymore when integrating new versions, correct?

AndrewBelt commented 5 years ago

Correct, updating the submodule is what should trigger the build, which triggers the manifest collection.

cschol commented 5 years ago

Cool. DrumKit, SynthKit, and CharredDesert are the first plugins that just have the submodule updated (without copying the plugin.json to manifests). Maybe that will help you test your build scripts.

AndrewBelt commented 5 years ago

Switched the default branch of this repo to v1.

First builds are ready for most of the repos.

AndrewBelt commented 5 years ago

It is now no longer worth it to review v0.6 builds. My time is better spent running automated v1 builds.

cschol commented 5 years ago

Once a build has been produced for v1 do you want to move the project board card to "Up-to-date"? Some builds are already at 1.0.0 and I want to make sure to notify developers when to update the version number.

AndrewBelt commented 5 years ago

Could we automate it when I close the issue?

cschol commented 5 years ago

Yeah, absolutely. I just want to make sure we are at that stage. I'll update the board.

cschol commented 5 years ago

Updated. Stoermelder-P1 and FrozenWasteland do not appear in the Plugin Manager yet as available for v1.0.0. Are there any issues with those?

AndrewBelt commented 5 years ago

Stoermelder-P1 is built, but FrozenWasteland requires libsamplerate and I need to either figure out cross-building ./configure scripts or build them manually.

AndrewBelt commented 5 years ago

Turned off automation on the v0.6 project

cschol commented 5 years ago

Are you officially stopping builds for 0.6? CountModula is integrated and waiting for an update and HolonicSystemsFree just came in with a 0.6.x request. I can close those with reference to 1.0.0, but I want to be sure you are switching over entirely.

Maybe an announcement in the Community Dev section and Facebook Dev group that from now on only 1.0.0 plugins will be built would also remind folks to provide updated plugins.

AndrewBelt commented 5 years ago

I'll make the statement "I will likely no longer make v0.6 builds of plugins for the VCV Library." I might later, or might not, but you can close the issues if you like.

AndrewBelt commented 5 years ago

Since plugin.json is written by plugin authors now, we should look out for incorrect formats and broken links. It might be a good idea to write a python script (you or me) to check manifests. Rack does a fairly good job at refusing to load plugins with invalid manifests, which solves 90% of the problem, but we should still make a script that checks things like

cschol commented 5 years ago

Good idea. You could distribute the validation tool as part of the SDK and authors can check themselves in addition to a sanity check during integration. I'll put something together.

cschol commented 5 years ago

I have a Python script that validates

Question on Tags: does tag case matter, i.e. do tags need to accurately match the tag in plugin.cpp? For example: "Ring Modulator" vs "Ring modulator" (Befaco)

Also: what about aliases? "Poly" vs "Polyphonic"?

AndrewBelt commented 5 years ago

Aliased should be fine. It's case insensitive so POLY and poly and POLYPHONIC are fine.

cschol commented 5 years ago

Sounds good. I'll add support for the aliasing. There are quite a few issues I found. I'll get them reported to developers and finish the rest of the integration later today.

cschol commented 5 years ago

To adjust the validation script a little more, which manifest keys are required vs optional?

REQUIRED_TOP_LEVEL_KEYS = [
    "slug",
    "name",
    "version",
    "license",
    "author",
    "brand",
    "authorEmail",
    "authorUrl",
    "pluginUrl",
    "manualUrl",
    "sourceUrl",
    "modules"
]

REQUIRED_MODULE_KEYS = [
    "slug",
    "name",
    "description",
    "tags"
]

I would assume brand and authorEmail are optional. What about description in the module section?

The script now also validates the slug per the rules defined in Rack's code.

AndrewBelt commented 5 years ago

Only slug, name, and version are required to be valid Rack plugins. But we should strongly recommend that as many properties are filled out as possible.

AndrewBelt commented 5 years ago

@cschol Are you able to lock threads? Could be handy for permanently closing duplicate threads.

cschol commented 5 years ago

Ah, indeed I can. I'll use that. Thanks.

AndrewBelt commented 5 years ago

Is this the correct/best way to review a plugin?

cd repos/XXX
git fetch
git diff ABCDEF
git checkout ABCDEF
cd ../../
git add repos/XXX
git commit
cschol commented 5 years ago

I just pull the changes and use tig (command line git client to review). If there is an issue I just run git submodule update to get to the original commit.

Then I build against the 1.0.0 SDK for all three platforms (using a script) and run the manifest validator script.

Final check is a load in Rack 1.0.0 and module browser visual verification (no crash).

AndrewBelt commented 5 years ago

Great, thanks!