Making standalone plugins usable before they're released to the plugin repo

[joemcgill]

Feature Description

Once #656 is complete, the intention was that new modules would be initially released as standalone plugins, and not bundled with the main Performance Lab plugin. This creates a situation where new plugin modules can not easily be tested or used prior to them being available on the plugin repo. Given the current long wait times for new plugins to be approved on the repo, this would create a significant problem for efficiently developing and testing out new features and no clear way of testing the feature prior to an initial release.

Currently, a new module can be activated from the Performance Lab settings screen before it has been removed as a standalone plugin, so a new module can be tested by checking out a feature branch of this repo that includes the new module and activating it in the settings screen.

Initial ideas to consider

As originally discussed in https://github.com/WordPress/performance/pull/904#issuecomment-1858540201, there are a few ways to address this issue:

1. Allow new modules to initially be included as experimental features that can be activated in the Performance Lab plugin prior to being released as a standalone plugin. This would require maintaining the migration logic developed in #656 in the plugin indefinitely, as it will be needed as new plugins are released.
2. Update the release process for this repo so that we can publish zip files of standalone modules here, before they are available on the plugin repo. This is probably worth looking into regardless, so we can test pre-release zips prior to publishing a new release of a standalone plugin. However, this is still less convenient than being able to test updates from a single repo.

Currently effected modules in development

• 556

• 805

• 886

• 897

• 904

[felixarntz]

I think I would prefer option 1, as it allows automatic updates easily (simply via Performance Lab). It's also not a lot of additional work required since most of the necessary infrastructure is implemented already.

Any code could start as a module, and once it's approved as standalone plugin, it can be deployed that way. Users would then see the migration notice as needed, and 1-2 releases later the module would be removed from PL.

The main difference that doing this long-term would potentially require is to adjust the standalone plugin build process slightly:

• Right now all plugins get built and deployed from their module code within the modules directory.
• Going forward, all plugins were originally going to be built and deployed from their code that's already in plugin shape (correct headers etc.), coming from another directory (e.g. plugins).
• If we keep the approach of migrating modules to plugins regularly, it's probably best to support both, i.e. the build process should be able to build and deploy standalone plugins from either of the two locations.
  • It should be relatively straightforward, since building modules into plugins is the most complex piece and that one is already implemented.
  • That said, we would still make a few adjustments, such as scanning both modules and plugins directories for the standalone plugins we have and depending on where the plugin code comes from use either the "module to plugin deployment" or the "plugin deployment" logic.

[joemcgill]

I put a bit of time into investigating what we would need to do to publish zips of standalone plugins to the releases page of this repo and if we did want to do this, we would need to update a few parts of our current workflow automation.

Currently, publishing a new release triggers both the main Deploy to WordPress.org workflow as well as the Deploy standalone plugins to WordPress.org workflow.

Both of these workflows would need to be altered so that we could publish a standalone plugin release (or pre-release) that would automate the creation of a new release tag without triggering additional deployment workflows. We could probably use tag filtering (step.if) to accomplish this, but we may want to rethink the whole automation flow so that the published release on the repo is the final result of the release workflow, rather than the trigger to initiate the workflow. We could also consider automating the creation of standalone plugin specific branches, from which each plugin could be tagged and deployed.

All this to say, I do think that option 1 will be less effort initially, though I would like for us to have a better long-term way of offering downloads of standalone plugins—including pre-release versions—for testing purposes outside of the official plugins repo. I do like the idea of moving standalone plugins to a separate folder that is organized as plugin folders rather than needing and think that if we make that change, then keeping a separation between what is a "plugin" and what is a "module" would simplify things a bit.

[felixarntz]

How about we start with the following workflow for modules that graduate into standalone plugins:

• New modules are added to the modules folder, the same way as today.
• Once ready for an MVP version, use npm run build-plugins (after temporarily adding the module to plugins.json, without actually committing) to get the plugin version of the module, then submit to .org for review.
• Regardless of .org approval, the module launches in the next PL release.
• Once the plugin is approved on .org:
  • Update the perflab_get_standalone_plugins_constants() function to include the module/plugin.
  • Amend plugins.json to include the module/plugin.
  • Update the perflab_get_standalone_plugins() function to include the plugin.
  • As part of the next Performance Lab release, the first version of the new standalone plugin will be deployed automatically. Users that have the module active will then receive a notice that they need to migrate.
• After 1-2 Performance Lab release cycles (i.e. typically 1-2 months):
  • Move the module's folder from within modules to the plugins folder (that would be a new top-level folder in the repository).
  • Update the module header to become a plugin header.
  • Remove references to the module in plugins.json and in the perflab_get_standalone_plugins_constants() function.
  • As part of the next Performance Lab release, the module will no longer be part of the plugin.

This would pretty much be the workflow.

To make that happen, only a few infrastructure changes would need to be implemented:

• Create plugins folder.
• Update plugins.json structure to allow managing both modules that are also deployed as plugins (i.e. from modules folder), as well as plugins that are purely standalone plugins (i.e. from plugins folder).
• Update the php-test-standalone-plugins.yml and deploy-standalone-plugins.yml commands to also cover plugins from the plugins folder. That should be quite straightforward, since, if anything, it involves less work, since those are already plugins, i.e. they don't need to be "built" from modules into plugins.

Once all of this infrastructure exists, we would document the workflow in the Performance Team Handbook, similar to the other documentation that already exists on writing a module. A few points in that existing documentation should probably be updated then as well.

LMKWYT.

[joemcgill]

TLGTM! 😜

For now I think we should separate discussion about moving standalone plugins to a separate built plugins directory from the problem this issue is trying to solve. For now, manually building a plugin before it's released seems like a fine workaround to get the review process started. We could even upload those zips manually to the GH release page if we wanted to make it easier for folks to test pre-releases before they're on the .org repo.

Until a module is available on the .org repo, keeping it available as an option to enable seems sensible. Once it's available, we can migrate it to whatever the current process is for managing the standalone plugins. Let's start a new issue for reorganizing published modules within the codebase in the meantime.

[felixarntz]

Happy to break this apart, but I think the separate built plugins directory is a given anyway. We can't have those continue to live in the modules directory when they shouldn't be modules in PL anymore.

I agree though that the infrastructural change related to having both modules and plugins directories should be implemented regardless of what we decide for this issue here. Even if we end up deciding that we won't start new standalone plugins as a module first, that'll be needed. In that case, we would probably start a new standalone plugin in the plugins directory right away, but then there would need to be another approach if we wanted to make it available before .org approval.

[westonruter]

Happy to break this apart, but I think the separate built plugins directory is a given anyway. We can't have those continue to live in the modules directory when they shouldn't be modules in PL anymore.

Sorry if this was already discussed...

What if they actually are kept in modules and all of the standalone plugins continue to get shipped as part of the PL plugin as modules? This would make migration to standalone plugins much easier, as a site owner could switch over to the standalone plugin at any time. There would be no back-compat breakage. In this way

• A site could decide to just install the one PL plugin as a mega plugin (like Jetpack) to activate all of the modules. This is also how it would be installed for development.
• Or they could install PL plugin, then start installing standalone plugins, and eventually uninstall the PL plugin with all desired module functionality remaining intact.
• Or they could install the standalone plugins without ever installing the PL plugin first.

As soon as someone installs a standalone plugin for a PL module, the module code would then short-circuit so that only one copy of the functionality is active at a time. The PL module could get deactivated (in the plugin DB option) when the standalone plugin is activated, ensuring that when a site owner deactivates the module it doesn't result in the module's behavior (from the PL plugin) from continuing to run unexpectedly. This would be the essence of the migration logic.

This would also solve the problem of the plugin review queue taking so long to approve new plugins. The long wait would not hold up immediate testing of the new module bundled with the PL plugin: it's just that the module wouldn't be able to be tested standalone until the standalone plugin is published on the directory.

[felixarntz]

@westonruter Having the features as modules and plugins was ruled out during the initial project definition as an option. See the conversation starting in https://github.com/WordPress/performance/issues/618#issuecomment-1432785852 for context.

[felixarntz]

@joemcgill Given that we won't use modules anymore, we may want to close this issue. Or at the very least rename it to maybe use "features" instead of "modules".

Personally, I'm not sure there's realistically a need for this in terms of making those features available at scale, as the plugin reviews have become much faster again, so I don't envision us being blocked from that anymore.

What is worth thinking about as you were mentioning the other day is how to make pre-release versions of features easier to test, but that's probably mostly a DX issue to improve for the contributors to the project rather than at scale (as in the 100k+ Performance Lab end users). It may fit better into a new dedicated issue that explains that goal.

[joemcgill]

Thanks @felixarntz. I would like to keep this issue open until we have a documented process for testing standalone plugins in development prior to being released on the .org repo. I've renamed the issue to no longer refer to "modules" in order to be more clear. There are two main reasons this is still needed:

1. During initial development, you need to be able to test the plugin installed on a local environment. This used to be done by activating the module, but that is no longer possible.
2. Once the plugin is released to the .org repo, we need to be able to do pre-release tests of new versions of standalone plugins prior to pushing a release to the .org repo, including making sure that there are no conflicts between standalone plugins when installed together (should be caught by our test suite, but still worth making an explicit part of our QA process).

As I've thought more about this, the easiest way for us to solve all of these issues is to make sure all of our standalone plugins are loaded automatically as part of the .wp-env.json config, like this:

{
  "plugins": [
    ".",
    "./plugins/auto-sizes/auto-sizes.php",
    "./plugins/speculation-rules/load.php"
  ]
}

That way the standalone plugins are auto-installed locally without additional workarounds.

Somewhat related, I think it would be nice to update our release workflows so that each plugin gets included individually in our releases page, with zip files attached for each standalone plugin release, along with changelogs, etc. This could also allow us to generate pre-releases if needed.

[felixarntz]

@joemcgill I like your wp-env proposal, that would be a very straightforward start that would already improve DX a lot.

[joemcgill]

👍🏻 PR incoming for that shortly.

[felixarntz]

Removing this from the 3.0.0 milestone as it is not really relevant for end users of the plugin and thus doesn't have to be solved by that release.

[westonruter]

@joemcgill I think this is done now, right?

[joemcgill]

Yes, I think we can close this. The standalone plugin folders are automatically loaded into the wp-env container since #1028. We can also build production versions locally using npm run build-plugins since #1033.