A corePress Plugin and Theme Repository

[BenOvermyer]

As we don't yet have much idea of what this will look like, I thought it'd be best to start an issue for discussion of the overarching vision.

When I first saw the conversation about this, I immediately thought of places like npm and packagist. So, I'm stealing liberally from those concepts.

I imagine there'd need to be an API underlying the whole thing. It would be in charge of the database and the file store. It would also handle authentication, probably token-based. The registry website would consume the API, as would ClassicPress, calmPress, and other such applications.

Concerns I see include:

• Integrity - how to make sure that malicious plugins/themes are automatically detected and rejected
• Compatibility - ensuring that the API contract is stable
• Operational Concerns - making sure the API is as lean as possible, to both keep performance high and to keep operational costs low
• Dependency Management - less of a problem for themes and plugins than for software packages, but should this registry be responsible for handling dependencies itself? Or continuing to leave that to end users?

[markkap]

lol we were writing at the same time.

I think dependency management should be left to the client, at the very least there might be different ways to handle dependency "errors" and I do not think it is wise to make the repository handle it.

For compatibility, best way is some standard wrapper code on the client side and maybe sample/extendible code for the API implementation on the repository side.

[nylen]

I think dependency management should be handled in the repository.

It is not easy, but if we can get this "right" then it will solve a persistent problem for the WP ecosystem.

[nylen]

dependency management should be handled in the repository

To clarify: not necessarily handled on the server itself, but at least there should be a standard way for clients using this repository to resolve and install plugins, and this should be the recommended and expected approach for all clients.

[nylen]

I'm aware of at least 3 ways to mirror the existing WP plugin and/or theme directories:

• https://wpackagist.org/
• https://github.com/wp-plugins
• https://github.com/markjaquith/WordPress-Plugin-Directory-Slurper and https://github.com/markjaquith/WordPress-Plugin-Directory-Slurper/issues/30

I quite like the idea of using composer to handle installation. There would undoubtedly be some challenges, but if we can pull it off then we get dependency management and a generally very well-tested solution for installing PHP dependencies "for free".

[nylen]

This issue is probably also a good place to leave links to existing solutions for WP dependency management.

I have not evaluated these, just listing solutions I'm aware of. Please feel free to add more and/or discuss.

• https://github.com/xwp/wp-plugin-dependencies
• https://github.com/afragen/wp-dependency-installer (and related https://github.com/afragen/github-updater)

[BenOvermyer]

I was unaware of WPackagist. Hmm. We could fork it, add in the ability to host plugins/themes as well as mirror, and then modify calmPress and ClassicPress to install themes/plugins via Composer.

The trick there is that Composer is really meant to be used as a CLI binary and not a web-triggered tool. That has security considerations. Calling system binaries from PHP code makes me feel ill.

[johnalarcon]

Another project. https://github.com/onigetoc/wpgit

[nylen]

I forgot a dependency management method as well, @invisnet has been working on this here: https://github.com/invisnet/ClassicPress/tree/v2

The trick there is that Composer is really meant to be used as a CLI binary and not a web-triggered tool. That has security considerations. Calling system binaries from PHP code makes me feel ill.

composer is still a great way to manage *Press sites: the production site is entirely read-only as far as WP itself is concerned, and plugins are added to the composer.json in the dev environment and added to the production environment via the deploy process.

Otherwise totally agree about the security concerns etc. I guess one of the reasons I am so interested in composer here is that it could be part of a standardized workflow which solves a lot of known problems using existing tools, but it probably shouldn't be the only option.

[markkap]

The trick there is that Composer is really meant to be used as a CLI binary and not a web-triggered tool.

Yes, which is why it is unlikely that you can just reuse it, but I do think that if we can "reuse" its requirements from repositories it will make making repositories that work both with composer and corePress much easier. Probably forking wpakagist will do 90% of the work.

[markkap]

@nylen I think that resolving dependencies in the repository will make them needlessly more complex without fully solving all needs.

Example, and agency manages 1000 sites. They like yoast but have a small plugin that uses yoast's filters to modify its behaviour. They would want to install their internal plugin and define that it depends on yoast.

Lets assume that the internal plugin contains a proprietary code, therefor it is hard to legally have it in an external repository. If we push into a direction in which it is the responsibility of the repository to resolve dependencies, they will have to make a repository in which they will host not only their own plugins, but also all their dependencies. This is probably something that can be done, but it is much more work since at the extreme case you need that the repository will be able to resolve more external dependencies.

I think it is much more natural to keep such logic at the client itself. Moving such logic to the repository will also require the client to send much more information which might bring privacy issues, and it will reduce flexibility.

[nylen]

I suppose that if we do have repository dependencies, they would all be required to live in the repository.

Otherwise you end up with a situation where "Some Cool Agency's Cool Feature" depends on "Some Cool Agency's Core Framework". The repository will let you download the "Cool Feature" plugin, but you can't actually use it without downloading something else too.

(Aside: there would be other ways to address this issue, like an automated check for each plugin that spins up the site, activates the plugin, visit its admin page (if any), and makes sure there are no fatals. I think this sort of testing would be a good idea for other reasons too.)

Undoubtedly having server-side dependencies would introduce more edge cases and issues (what do we do if the "Core Framework" plugin is removed?), and we haven't even started to think about versions yet.

Anyway I don't have strong opinions about any of this, and I think you've thought through the details more than I have.

[markkap]

I am just following composer here. AFAIK it do not expect repositories to be able to do anything more than give meta information about the library and where to get it.

But then, composer main use case seems to be working against github in which you do not have any binding between the various library, except for the fact that they are hosted on GIT and on a specific domain.

Maybe we can save some requests by the repository announcing dependencies it knows about the requested plugin which reside in the repository itself, but this IMO should come in latter stages. For MVP I think simpler is better.

On Sat, Nov 3, 2018 at 7:19 AM James Nylen notifications@github.com wrote:

I suppose that if we do have repository dependencies, they would all be required to live in the repository.

Otherwise you end up with a situation where "Some Cool Agency's Cool Feature" depends on "Some Cool Agency's Core Framework". The repository will let you download the "Cool Feature" plugin, but you can't actually use it without downloading something else too.

(Aside: there would be other ways to address this issue, like an automated check for each plugin that spins up the site, activates the plugin, visit its admin page (if any), and makes sure there are no fatals. I think this sort of testing would be a good idea for other reasons too.)

Undoubtedly having server-side dependencies would introduce more edge cases and issues (what do we do if the "Core Framework" plugin is removed?), and we haven't even started to think about versions yet.

Anyway I don't have strong opinions about any of this, and I think you've thought through the details more than I have.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/corePress/corePress/issues/1#issuecomment-435561635, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJnmkm3z_2nIxotS8kbPRgfz3qxgJ04ks5urSdPgaJpZM4YLqcx .

[BenOvermyer]

I agree. Let's explore what forking wpackagist and using it for corePress would look like.

[stefanos82]

@markkap as far as I know from examples that I have seen online, you can pull data from a GitHub repository. See this example if you like https://drupal.stackexchange.com/a/243585

But I agree with your last line: "For MVP I think simpler is better."

Indeed this line is a pure gold and the root of success for any project out there.

[nylen]

Sure, it doesn't hurt to think about future requirements, but definitely design and implement for the requirements you know you have today, and keep things simple.

[jessuppi]

Came here via a link via an email from Scott Bowler...

I don't know what my future views on ClassicPress, CalmPress, or corePress might be, but regardless I think there should be more effort in freeing plugins and themes from WordPress.org politics.

As everyone knows, the entire system is currently based on how much one man, Samuel Woods, likes you or doesn't like you. That is really not an open source community.

I was planning on suggesting here experimentation with Composer and saw @johnalarcon link in regard to the WPGIT project, never saw that before and very interesting... we had been considering playing with a similar plugin next year at LittleBizzy.

From my understanding wpackagist.org simply repacks plugins from WP.org (only). So not sure how forking that would help anything really... yes, it could mean a broader array of available plugins if various public projects are "imported" (converted?) but it also might lead to over-centralization... then again, Packagist is still centralized, but they are extremely liberal about indexing projects as long as there exists a proper composer.json file... could this project guarantee similar treatment?

To truly get a "free" headstart that @nylen talks about, it would require something like political neutrality re: extensions, so that any WordPress fork could take advantage, and to avoid emulating the fiefdoms that make developers roll their eyes every time a new fork pops up.

In other words, I think this topic/project is more important than any actual forks of WordPress, and could end up powering dozens of WordPress forks, creating a truly decentralized ecosystem...

[markkap]

In other words, I think this topic/project is more important than any actual forks of WordPress, and could end up powering dozens of WordPress forks, creating a truly decentralized ecosystem...

In the end those two things go hand in hand. Since wp.org refuses to host plugins which are "store fronts" (oh, unless otto likes you), and core refuses to integrate with anything that is not under WP full control, you just can not have a meaningful repository without a fork.

I think we started fast here and then totally stopped, or at least that is my impression from the silence going on here ;). I think that we need some plan how to move forward on this.

I assume we are all busy, so more like a plan with some incremental steps that can be followed, not some commitment to unrealistic schedules. From the calmPress POV, I think I will get into making a core plugins repository before year end (hopefully sooner) and then will ask for feedback to see if the scheme can be generalized, to fit both ClassicPress and be good enough for general plugin and theme repositories.

[BenOvermyer]

For my part, I got slammed by day job, side job, and other FOSS projects that I'm working on.

I'm evaluating which projects I have time for right now. I think corePress is really important, but I'm not sure if I'm going to be able to give it the attention it needs.

[stefanos82]

The reason I got silent for a while is because I'm working on improving my skills around PHP and git as a whole so I can become fully confident in order to take certain tasks and meet the deadlines whenever is plausible and possible.

[nylen]

Related discussion:

https://forums.classicpress.net/t/classicpress-plugin-directory/153

[nomadicjosh]

I will be posting this on ClassicPress' forum as well, but If something like wpackagist is something that is of interest, I think it can be accomplished by using Gitlab, Satis Gitlab, composer installers extender and a cronjob. With this combination, you can give plugin authors an account on your gitlab instance (public or set to private if commercial plugin) and have a cronjob update the Satis Gitlab instance (Plugin Repository) when plugins are published and/or updated. This will give users the ability to install and/or update plugins via composer and you could even implement this into something like wp-cli.

[nylen]

@parkerj I think it would be best to start with GitHub, with an eye towards allowing GitLab as an option.

Expecting all plugin developers to set up their own GitLab instance is too much, I think. We are already asking a lot by requiring that all development move to git. I'm fully in favor of that, but I don't think we should go much further as far as mandatory steps to release something.

[nomadicjosh]

@nylen That is completely understandable.

But just to clarify, developers would not be setting up their own Gitlab instance. You or whoever is over the corePress project would set up one Gitlab instance for corePress using the community edition of Gitlab (i.e. corepress.org), and all developers could have an account on that one Gitlab instance to develop their plugins. But again, l understand with wanting to stick with Github.

[nylen]

Ah, I see. That would certainly work. Still, I think it is worth keeping the number of services that need to be hosted to a minimum.

[jessuppi]

Some great comments, this is worth repeating:

The less centralized, the more opportunity (and support).

If this is done right, it can power dozens of WordPress forks and avoid centralized fiefdoms. Obviously at some level there would need to be maintenance/decisions, however.

[markkap]

satis is something I was intending to use for the calmPress' core plugins repository.

As for yes/no for centralized repository, the way I see it a centralized repository should serve as a complementary for the ability to add plugins from anywhere. I actually think that at the very basic level all you need to host is the url to the plugin and let the repository extract information from the readme file, minimizing the need of the plugin author to understand how the repository work.

IMHO a repository is going to be an iterative kind of work, something that you will not fully understand what it needs until the needs will present themselves from analyzing how users and authors interact with it, therefor starting simple is probably the way to go.

[nomadicjosh]

And you can definitely swap out my scenario with Gitlab+Satis and instead do a Github+Satis setup to power both the plugin and theme repository(ies).

[jessuppi]

We've just been banned from WordPress.org.

Whoever is following this thread, let us know how we can contribute our dozens of free plugins (more on the way) to your various WordPress forks. If we need a composer.json let us know.

https://github.com/littlebizzy