RedBean extras, community participation/plugins

[daviddeutsch]

So, I think it makes sense to make this into a ticket. On the mailing list, we already talked a little about this. To recap:

• While we are building RB4, a lot of the new and fancy stuff should and will happen in plugins - this keeps the core clean and concise
• Ideally, each plugin should be maintained mostly by one member of the community who takes responsibility for it
• It is also the responsibility of the maintainer to keep up the same level of testability and "documentation by test" noted in #308
• Having a separate repository is probably the way to go. What about redbean-extras? Although I would prefer redbean-sugar, but I might be pushing it with the coated beans idea
• One of the goals is that with redbean-sugar, we create a source for building a custom RB distribution

[daviddeutsch]

Submissions process (proposal)

1. (might skip this) Open a ticket to discuss your proposal, get approval from RB maintainers / RB community
2. Create Pull Request with your code (must follow code guidelines and include properly documented tests, see #308)
3. PR is tested by RB maintainers
4. Extra/Sugar is included in repository

Maintenance

• Since I'm a sucker for clean code, we will prefer clean code in the submission process, but I will probably also work on this after the PR is accepted (just a heads up, @gabordemooij can tell a tale of that :wink: )
• Keeping up with possible changes in the API would be handled through tickets where we discuss changes. In general, the tickets should be used to either submit proposals, or to have public discourse on how both sides - RB and Sugar maintainers - can help eachother out
• Keeping the code maintained in general would be the job of the proposal submitters and they should be ready to work with others in the community to fix bugs and add features
• Maintenance in general is handled via PRs - If a Sugar maintainer submits a fix or feature, that usually means it's a fast track merge. If somebody in the community submits a PR, the maintainer is in charge of approving it so it can be merged by the RB maintainers

[gabordemooij]

:+1: sounds good. Looks like we had to have this meta discussion before but now is a great time. Let's try this process and see how it works out.

[marcioAlmada]

When things get to this point, most of the time, I see people creating github organizations. So all plugins ecosystem would live inside that Redbean organization, side by side with Redbean project too.

Examples:

• https://github.com/laravel
• https://github.com/propelorm
• https://github.com/symfony

[daviddeutsch]

@marcioAlmada Yeah, but I think a single redbean-sugar repo would work best. That way, community collaborators can work closely together. Otherwise, you might end up with a multitude of repos that grow disconnected and where communication is unnecessarily hard.

Question in general: What will the name be? Is my idea with the "sugar" and "coating" stuff compelling or do we use something more generic?

[marcioAlmada]

@daviddeutsch I particularly prefer a single repository too, because in case of a Redbean internal plugin API change it's easier to update all plugins at once.

About naming, I'm always inclined to less smart names, "RedbeanPlugins" is much easier for a newcomer to understand what's happening. But I'm happy with any decision.

Bringing composer to the mix: I took a look at plugins already avalilable, it seems every plugin has i'ts own rb.php file... isn't it time to get a better composer support to avoid stuff like that?

[zewa666]

@daviddeutsch Although I like your names I would go with @marcioAlmada proposal to choose something simple. Guess in the end it makes things clearer.

Another idea which came to my mind would be to create a Boilerplate Plugin as Github repo. The plugin shouldn't really do a lot useful stuff, but simply show off how plugin developers should model their own implementations. We could show interaction with following things:

• How to organize files
• How to leverage composer
• Proper way to design UnitTests
• Simple Documentation on how to setup CodeCoverage
• Show simple interactions with QueryWriter
• How to go about the new namespacing

for me personally that would be a tremendous help, since I'm pretty new to all those concepts and I guess others could use a nice guide too. Not to say that we encourage this way the use of set standards.

[marcioAlmada]

@zewa666 The boilerplate idea is awesome.

@daviddeutsch But if all plugins are going to live on the same main plugins repo, how will new plugins get imported into that main repo?

Example: I have just started a plugin to implement issue #311, I'ts called redsql how could it be imported to that hypothetical plugins repository?

[daviddeutsch]

@marcioAlmada You simply create a Pull Request that has a new directory we will probably have a structure set up like this:

redbean-extra
   \-- Search
   |  \-- FlowerSearch
   |  |  \-- tests
   |  |  \-- FlowerSearch.php
   |  |  \-- README.md
   |  \-- ElephantSearch
   |  |  \-- tests
   |  |  \-- ElephantSearch.php
   |  |  \-- README.md
   \-- Query
      \-- Redsql
      |  \-- tests
      |  \-- README.md
      |  \-- Redsql.php
      \-- Bluesql
         \-- tests
         \-- Bluesql.php
         \-- README.md

That also makes it easy to assign responsiblity to a maintainer - they're simply responsible for their own directory. Likewise, it will make it easier in the future to use this in building distributions.

@zewa666 I think this planning is actually the best way - lead by example. It's all nice and well to put together documentation, but if you can show people right away, if they can look into the discussions and see code repositories change as we make new decisions, that will be more helpful than a documentation that we might end up being too lazy to keep updated :wink:. In the end, probably just the introduction to the redbean-extras repo would suffice to get you started on figuring things out for yourself. It will also help embedd people better into the community, I think.

[zewa666]

@daviddeutsch fully agree

[marcioAlmada]

@daviddeutsch agree too + I would eliminate the categories folder (Query, Search) to keep namespaces smaller: instead of RedBeanPHP\Extra\Query\YellowSQL we would have RedBeanPHP\Extra\YellowSQL

Repo would look like this:

redbean-extra
   \-- FlowerSearch
   |  \-- tests
   |  |  \-- FlowerSearch.php
   |  |  \-- README.md
   \-- ElephantSearch
   |  \-- tests
   |  \-- ElephantSearch.php
   |  \-- README.md
   \-- Redsql
   |  \-- tests
   |  \-- README.md
   |  \-- Redsql.php
   \-- Bluesql
   |  \-- tests
   |  \-- Bluesql.php
   |  \-- README.md

Other requirements:

• plugins should be tested using phpunit
• plugins should be PSR-0 compatible

[marcioAlmada]

But in the middle of the track I had one other idea. Maybe we should create a repo that is just a composer meta package and would have only this:

redbean-extra
  \-- composer.json
  \-- phpunit.xml
  \-- README.md

composer.json <<<

{
  "name": "redbean/extra",
  "type": "library",
  "description": "Redbean sugar plugins",
  "require" : {
    "david/rx" : "~0.1",
    "marcioAlmada/redsql" : "~0.1",
    "zewa666/QEB" : "~0.1"
    // etc...
  }
}

To add a new plugin, contributor would do a pull request to this repo, adding a new entry in require section. Main redbean repo would just need to add a dependency:

{
  "require" : {
    "redbean/extra" : "~0.1"
  }
}

Advantages:

• It's easier to keep semantic version for each plugin
• It's easier to keep semantic version for complete plugin horde in sync with specific Redbean versions.
• easier to automate tests
• fully adheres to composer.

Disvantages:

• we would still have a lot of repos :smile:
• depends more on composer.

Well, just an idea. Maybe it can inspire improvements on previous idea.

[gabordemooij]

I like the meta package approach.

[gabordemooij]

Just added a mount option to the phar-builder. Those who download the phar version of RedBeanPHP will be able to mount the plugins from within the far by using:

 REDBEANPHP_EXTRA = '/path/to/plugins';

[marcioAlmada]

Bump!

@gabordemooij Are you really going with the meta package approach? How can we help?

[daviddeutsch]

I agree that the way to go is to have separate repositories with one main redbean-extra repository that manages suggested (= accepted to be up to standard by the main project) repositories.

The main constraints that we're dealing with are the repository and composer path structures. redbean-extra would be its own repository, of course, but having things in subfolders would be confusing for composer, as it cannot deal with stuff like that. Then again, even if we just had it as a reference point for further packages, that would still be a bit confusing, since you don't want to always install all plugins at the same time.

The other consideration is more of a social/community consideration: It would be nice for everybody to maintain "their own" repository that is referenced as that makes things less centralized and I think depending on the number of extensions, having a central bug tracker etc. won't scale in the long run. Mostly what I'm worried about is that might end up making it harder to keep a common level of quality and communication between participating community members.

But the technical roadblocks trump that. Composer/repository structure - might be solvable by using composer to ship a kind of installer in redbean-extra which would pull from another repository (say, redbean-sugar) that actually holds the individual plugins. But that would still mean that you don't get a "run composer and you're done" situation, which is simply what people will mostly be using this.

The other consideration is unit testing, because it would be kind of hard to manage clean testing if one plugin failure would block other tests. You could solve this with branches, of course, but that's another overhead that might not scale.

So: redbean-extra would provide basic features, like the root classes that extra libraries would be using, plus extensive documentation. It would also feature the following composer.json:

{
  "name": "gabordemooij/redbean-extra",
  "type": "library",
  "description": "Redbean Extra",
  "suggest" : {
    "david/rx" : "~0.1",
    "marcioAlmada/redsql" : "~0.1",
    "zewa666/QEB" : "~0.1"
    // etc...
  }
}

Note the suggest. And those projects in turn might use the provide keyword to specify that they provide for redbean-extra - although I'm not sure whether that allows for multiple entries... Have to check on that.

But I think the most important thing by far will be the documentation and here is where the original "let's collaborate on a single repo" aspect should still work out. Of course, all repos would have their own documentation, but the redbean-extra repo would hold a concise version and overview pages on those. In a way, it would be the repository where people can learn about what they can do with RB addons.

[daviddeutsch]

Then again...

I've been looking at other projects to see how they're using composer (thanks for the links @marcioAlmada). I found laravels approach quite intriguing: laravel/composer.json

Particularlly the way they use post-install commands. This somewhat ties back to my other idea of optimizing RedBean by using static compilation, basically transforming RB commands into plain SQL snippets during installation. So if redbean/extras had a common interface that would, post-install, trigger common interfaces in those plugins, we would end up with precisely the hook that we need for that.

Now, of course - with the way composer is set up, redbean plugins could always simply declare their own post-install actions, but maybe a common interface is still a good thing here. Hmm...

[marcioAlmada]

@daviddeutsch It seems the post install command is executed only from the main composer.json. I guess that if there is a post install inside a composer.json below vendor folder it will not be executed. Hope it's not a problem.

I also like how Sublime Packages are handled too. They have a channel.json file (like a composer.json file) and an application that list those packages https://sublime.wbond.net/ and shows documentations for each project (a glorified README.md system).

With a "plugin channel", plugin owners would just need to follow specific guidelines to make plugin installation through composer automatic.

The good news is that our plugin channel already exists, it's called composer :smile: so maybe we only need to reserve a composer vendor for community redbean plugins: redbean-extra/plugin-name and do a web plugin hub just like sublime packages did.

[daviddeutsch]

@marcioAlmada Good info! And yeah, I guess that only calls all the more for an approach that is somewhat "managed" by the RB project. It's really just an additional convention. So a plain composer.json that requires redbeanphp/redbeanphp and redbeanphp/redbeanphp-extras will only get the software. But if you also do, say:

"scripts": {
    "post-install-cmd": [
        "RedBean\Deploy::all"
    ]

This triggers the real magic to happen - and it will also trigger -extras specific install/deploy magic.

The only thing we can never have is redbean-extra/plugin-name, since the project prefix redbeanphp/ will always be the first one.

To sum up, a composer.json for a project using RB and RBe would look like so:

{
  "name": "daviddeutsch/mycustomsite",
  "type": "website",
  "description": "Just my custom site using RedBean and some other stuff",
  "require" : {
    "redbeanphp/redbeanphp" : "dev-master",
    "redbeanphp/extras" : "dev-master",
    "daviddeutsch/rx" : "dev-master"
    // etc...
    "scripts": {
        "post-install-cmd": [
            "RedBean\Deploy::all"
        ]
    }
}

Another neat thing with this approach, a thing I've been talking about with @gabordemooij - It would be nice to have some kind of an interactive redbean kickstart. So basically like a one-page html app (probably angular driven with a tiny php server backend) that would lead you through the setup interactively and plugins could also hook into that - say if you have not made up your mind yet, they could show you "hey, would you like to write RB queries like this?" and/or "the plugins you have selected can do the following stuff, now that you have them installed" and it could lead to more crazy stuff.

[marcioAlmada]

@daviddeutsch

Don't you think "post-install-cmd" usage is too much? Laravel is a framework and every new Laravel project uses an app skeleton as starting point, they have artisan which is a command line tool integrated with the framework, so there is an infrastructure behind it that reinforces the "framework culture" and peripheral knowledge to understand how it works. Too much for an ORM that will be attached to any kind of project IMMO.

The most simple and effective idea right now looks like this:

1. Create a redbean-channel repository with a file to catalog all existing and approved plugins (yeaah like sublime channel but using composer). Plugin owners would create and maintain new plugins in their own repositories. To add their plugin to the mainstream channel, plugin owners would just create a pull request adding a line to the catalog.
2. Create a simple hub web app listing all plugins from catalog.
3. Plugin seekers would search for plugins and be able to see the documentation etc. Everything necessary to install a particular plugin would be on the readme - "plug this require in your composer.json". Plugin owners would be responsible to maintain their docs. Plugin seekers would be able to pick only the plugins they really need, instead of installing a full plugin meta package.
4. Just composer install would do the trick, leaving the post install for other cases like application deploys, etc.

[zewa666]

Sounds nice but what would be the workaround for people not working with composer?

[marcioAlmada]

It seems non composer users will have phar file available:

Just added a mount option to the phar-builder. Those who download the phar version of RedBeanPHP will be able to mount the plugins from within the far by using: REDBEANPHP_EXTRA = '/path/to/plugins';

But I confess I don't know exactly how it will work.