New CLI Architecture

[paganotoni]

This PR changes the approach we've been taking on the CLI aiming to provide the ability to plugin custom commands to the Buffalo CLI, at a higher level this would compact the stack that will be supported by the Buffalo team and allow Buffalo developers to write plugins that solve their workflow specific needs.

In terms of the ability to override the CLI there are 2 main needs:

• Allowing to add plugins on a Buffalo codebase
• Allowing the use of custom plugins to generate Buffalo applications

These two capacities are vital to improving the out-of-the-box experience for Buffalo by allowing developers to provide support for their own tools and let the Buffalo maintainers focus on a better experience for the frontend stack.

Also, Some of the resulting effects of this new shape of the Buffalo CLI are:

• Reducing third-party dependencies (using more of the standard lib)
• Cleaning up old issues
• Improving the maintainability of the Buffalo CLI.

In order to achieve those 2 overall goals we should complete the following steps:

• [x] replace help instructions
• [x] convert base commands to plugins
• [x] rewrite pop plugin
• [x] add plugins command (to list plugins)
• [x] implement plugin override API (remove, add, clear)
• [x] implement custom CLI main feature (in cmd/buffalo)
• [x] implement global per-user default plugins (in $HOME/.buffalo/cmd).

Beyond these tasks we will need to do other tasks, these would be worked in separate PR's to keep the progress flowing.

• [ ] Rewrite plugin documentation (https://github.com/gobuffalo/docs/pull/684)
• [ ] Add Generators to override CLI
• [ ] Improve our testing coverage
• [ ] Extract Pop plugin and move to its repo
• [ ] Review and update CLI documentation
• [ ] Redo our integration testing
• [ ] Pluginize existing commands to allow these to be more extensible.

[paganotoni]

Got started with the documentation: https://github.com/gobuffalo/docs/pull/684

[sio4]

Congratulations and huge thanks for your hard work!

I just quickly checked the command line options and help messages along with checking the PR for docs, and it seems like some features are broken (such as the result of buffalo g resource book title:string is not the same as my expectation but it was the same as the result of previous command buffalo g resource resource book:string title:string).

However, I think the focusing area of this PR is the brand new plugin architecture. Actually, I am not fully sure of the direction but what I feel is that:

• the legacy method which is allowing any binary/script with command/output protocol is fully deprecated.
• things in https://deploy-preview-684--gobuffalo.netlify.app/documentation/cli/extending/ is the core of this PR.
• now plugins should be written in golang and compiled in the user-specific and/or project-specific buffalo command.

Is my understanding correct?

I am somewhat surprised because, actually, I love the binary/script style plugin approach since .so /.dll style plugins are not a possible way in the go world and the previous approach successfully covers the limitation (adding plugins without modifying/compiling the core). Also, I feel that this extending approach is more likely closer to the project-specific tasks rather than command line plugins.

I think the concept of overriding(replacing) the default commands could be a good thing, but it is quite an advanced feature, and it could make it hard to keep the concept of buffalo as "easier web development in go".

Could you please correct me if I misunderstood, and could you provide some direction or concept behind the future direction?

[paganotoni]

@sio4 I think you're right regarding the direction. The focus of the PR is the brand new architecture. While on it I noticed a lot of areas for improvement on existing commands but decided to reduce the change there as much as possible.

In terms of the first point (the legacy method), I think it will be up to the developer to extend the plugins, developers would be free to invoke or implement the old mechanic within the commands being added by shelling out from them. But it will be up to the developer to do that.

Regarding the direction or concept for the future here are a few ideas I'm hoping this facilitates:

• Pop plugins are one set of plugins and not a mandatory set.
• The CLI provides a simpler frontend stack and moves the one with Webpack to be optional.
• Focus on providing a better experience with a simpler default stack, particularly around accelerating the development workflow.
• Reduce the amount of code/configuration generated/maintained by the users. Even tho it may look like magic (Convention over configuration).
• The CLI mechanics are simpler
• The Integration tests coverage is higher in the CLI

I would love we move in that direction as we develop the CLI. I think the more we can simplify the default stack the better the experience developers will have. This new CLI version would let the door open for those advanced users who want to have their custom tooling included.

[sio4]

Thank you for more explanation. Actually, I just had quick simple tests so I am not sure if I understand the new behavior correctly. At the first sight, there are still a few questions regarding the direction and the implementation.

In terms of the first point (the legacy method), I think it will be up to the developer to extend the plugins, developers would be free to invoke or implement the old mechanic within the commands being added by shelling out from them.

Right, plugin developers still have a chance to invoke external binary/script within their plugin when they implement what they want to do but this was not the point of my question. The main point of my understanding of the legacy approach is that it provides a way for real drop-in plugins. Installing a plugin is super easy, just drop them in the path and add a config (mostly done by buffalo plugins install ... though).

But with this new approach, if I understand correctly, I think users need to write cmd/buffalo/main.go with all plugins they need to use, and then every time they run the plugin commands, buffalo will build and run it. (projectOverrider())

I am not sure which problems we solved with this new plugin architecture. Maybe I need to understand more about the actual user's workflow for example, how this current workflow will be changed?

• buffalo new coke ...; cd coke
• buffalo plugins install github.com/gobuffalo/buffalo-auth
• buffalo g resource book title:string
• buffalo dev

The above sequence will generate a semi-working scaffolded application with four or some more commands. How it will be changed in the new architecture?

By the way,

Since this new architecture is not compatible with the old version in both the code level and usage, and it seems like we need more POC on the new version while we could need to fix issues on the legacy, I would like to suggest you the following managing steps (rough idea, maybe we need to consider some more):

• drop the main branch. we don't need main/development approach anymore in the age of go module.
• rename development branch to v1, make the legecy CLI as v1, and freeze it for a new features. (just fixes)
• rename this branch as main or just v2, releasing snapshots as alpha so we can control the dev easier.
• the future development will be on another branch and will be PRed against this v2 branch so the management and review could be easier.

[paganotoni]

Thanks for giving it thought @sio4! I'm loving the conversation we're having here, I had a rough week which gave me the opportunity to think about your questions and think about the possibilities I have not outlined here. And made me notice I need to add more examples on the docs!

I am not sure which problems we solved with this new plugin architecture.

Sure, when we see the Command example there is nothing different than what we had with the previous architecture but that's the only thing this new CLI will allow. The previous way of doing it is more of an all-or-nothing where you had to write the entire command start-to-end, and you cannot extend existing functionality, with the new approach extensibility gets more granular, here are some examples of things developers could do:

• Add its own way of testing without writing the test command
• Start/Shut-down services needed for their tests
• Run separate commands when dev is invoked
• Write their own generators (p.e Resources) within the same generate command

And even add commands that could be extended by the use of interfaces. In terms of the current workflow

Maybe I need to understand more about the actual user's workflow for example, how this current workflow will be changed?

This will change it totally unless we provide a way to manipulate the plugins via code generation which still can be an option, we're just not there yet.

Regarding main/development I'm with you. I think we have discussed it makes remove that separation as it only adds more complexity to our development/deployment flows.

[paganotoni]

Hey @sio4!

I did some progress on the branches.

• Updated main with development
• Set main as our main branch
• Removed the development dockerhub task
• Updated Dependabot to send PRs to main
• Removed the development branch.

I'm hoping to do more progress on this soon.

[paganotoni]

@sio4 One more thought I've been having is the following. the CLI is v0, and v0 means (AFAIK) that it's rapidly changing and there are no API guarantees. Which I think is the reason most people want to see Buffalo hit v1.

I wonder if we're being too careful with Buffalo CLI and Library when implementing changes, what do you think?