[Composability] Generators composability

SBoudrias commented 11 years ago

Hi there,

I would've post on Yo.next repo, but issues are closed over there, so...

Anyway, we spoke about how 2.0 should allow better composability between different generators. There wasn't any clear way of possibility, so we should brainstorm over how generators could work togheter.

I was thinking about the issue, and I felt like an intent API (like web intent or the android core intent system) could make sense in this case.

Let's say Angular generator generates a controller and would like to scaffhold a test. It could trigger an intent like .intent('test', { filePath: 'controllers/foo.js' }) and then other generators who're listening to the test event can respond accordingly. Of course test is an easy one, I can see how some types of intent could be harder to define... But it is a way of specifying standard communication protocol between multiple generators.

wesleytodd commented 11 years ago

+1 for this. I messed around with composing generators a few weeks ago and it was painful with the current api offerings. The idea of intents seems perfect. I think it would be good to have a menu display if multiple generators are registered for the same intent, something like:

Which test generator would you like to use?
1) Jasmine
2) Mocha
3) Karma

Input number:

btilford commented 11 years ago

+1 I like that idea.

On Sun, Sep 15, 2013 at 3:00 PM, Wes Todd notifications@github.com wrote:

+1 for this. I messed around with composing generators a few weeks ago and it was painful with the current api offerings. The idea of intents seems perfect. I think it would be good to have a menu display if multiple generators are registered for the same intent, something like:

Which test generator would you like to use? 1) Jasmine 2) Mocha 3) Karma

Input number:

— Reply to this email directly or view it on GitHubhttps://github.com/yeoman/generator/issues/345#issuecomment-24480074 .

addyosmani commented 11 years ago

To follow up, we're going to put together a design doc for composability very soon and establish a timeline for getting this in place. It's definitely the next big step for the project and is important to us.

SBoudrias commented 11 years ago

Still vague ideas, but I'll throw them here

Queueing

Follow up after a concerns in #162 brought up an idea about managing queues (like Backburner.js).

Composable generators could simply be called once, and they'd register all of their actions in a queue.

For example, a Generator-Angular could specify it is compatible with a test generator:

this.allowComposability('test', {
  default: 'generator-karma'
});

Once this method is triggered, the generator looks for a matching generator (maybe ask which available test generator to use on the prompt), then runs it. When running, Generator-karma simply add his required methods to relevant queues. This way they are both run in symbiose, not one after the other.

addyosmani commented 11 years ago

@SBoudrias I like the idea. Could you flesh out further ones for composability support in the Google doc we have going? Should have a link to it from todays email :)

SBoudrias commented 11 years ago

Yeah, I wasn't sure to write directly in the Google docs as it is a more formatted doc, and I just want to throw idea around for now. I'll try to put my mind together and make something more consistent.

Command

Also, an issue I see with composability is the case where two generator edit the same file (Gruntfile) and create a conflicts. This issue can't be resolved with actual "template" solution to fill file content.

A potential solution could be to extract away the common file generation to the generator system.

this.gruntfile.addTask("grunt-contrib-compass", {
    compass: {
        options: {
            sassDir: "assets/sass"
        }
    }
});

In a similar fashion, common configs could be held by the environment and shared between generators. For example, paths as to where the test folder is, etc.

simonfan commented 11 years ago

Guys, I've been fiddling with the idea of generator inheritance and extendability and have just put up some stuff, but haven't actually tested the idea. The idea is to allow generators to inherit other generators' subgenerators through invocations.

As generators' subgenerators are inferred through directory structure

generator-<%= generatorName %>/     // root
    app/                                // app-subgenerator (main) (generator:app | generator)
        index.js                        // runs app-generation logic
        ...
    <%= subgeneratorName %>/            // subgenerator-subgenerator (generator:subgenerator)
        index.js                        // runs subgenerator-generation logic
        ...

We could emulate that directory structure and create "proxy-generators" that invoke other generators and do stuff after the invocation is done..

generator-express/
    app/
        index.js            // scaffolds what express(1).
        templates/

inherit generator-express:

generator-express-base/
    app/
        index.js            // invokes 'express:app'
    route/              // creates a route
        index.js
        templates/

inherit generator-express-base:

generator-express-rest/
    app/
        index.js            // invokes 'express:app'
    endpoint/
        index.js            // invokes 'express-base:route' + endpoint specific tasks
        templates/
            _endpoint.js

inherit generator-express-base:

generator-express-markdown/
    app/
        index.js            // invokes 'express:app'
    markdown-server/
        index.js            // invokes express-base:route + markdown-server specific tasks

inherit generator-express-markdown AND generator-express-rest:

generator-express-app-server/
    app/
        index.js            // invokes 'express:app' + app specific tasks
    markdown-server/
        index.js            // invokes express-markdown:markdown-server
    endpoint/
        index.js            // invokes express-rest:endpoint

That way, "module generators" could be very specific on what they do, allowing for "app generators" to "inherit" multiple "module generators" functionalities..

Again I have not thoroughly tested the idea, but so far it seems to work... what are your ideas?

SBoudrias commented 11 years ago

@simonfan This seems good for extending an existing generator, but I think we're trying to find a solution to compose generator who'll work the other side.

I think the problem with the idea you're presenting is that it's easy for an author to customize/mixin an existing generator into its solution, but it is hard for a end user to combine with unrelated generators (what if they want to use express-markdown + less bootstrap generator?)

Though, that got me thinking looking through filesystem for commands and sub-generator may not be optimal for user who'd want to extend an existing generator. Maybe instead author should manually register commands on the environment.

module.exports = {
  app: require('./generators/app'),
  controller: require('./generators/controller')
}

Then author wishing to extend an existing generator with their new commands or overwriting an existing one could simply extend it:

module.exports = _.extend(
  require('express-generator'),
  {
    app: require('./generator/app')
  }
);

I'm not sure this would really be useful though - and it'd concern me about backwards compatibility.

addyosmani commented 11 years ago

This would break backwards compatibility as far as I can ascertain. That said, I do like the general idea of being able to easily extend generators in this fashion. Would it be possible for us to add more of these suggestions to the design doc for composability we have going at the moment?

csantanapr commented 10 years ago

I created a generator yo dude that its based on other generators

It's mainly to create Mobile Web Apps using Apache Cordova. User first selects one web app generators from a list (dapp, webapp, angular, mobile, polymer, backbone, jquery-mobile) then it runs the generator-cordovacli

In short dude is a generator created of two other generators.

I'm having difficulty on the unit testing side to pass parameters to the other generators from the the main one.

Would the new workflow help make it easier to compose generators of other generators and take testing into account?

Or this is an invalid use case for this workflow?

SBoudrias commented 10 years ago

@csantanapr This is exactly what we strive for.

rvangundy commented 10 years ago

Wanted to toss up a couple thoughts on this.

Thought 1: End user configurability

It would be nice to allow end users to configure of their personal generator environment without getting in to the existing generator class and inheritance pattern. To be strictly analogous with Grunt (this is only for high-level conceptual purposes):

yeoman.initConfig({
  angular : { // some config options },

  deployPack : {  // This is a reference to a hypothetical generator add-in I'm considering...
    heroku : true,
    ftp : true
  },

  coffee : {
    usePrompt : false  // Ignore prompting and force coffeescript conversion
  }
});

yeoman.registerGenerator('myGenerator', [
  'angular',
  'coffee',  // A hypothetical generator add-on that converts everything to coffee script
  'deployPack'
]);

yeoman.registerGenerator('myGenerator:model', [
  'angular:model',
  'coffee'
]);

Then the user can call their own generator with yo myGenerator and yo myGenerator:model.

Thought 2: Functional composition vs. classical composition

Rather than consider all the ways you can mix in Generator classes, maybe simplify the interface a bit. For a given generator script, make it possible to export a simple function rather than the generator constructor. Make the only parameter to that function a root generator, like so:

module.exports = function(generator) {

  // Push new prompts on to the generator
  generator.prompt(// some new prompts);

  // Add some new algorithm for working with the current context
  generator.use(function(ctx) {
      var prompts = ctx.prompts;
      var fs = ctx.fileSystem;
      var pkg = ctx.pkg;

      // Do whatever you like in the current generator context
  });
};

The generator algorithm would work as follows:

Instantiate a new base generator
Pass the generator consecutively to all tasks as registered in 1
Run prompts
Run other procedural methods in the order specified in the registerGenerator method

So this may seem like a major push away from how everything is done now, but I think it offers a certain flexibility to both the end user and plugin authors. It might be possible to support legacy generators as well, since the "root generator" that is passed through consecutive functions could be instantiated based on the base generator class or any other legacy generator class.

SBoudrias commented 10 years ago

@rvangundy About your first though, don't you feel the base idea is resolved with the use of .yo-rc.json file?

rvangundy commented 10 years ago

@SBoudrias Yes, that does solve the configuration issue.

The second part to the first thought is the ability for the end user to compose their own flavor of generator through the registerGenerator function. Pushing the composition responsibilities up to the end user allows for greater flexibility at the config level, and forces generator module authors to be more generic in their implementations. I think by encouraging composition deeper in generator code you may create a lot of complex dependency chains and more rigidly designed generator modules.

SBoudrias commented 10 years ago

We sure plan on giving end user the choice of the generator being composed. But Yeoman is a scaffolding tools, as such, it shouldn't need to be configured by a config file prior to be runned.

IMO (in regard to my previous point) all the composability features should be managed and available directly in the scaffolding flow bundled in the tool.

addyosmani commented 10 years ago

@csantanapr nice work on the generator! This brings up interesting best practice questions about unit testing composed generators. Each individual generator you call will likely have its own unit tests but I imagine things get complex when they're executed together in sequence. Could you summarize some of the issues you ran into so we can keep these in mind?

csantanapr commented 10 years ago

@addyosmani The issues I hit was calling the generators in sequence and passing options to each one in the chain, I failed trying to do it in unit test.

I want to be able to have a simplified way of specifying all options upfront, and then call a single function that starts the sequence.

var allOptions = { "gen1": options: { "opt1": }, "gen2":options: { }

rungenerators(["gen1","gen2"],allOptions,function(err,results){

JohnMorales commented 10 years ago

Any thoughts on supporting files that can be modified by a user? It seems like if you were to address files beyond paths then it would be possible, at least for files that have structure.. e.g. /Gruntfile.js#yeoman/initConfig/deployPack, then you could inject code into the file without overwriting the whole file. Therefore persisting changes made by other generators and/or users. it would of course require parsing the file.

I realize this is a bigger change, but it would satisfy two needs for the price of one.

SBoudrias commented 10 years ago

@JohnMorales We already worked on some tools to allow generators author from injecting content into a file. For example, the whole wiring mixin methods are helpers to edit HTML content.

Then, for more specific use cases, it's the job of the author to use code modification tools like AST Query (for JS) and Cheerio (for HTML). I don't think Yeoman need to wrap these tools because they already offers a façade to simplify file edition.

JohnMorales commented 10 years ago

I see your points and I didn't know these tools existed, I'm having a hard time then seeing why don't we leverage them for this? Are they not good enough for the job?

SBoudrias commented 10 years ago

I personally use them in Generator-BBB. We're going to improve documentation soon, I believe tips and tools to help with some common tasks is in the scope. https://github.com/yeoman/yeoman/issues/1259

robwierzbowski commented 10 years ago

Composability fundamentally changes the way authors will write and users will use generators. If APIs can be simplified and improved by breaking compatibility, I don't think Yeoman should shy away from a 2.0 release, or a composable / classic split.

robwierzbowski commented 10 years ago

The biggest challenge I see is in composing custom tasks in the Gruntfile. The serve and build tasks in projects I've worked on have gotten pretty byzantine.

Enforcing more workflow/directory standardization and breaking generators into atomic pieces with their own conventions (asset preprocessor generators (Sass, Less, CoffeeScript), asset postprocessor generators (autoprefixer), asset minifiers) will probably help.