plopping projects - request for input

[amwmedia]

I'm going to be soon introducing the ability to plop an entire project codebase and I want to throw out the approach I'm currently considering for feedback. My plan is to enable a syntax something like the following.

$ plop --project http://github.com/some/repo --destination ./some-repo or shorthand $ plop -p http://github.com/some/repo -d ./some-repo

this command would trigger the following...

• inquirer prompts to request the destination directory (if --destination was not provided)
• git clone of the repo
• locate a file by the name of plopinit.js. This would use standard plopfile syntax to setup helpers, partials, and generators to be used for initialization
• plop would be executed against this file. If only 1 generator is found, that generator would be executed (prompts and actions)
• before any of the actions execute, all files in the codebase would be parsed for double curly braces {{ and each instance found would be replaced with an escaped version \{{. This replacement would also search for another set of tokens (thinking {{# #}} or {{% %}}). These would be replaced with standard curly braces.
• plop would then run every file in the codebase through handlebars for processing
• the escaped curly braces would then be reverted back to their previous state
• the actions array would execute
• the plopinit.js file would be removed

Benefits to This Approach

• uses all of the same plop concepts, very low additional learning curve
• allows the possibility of flexibility by defining multiple generators (more than one way to initialize the project) without requiring it (if there's only 1 generator, it's executed by default)
• allows close to standard handlebar templates to be used for initialization while keeping any use of curly braces within the codebase safe from accidental replacement during initialization.

/cc @nicoespeon @kentcdodds @mxstbr @calcaide @kevin-wolf

Looking for questions or input

[tpizza]

I like the idea for sure, man.

This would actually solve the challenge I was facing earlier today for automating a good portion of project setup. Instead of having to run a whole scaffolding routine via Gulp, this would cut out that entire step.

One thing that would be cool for project initialization is a cleanup method. Haha, something like:

plop.slop([
  'file1.txt'
  'file2.txt',
  'path/*'
]);

Just thinking maybe having some means of deleting things like md files that are we'd only really ever want to see in that template codebase's repo.

[nicoespeon]

Hi @amwmedia!

I did not use plop on my recent projects so I'm a bit out of date.

Still, I find this to be a nice idea. It sounds not adding too much complexity and I can see the use cases 👍

I was wondering "maybe we should allow to retrieve the project locally, not just on a distant repo like the example", but I think that can be delayed for later if people start asking for it − or maybe there is no big deal if we are just cloning the repo given as the input, whether it's a local or remote one.

Thanks for continuously improving plop, that's cool 😃

[kentcdodds]

I use yeoman to create new OSS projects and it's really useful. I may think about swapping to this though. I have a few thoughts/questions:

1. If you're escaping {{, how would you do actual interpolation?
2. I would definitely like the ability to point to a directory rather than a git repo. Having both would be useful.

Sounds like a great effort :+1:

[amwmedia]

@tpizza Absolutely, what I was originally thinking is that this would be easy to do with a simple action function. You could run rimraf to clean up any files that are not needed in the end, however it might make sense to make an official action for this because it's definitely going to be a common need. Thoughts?

[amwmedia]

@nicoespeon yes, i would definitely see a benefit to being able to get the codebase from the local file system, a corporate network shared drive, etc. good suggestion 👍

[amwmedia]

@kentcdodds i probably didn't explain this very clearly (I WAS writing it at almost 1am after all haha). That's where the {{% %}} would come in. For interpolation during the init process, you would use {{% %}} in place of your normal {{ }}. That way if your codebase uses double curly braces for anything (very common), they won't be processed during initialization.

so this

<h1>{{% upperCase siteName %}}</h1>
<a href="{this.state.url}" style="{{fontSize: 14}}">go somewhere</a>

would turn into this

<h1>{{ upperCase siteName }}</h1>
<a href="{this.state.url}" style="\{{fontSize: 14}}">go somewhere</a>

for initialization, the h1 would be processed by handlebars, while the \{{fontSize: 14}} would be processed by handlebars to be back to the original {{fontSize: 14}} so that the end result is functional jsx.

I hope that was clear 🙂

[amwmedia]

Also, what would you all think about implementing a search interface where you could search a central repo of seeds projects? something like plop --project without a value could kick it off. Something like that might really make the overall ecosystem more robust.

[kentcdodds]

Actually, why don't you do the community/registry thing with npm like Yeoman does? Just have a keyword they must have and a website that shows packages with that keyword or something...

[tpizza]

@amwmedia Yeah, I am thinking the use-case for folding that in as a core action would be common enough to where including it makes sense. As usage continues to expand throughout the community, it'll definitely come in handy for it to be in the API versus creating a custom action in every plopinit.

I think what you're proposing for the search interface could be highly valuable. I'm wondering if the central repo could actually be a reference to a key within an object of repos containing projects to seed? The idea then would be to have another key in there that's an array of additional repos that could contain either community-driven or private seeds to start projects from. That's assuming of course the architecture required to make that work can be reproduced within a private repo containing those seeds. Something of that nature would be highly valuable to teams out there who are unable to make all of their code publicly available for political reasons within their organizations. I have a few other ideas for that if it seems feasible. Ultimately though, I think this could be ridiculously useful, especially if its architecture by design keeps in mind filtering out the kind of noise we see in the Yeoman registry.

[calcaide]

@amwmedia looks pretty good idea and I like the approach. As @nicoespeon suggest, I think is a good idea to allow retrieve project locally. Also, I like what @kentcdodds is suggesting, in my plop generator projects that I have in NPM, I use the word plop- maybe is not the best word (too generic maybe), but when I did it plop-generator it seemed too long. And yeah, for sure, as you and @tpizza are mentioning, it makes sense to add that action to clean up as a official action.

[amwmedia]

@tpizza I'm not sure I'm following what you're suggesting here as far as the private repo stuff goes. Can you clarify further?

[amwmedia]

@kentcdodds Seems like npm may be a logical vehicle for this. It allows public and private projects, offline, versioning, etc.

The thing I want to stay away from is the idea that you "install" this "program" that generates a site. I see it more as a codebase that is copied/downloaded, then prepared and customized via an initialization script. Maybe I'm splitting hairs here and there's not really much of a difference, but I don't believe so.

/cc @budacoder

[tpizza]

@amwmedia What I was thinking in terms of adding a private project repo would be adding additional sources for plop --project to pull from. Sort of like adding additional apt-get repos in Ubuntu.

A general use-case I'm thinking of for this is I know the security folks at my organization would likely take issue if our templates were publicly visible. So having the option of setting our local plop installs to pull from custom repos would be beneficial.

[amwmedia]

So what about something like this.

$ plop --project npm-module-installed-local-or-global

copies and initializes the codebase in this module to the current folder as the destination. We might want to enforce the idea that the cwd needs to be empty first.

---

$ git clone http://github.com/someone/repo ./my-proj
$ cd ./my-proj
$ plop

plop detects the uninitialized repo and triggers the init process

---

I think we could easily use the package.json file to configure the init process (see the example below). The json parameters initPlopfile and initDependencies would always get cleaned up automatically after init completed.

package.json example

{
  "name": "something",
  "initPlopfile": "init-script.js",
  "initDependencies": {
    "things-needed-by-the-init-script": "^1.0.0",
    "not-likely-needed-in-most-cases": "^1.0.0"
  }
}

[amwmedia]

@tpizza does the npm approach solve this for your use case? I would assume you could use artifactory to pull your enterprise codebases for generation?

[tpizza]

@amwmedia you know what? Yeah that would be more than adequate.

The idea I had in my head at first is a bit more complex than it needed to be. It seemed like a good idea in my head at the time, but not so much IRL as I typed it out. In short it would have involved pulling a private repo to an internal location which would contain all your different project templates. From there then you'd use a command like plop --project --private which would then give you a list of project generators in the same fashion you see when plopping anything else. It would be a cool uniform DX, but sort of overkill the more I think about it.

[amwmedia]

@tpizza ah, i see, so you were thinking of a way to drive the search interface via an internal registry. I think the only real problem with that is that it could be confusing for the developer unless we make some clear signifier that shows what registry you are pointed to, and how to change it.

This might be something we could address in the future though.

[nerdgore]

I like the idea very much. IMO it would be nice that if npm is used you don't need to type anything extra. npm knows of private repos and probably does the proxying anyway. My ideal would be something even simpler like plop init plop-project-base which would utilise npm to resolve whether it is a repo, published module or local path. If no destination is given, plop would query whether I want to init to the current folder.

If I just call plop init it would be great to make an npm search for project names starting with plop-* and use that as completion list. ( I can dream, can I?)

[amwmedia]

@nerdgore I like this! The idea of repurposing the -init is great because that feature really doesn't do much at this point anyway, and I'm all for keeping things simple for the user. It also plays nicely into a widely accepted pattern (npm init, git init, etc)

[amwmedia]

After thinking this over in more detail. I believe that my original idea of copying and processing the whole codebase may be lacking. I don't like how it introduces more concepts and complexity. After thinking further about this, I believe we can use the existing plopfile interface to do project scaffolding as well. Here's what I'm thinking...

• a project generator would be a normal plopfile that can be referenced via path, or globally installed via npm
• when plop is executed with --init some-plopfile we find the plopfile referenced and run it.
• if the plopfile specified has only 1 generator, that generator is immediately run

The one downside to this is that there are (potentially) a LOT of files that need to be plopped so the list of actions could get long. To mitigate this, we may need additional actions that can work on collections of files instead of individual ones. I already have in place (soon to be released) the ability to create custom actions, so that will help a lot too.

Thoughts?

[nicoespeon]

I like the idea of not adding too much concepts / complexity to the tool, so +1 for this one.

Maybe we can make a POC of it, to feel how it's going on.

[MartinMuzatko]

I'd love having plop as a scaffolding tool for entire new projects. I imagine that managing new projects using the approach outlined in the previous post by @amwmedia to be a great way of tackling this issue. Do you think about adding a generator directory similar like yeoman to the github pages?

I saw they manage their generators using package.json keywords. (yeoman-generator) Are there any plans yet how to deal with that?

[amwmedia]

@MartinMuzatko Yes! I would definitely want to set up a publically searchable directory. I'm a little torn on the method, though. The NPM keyword search seems like the obvious choice at first, but I also think there's value to using a separate list because it would keep the number of choices down to just the projects that are intended for general consumption. I think yeomans approach currently suffers from project dilution. Everyone who used the generator-generator to generate a generator (I just love saying that) is now in the search results so it's hard to find the quality projects.

I'm open to ideas though. :-)

[brandonbuttars]

I think the current generators are fine, I like the idea of adding a new default action for copying a folder of files or any type of file without trying to replace text. Maybe just add a default action called 'copy'.

Just Set a New Generator

plop.setGenerator('framework-name', {
    ...
});

Add Actions to the Generator

if copying a directory

actions: [{
    type: 'copy',
    path: 'app/new-directory,
    copyPath: 'plop/templates/scaffold-directory',
    directory: true
}]

if copying non-text file or whatever

actions: [{
    type: 'copy',
    path: 'app/directory/new-file.ico,
    copyPath: 'plop/templates/favicons/favicon-1.ico',
    directory: false
}]

[amwmedia]

@brandonbuttars yes, I was thinking the same thing. Just adding an addMany, copy, and copyMany would suit the need I believe.

[brandonbuttars]

Yeah, I like the idea of addMany, copy, and copyMany. Perfect! I'm excited to see this implemented. That will make it possible to generate pretty much anything from my perspective.

[MartinMuzatko]

@amwmedia @brandonbuttars, shouldn't a glob pattern imply, that we are copying/adding many?

[amwmedia]

@MartinMuzatko it would, and we would likely use a glob pattern. I'd have to think about whether it would be possible to roll the addMany functionality into the add action. Not sure it's possible without breaking backward compatibility, but maybe.

[rosshadden]

Is it (still) the case that plop doesn't allow you to bootstrap entire projects? Is there a concept of a global plopfile, which could be used from anywhere and thus be used for such a thing?

I was looking into seeing what was out there besides yeoman, but what I want is something that can generate both an initial project structure and specific files/workflows afterward.

[amwmedia]

This is not currently in place, but it's one of the next things that I'm looking to add. I don't believe it will particularly hard to implement, just haven't gotten to it yet.

react-boilerplate and others have been using the repo cloning process for their initial project scaffolding, then they use plop for asset generation going forward.

[amwmedia]

I've been thinking about this more lately and something hit me the other day. Wouldn't it be better to provide a way to generate a CLI from a plopfile? This would remove the whole step of installing plop from your "getting started" process. If we could make it super easy to build a custom CLI that uses the plopfile API to execute prompts and actions, wouldn't that be better than something that requires plop to run?

[nicoespeon]

🤔 I personally tend to install plop as a dev dependency and create a npm alias so people don't have to install plop if they don't want to. Doesn't it solve this need already?

Concerning the issue: we now have addMany. Do we still need to implement copy / copyMany?

For what I understand, this is something that can be achieved by addMany if there is no variable to replace − but I may be missing something.

[amwmedia]

I can see why people want to be able to drop a whole seed project with a CLI instead of using a clonable repo like other projects have done. You get more control over how things are created, you can conditionally install certain parts of the codebase, etc. But I'm kind of unsure if it's better to have a package that is read by the plop binary (the way yeoman does it) or if there's a better, simpler, way to do the project scaffolding.

add and addMany would still handle the data in the file as a template, so it won't work well for copying files as demonstrated by issue #86.

[alexkreidler]

Any updates on this?

[SanBen]

• before any of the actions execute, all files in the codebase would be parsed for double curly braces {{ and each instance found would be replaced with an escaped version \{{. This replacement would also search for another set of tokens (thinking {{# #}} or {{% %}}). These would be replaced with standard curly braces.

@amwmedia How would one go about to do this? It would be very useful to leave the existing curly brackets intact and use something like {% %} but currently it seems quite difficult to achieve without much effort. The idea is to scaffold an entire project as you described.

What is the status on these features or has anybody run into similar problems, any suggestions? 😄