Pull Third-Party Stencil Collections When Building

[prikhi]

We should modify the build script to download or update local copies of 3rd party extensions(possibly through git modules?) so that they can be included in Pencil releases.

[zenware]

I mentioned dependency management in #678 and I think that would actually lend well to this also. The best way to implement it might be GitHub, but I feel like instead of submodules (although perhaps submodules for the pre-packaged ones) we should use some easily parsable file type, perhaps JSON, to include data like this:

priki/collection1
zenware/collection2
<github_username>/<collection>

I'll be reading through the codebase this weekend to see if I can help contribute anything, and figure out more about whether what I'm talking about is possible the way I want to realize it.

[prikhi]

If we were to use JSON I think it'd be best to have something like this:

[
  { "name": "Material Icons",
    "repo": "http://github.com/nathanielw/Material-Icons-for-Pencil.git",
    "buildCommand": "",
  },
]

That way:

• we can use name to determine the folder to create in the content/pencil/stencil/ folder(stripping spaces or replacing them with underscores)
• the repo isn't tied to github
• a buildCommand can be specified, or left blank to indicate no building is required

We would have to pick some cli utility to parse the JSON file for use in build.sh (jq or jshon come to mind - not sure which is more popular).

As an aside: I'd really love to use YAML instead of JSON for human-edited things like this, but the dearth of YAML CLI utilities makes this seem improbable.

If we go with submodules, we'd probably stick them somewhere like build/Stencils/ & copy the folders directly(so we wouldn't need a name). We could check for a Definition.xml file to figure out whether the collection needs to be built or not. I think the build commands would have to be hardcoded into build.sh - unless we asked stencil devs to have a standardized build file for us to hook up to. Updating the submodules would have to become part of the release process(git submodule foreach git pull origin master).

So it seems like JSON would be more maintainable & expandable, although it would add a little more knowledge overhead for comprehending the build system(knowing JSON & whatever utility we use to parse it).

[zenware]

That's why I'm saying it could be even more simple like one username and one repo per line, then users wouldn't need the overhead of knowing a new markup language really. Stencil collection maintainers shouldn't have a problem creating some sort of standardized manifest file which contains the name, version, and build commands though.

[prikhi]

Sorry I didn't mean to make it seem like I was against using JSON, I just wanted to list out the pros & cons, so I could think on it. I think it's an OK trade off, but I imagined it would only be knowledge required for Pencil's developers/maintainers & stencil developers. So this little bit confuses me:

users wouldn't need the overhead of knowing a new markup language really

Are you saying we should let end-users install collections by adding a username & repo in a config file, similar to antigen or vim-plug?

I'm thinking of something like getting stencil devs to supply a manifest.json with their collection that contains a bunch of metadata, including the following attributes:

• name
• version
• downloadUrl
• requiresBuild
• buildCommands
• manifestUrl

For including collections with Pencil, we just add the repo to a file in build/. The build script can clone/update all repos listed in the file & check the manifest.json to see if a build is required.

• If requiresBuild is true & buildCommands doesn't exist, the script will download the pre-built zip from downloadUrl(into some location dependent on the version in order to prevent redundant downloads). If buildCommands exists, the script will instead run them inside the cloned folder.
• If requiresBuild is false, it'll just copy the files into Outputs/Pencil/.../stencil/${name} where ${name} comes from manifest.json.

We can also eventually expand this into something that allows users to easily install & auto-upgrade collections. Users could enter a file path or url to a manifest.json or the collection's .zip. Pencil would download & compare manifestUrl to see if a newer version is available & download that using the downloadUrl instead. Then it could be handled like regular zip imports. For automatic updates, Pencil could compare the manfiest at manifestUrl to the installed manifest.json, if the version differs, it would automatically download & replace the collection from downloadUrl.

Then, way in the future, we can use the manifest.json to create a Pencil Stencil Repository or something similar, where stencil devs can upload their manifest.json, screenshots, the collection's .zip, etc to some server. We could add a window to Pencil that talks to the server and fetches the manifest.jsons and allows the user to browse, search & install all collections listed in the repository.

This heavily rips off of how firefox does addons, here's an example of what firefox stores(in your profile's addons.json): https://gist.github.com/prikhi/795a025c829b45769aa7

Comments? Suggestions? Feedback?

I'd also love to hear from @nathanielw about all this, as he's the only active stencil developer I know of :P

[zenware]

Yes, the actual concept I had stewing in my head was similar to that of systems like vim-plug, antigen, NeoBundler, etc.

I just mentioned JSON because it's been overly popular lately and I figured it would be more likely to be accepted.

I definitely like your idea though, especially when you started talking about the Stencil Repository, probably because I imagined it as a handy web interface like https://rubygems.org/ or https://www.npmjs.com/

I still think hosting could be entirely done with Github though, it looks like it works incredibly well for a ton of projects, but maybe that's just me.

[nathanielw]

Cheers for the mention @prikhi, I completely forgot to subscribe to this issue.

Your description of the Firefox-like addon system with manifests and update urls is exactly what I was imagining when I mentioned an "online stencil registry" over in #490. As you said, in the future it would make it really easy for user to update and install collections with the click of a button.

Regarding building bundled collections, everything you've described sounds good to me. The only downside I can see is that anyone building Pencil ends up having to install a bunch of extra dependencies (e.g. my Material Icons collection relies on a few Python modules). Maybe there should be a flag to force the pre-built zip to be downloaded, even if buildCommands is set.

[zenware]

@nathanielw Do you think there would ever be a situation in which prebuilt stencils is not a good idea? I feel like if we need to be eliminating dependencies, then it may as well be that just the prebuilt stencils are served. If anyone actually wanted to build it themselves they should have the knowledge and free time to do that. This would also remove two lines from the manifest and the need to implement that functionality entirely. No code is better than feature bloat.

[nathanielw]

@zenware I was thinking about that - I'm struggling to come up with any major advantages of building the stencils, apart from the possibility that the download for the compiled version might disappear/go down (but I suppose the stencil's source repo could just as easily disappear). It'd also mean that building would require an internet connection (unless the pre-built stencils are committed to the main repo, but I think avoiding that was one of the reasons the idea of building stencils came up in the first place).

I see now how your idea over in #490 :

What if stencil repositories just had a "built" branch and it would pull from that instead of the master branch.

...would be useful, but it would make things more difficult for stencil developers to set up if they wanted their collection included by default.

[zenware]

I'm aware you are a stencil developer so of course you don't want any added complexity and I understand that entirely. This discussion seems to be "To whom do we leave the responsibility, maintainers, distributors, or users."

Since the answer is definitely not the users something extra has to happen for the pencil project itself, and for the stencils, no matter what those somethings are, they will be occurring in both places. My goal is to just poke and prod at various ideas until we find the one with the net-least-effort.

[prikhi]

I'm fine with ditching building collections, I only imagined building collections when building Pencil - not when users update or install a collection. The only benefit I saw(which seems small) was that whenever I build/release a new version of Pencil it would include the latest versions of the bundled collections instead of the last released ones.

[zenware]

@prikhi So, only add it if there is demand?

[prikhi]

On Sun, Apr 26, 2015 at 05:54:35AM -0700, Jay Looney wrote:

@prikhi So, only add it if there is demand? Yeah seems fine to me.