Recipes feature review

[faustinoaq]

Description

Distributing recipes as shards feels like it'll be quite a wedge. They're not versioned, have no dependencies, and really aren't even crystal code. They're just collections of templates and folders. A zip is a great way to package a recipe for download, I just don't think that this way of organizing and distributing "official" amber recipes is going to scale well.

If Amber is really going to provide a handful of blessed recipes, I see no reason why they can't all live in the same repository. But I'd like them to live as code, not as zips, so pull requests are functional and file history works. A CI system can create zips and publish them to s3 or something for distribution.

I think it also makes sense for others to be able to publish a recipe as a single git repository, but keeping up with N repositories for blessed recipes feels exausting. Amber is already 21 separate repositories. Imagine trying to update for a breaking change across N repositories, that requires coordinating pull requests and branches across every repository to match the amber core.

by @robacarp

Steps to Reproduce

Not applicable

Versions

Master https://github.com/amberframework/amber/commit/bea1063f640e0c7466989fd50cd0453b812d7a39

Additional Information

/cc @damianham @drujensen @Adam-Stomski

[faustinoaq]

I think keeping recipes source code is fine, then create zip files from source and storing them inside recipe directory is a nice option.

[drujensen]

Distributing recipes as shards feels like it'll be quite a wedge. They're not versioned, have no dependencies, and really aren't even crystal code.

I'm not following this logic. A recipe will be dependent on the version of Amber and possibly other shards. For example, it may be a MongoDB recipe that would inject a shard dependency. The version of this injected dependency will be specified by the recipe. IMO, It is code, even though its in a templating language. My take is that we will need a packaging management to handle recipes and that a shard is the best paradigm in crystal.

[damianham]

The recipe contains an app component which has a shard.yml with the dependencies listed so once you create an app from the recipe you are using the versions of the dependencies give in the shard.yml.

[drujensen]

@damianham what happens when Amber has a breaking change? Do we need to update all the recipes to avoid broken templates? How is versioning going to be handled with zip files?

[faustinoaq]

@drujensen You have a good point, so I think we need to use one (1) recipe per repo anyway, and keeping the .zip packaging in the release page of each repo. No no No :no_good_man:

@damianham I guess this feature already support fetching from external repositories. I think we can keep the .zip feature, allowing just one (1) recipe per repository, storing the .zip files in the /release page of each recipe repo, so we can finally download .zip files according to the tag. neither :no_good_man:

And we can create an amber repository for each recipe like this (anyway):

1. amberframework/recipe-react
2. amberframework/recipe-vue
3. amberframework/recipe-angular
4. amberframework/recipe-modular
5. amberframework/recipe-graphql
6. amberframework/recipe-mongo

Update: :point_up: this is a really bad idea, so no way :no_entry: See my comment below :point_down: :sunglasses:

[faustinoaq]

Hey @damianham @drujensen I got a new great Idea. I think We can keep just one repo amberframework/recipes for the source code, and use the release page to manage .zip files, versions and tags

what happens when Amber has a breaking change? Do we need to update all the recipes to avoid broken templates? How is versioning going to be handled with zip files?

We already can manage versions of .zip files if we use github releases, so:

1. Publish a new release, (ex: tag v0.1.0 and release v0.1.0) then
2. Upload manually (or automated) your .zip recipe in the release page for this version (tag)
3. Finally We can fetch from a specific release of amberframework/recipes a .zip file corresponding to a recipe. So the recipes will be versioned by the recipes repo itself.
4. One recipe doesn't break another recipe because we can modify the uploaded files for each release, so if we publish the release v0.1.0 of amberframework/recipes then we can upload in this recipe some recipes like react_recipe.zip, vue_recipe.zip, modular_recipe.zip.
5. What happens if I want to publish a new version of react_recipe.zip? Res: then we release v0.1.1 for amberframework/recipes
6. And if I want to publish a new version of vue_recipe.zip after react_recipe.zip was published? Res: then we upload the vue_recipe.zip inside the v0.1.1 release page already created.

I don't see any drawback for this :sweat_smile:

@damianham I'm already working in a bunch of PR, if you want I can try to change the current recipe feature to fetch a .zip file from the releases page according to an specified version :wink:

With this logic

1. amber new blog -r react: Uses the latest version of react_recipe.zip downloaded from amberframework/recipes/releases page
2. amber new blog -r react@0.1.0: Download the react_recipe.zip file from release page at version 0.1.0

WDYT? I think this approach is perfect :heart_eyes:

Edit: :no_good_man: I spoke too soon, see comments below :point_down:

[faustinoaq]

And we can manage recipes plugins by fetching something like react_plugin.zip in the same release page :wink:

Edit: :no_good_man: I spoke too soon, see comments below :point_down:

[faustinoaq]

And the repository structure can be the same, each folder can be zipped as:

(plugins not implemented yet)

(git repo with folders of source code)
default       # => zip files without author official plugins and recipes
├── auth/     # auth_plugin.zip              `amber new myapp -p auth` 
├── webpack/  # webpack_plugin.zip           `amber new myapp -p webpack`
└── api/      # api_recipe                   `amber new myapp -r api`
damianham/
├── modular/  # damianham_modular_recipe.zip `amber new myapp -r damianham_modular`
├── react/    # damianham_react_recipe.zip   `amber new myapp -r damianham_react`
└── default/  # damianham_default_recipe.zip `amber new myapp -r damianham_default`
faustinoaq/
├── vue/      # faustinoaq_vue_recipe.zip    `amber new myapp -r faustinoaq_vue` 
└── emberjs/  # faustinoaq_ember_recipe.zip  `amber new myapp -r faustinoaq_ember` 
drujensen/
├── graphql/  # drujensen_grapql_recipe.zip  `amber new myapp -r drujensen_grapql`
└── mongodb/  # drujensen_mongodb_recipe.zip `amber new myapp -r drujensen_mongodb`

Finally you can upload the .zip files you want in your specific release of amberframework/recipes

I traveled to the future and I saw this :man_astronaut: :rocket:

amberframework/recipes/releases (the release GitHub page, NOT a folder)
├── 0.0.1: damianham_modular_recipe.zip, damianham_default_recipe.zip [added] (today)
├── 0.0.1: api_recipe.zip, webpack_recipe.zip [added] (tomorrow)
├── 0.0.2: damianham_modular_recipe.zip       [added] (in two days)
├── 0.0.3: damianham_modular_recipe.zip       [added] (in one week)
├── 0.0.2: api_recipe.zip                     [added] (in two weeks)
├── 0.0.1: damianham_modular_react.zip        [added] (in three weeks)
├── 0.0.1: faustinoaq_vue_recipe.zip          [added] (in one month)
└── and so on...

And we got this:

amberframework/recipes/releases (the release GitHub page, NOT a folder)
├── 0.0.1
│   ├── damianham_modular_recipe.zip
│   ├── damianham_default_recipe.zip
│   ├── damianham_react_recipe.zip
│   ├── api_recipe.zip
│   └── webpack_recipe.zip
├── 0.0.2
│   ├── damianham_modular_recipe.zip
│   ├── damianham_default_recipe.zip
│   ├── api_recipe.zip
│   └── webpack_recipe.zip
└── 0.0.3
    └── damianham_modular_recipe.zip 

So, please feel free to share your comments :100: :+1:

[damianham]

@drujensen a recipe has a version of amber given in the app template for the recipe so even if amber has a breaking change, if a user generates an app with a recipe that has not been updated they will use the older version of amber given in the recipe shard.yml file. A recipe is a flavour of amber and the versions of shards are encapsulated in the recipe shard.yml. Of course recipes should be updated over time to use newer versions of amber.

@faustinoaq I don't see the need to use the word recipe in any file name - that is at the top of the tree anyway so everything within the folder tree is a recipe.

If we keep the 'blessed' recipes as code in the repo then the blessed recipes are curated by the team like any other code. Thus all recipes will have the same release tag corresponding to the tag of the amberframework/recipes repo. I think the main thing we have to work out is the mechanism to package a recipe into a zip for distribution. The recipes could be stored in deeply nested folders and we need a CI mechanism to package each recipe into a zip from a possibly deeply nested path into a corresponding deeply nested path, e.g

amberframework/recipes/   CI packages to zips in amberframework.org/recipes
├── basic
│   ├── modular/   -> basic/modular.zip   #  amber new myapp -r basic/modular
│   ├── default/   -> basic/default.zip   #  amber new myapp -r basic/default
│   ├── api/ -> basic/api.zip    #  amber new myapp -r basic/api
├── react
│   ├── preact/modular/ -> react/preact/modular.zip # amber new myapp -r react/preact/modular
│   ├── redux/   -> react/redux.zip  # amber new myapp -r react/redux
├── angular
 |  ├── modular/  -> angular/modular.zip  # amber new myapp -r angular/modular
├── plugins
│   ├── auth/ -> plugins/auth.zip  # amber g plugin auth
│   ├── parcel/ ->plugins/parcel.zip  # amber g plugin parcel
│   ├── browserify/ -> plugins/browserify.zip # amber g plugin browserify
│   └── webpack/ -> plugins/webpack.zip  # amber g plugin webpack

Would a crystal script that is driven by a yaml file be a good option for packaging the recipes folders into zips ? i.e. each leaf in the yaml file denotes a folder with a recipe, or just a bash script with a list of packaging and distribution commands?

Do we drop the contributor name folders and the team curates all recipes ?

[ChangJoo-Park]

wow.. I need Vue.js 👍

[ChangJoo-Park]

@faustinoaq What do you think about rails's webpacker? It just interface between webpack and rails. if gives only webpack (or something bundler) to user, after situations are depends on user's choice.

[damianham]

@ChangJoo-Park I think what we are going to do with regards to other build systems is provide them in plugin recipes so the user will have the choice of using whatever build system they prefer.

[faustinoaq]

A recipe is a flavour of amber and the versions of shards are encapsulated in the recipe shard.yml

@damianham You're right :+1:

I don't see the need to use the word recipe in any file name - that is at the top of the tree anyway so everything within the folder tree is a recipe.

Yeah, no problem, was just an idea :sweat_smile:

The recipes could be stored in deeply nested folders and we need a CI mechanism to package each recipe into a zip from a possibly deeply nested path into a corresponding deeply nested path, e.g

Why Can't we use the release page to store .zip files? as I commented here: https://github.com/amberframework/amber/issues/750#issuecomment-381016283

Travis and other CI, already provide nice settings to publish things on release page

The release page is this one: https://github.com/amberframework/recipes/releases

So, we don't need to store .zip files in the repo but only source code, then we can use the releases page to store the .zip files for an specified release. So if someone want to use the recipe basic_modular.zip@0.0.3 that uses amber v0.8.0, then is possible downloading the right version v0.0.3 in the release page. Otherwise if we don't use the release page to manage versions, would be imposible for someone to use the recipe basic_modular.zip@0.0.1 that comes amber v0.7.2 because the .zip files was updated and no old version is available.

@damianham I hope you can understand me :sweat_smile:

amberframework/recipes/   CI packages to zips in amberframework.org/recipes
├── basic
│   ├── modular/        -> basic_modular.zip        # amber new myapp -r basic/modular
│   ├── default/        -> basic_default.zip        # amber new myapp -r basic/default
│   └── api/            -> basic_api.zip            # amber new myapp -r basic/api
├── react
│   ├── preact/modular/ -> react_preact_modular.zip # amber new myapp -r react/preact/modular
│   └── redux/          -> react_redux.zip          # amber new myapp -r react/redux
├── angular
|   └── modular/        -> angular_modular.zip      # amber new myapp -r angular/modular
└── plugins
    ├── auth/           -> plugins_auth.zip         # amber g plugin auth
    ├── parcel/         -> plugins_parcel.zip       # amber g plugin parcel
    ├── browserify/     -> plugins_browserify.zip   # amber g plugin browserify
    └── webpack/        -> plugins_webpack.zip      # amber g plugin webpack

By example all these files can be stored in different releases in the release page

Versions available are according to releases in the releases page

      Recipe package     |     Versions available 
------------------------ | --------------------------
basic_modular.zip        |     0.0.1, 0.0.2, 0.0.3    
basic_default.zip        |     0.0.1     
basic_api.zip            |     0.0.1
react_preact_modular.zip |     0.0.1, 0.0.2
react_redux.zip          |     0.0.1 
angular_modular.zip      |     0.0.1, 0.0.2, 0.0.3      
plugins_auth.zip         |     0.0.1, 0.0.2
plugins_parcel.zip       |     0.0.1, 0.0.2, 0.0.3     
plugins_browserify.zip   |     0.0.1, 0.0.2
plugins_webpack.zip      |     0.0.1

Of course recipes should be updated over time to use newer versions of amber.

That's why I say we should use the releases page to manage and storage the .zip files and their versions.

Edit: :no_good_man: I spoke too soon, see comments below :point_down:

[eliasjpr]

@faustinoaq I think this idea is great and it resembles what homebrew already does. Now I think that we should not limit it to be in one single repository, and I agree with @drujensen we would have to build a package manager when crystal shards already exists.

One flaw that I see is with your approach tis hat if someone works hard on a recipe and wants to use his own recipe in his project he will have to wait until it gets merged into the recipe repo and release, (we experienced this when amber was a brew tap), this is why homebrew had to come up with brew tap to allow people install their own formulas that has not yet been merge into the homebrew formulas repo

By having one repo to manage all recipes it adds a lot of complexity. We should treat recipes as shards:

1. Generate Project Skeleton based on recipe:

Amber g {project} -r https://github.com/some-recipe/name
or 
Amber g {project} -r some-recipe/name # look up on github in the background
or 
Amber g {project} -r https://my-git-server.com/some-recipe/name

The above will generate the project skeleton based on the recipe

2. Then run shards install this will install the shards including the recipe
3. amber g model will use recipe by default for generation since we have the files locally cached

This can then generate the project based on that recipe, and I see the following benefits:

1. Leverage on crystal shards for caching and downloading etc,
2. Removes all the maintenance complexity of having a recipe repo
3. People are free to install from any git server
4. Recipe repo can continue to follow the module structure established by @damianham (zip files)
5. We can leave the recipe repo for Amber maintained recipes

[ChangJoo-Park]

@damianham I saw recipe repo. It is good approach :)

[drujensen]

If we keep the 'blessed' recipes as code in the repo then the blessed recipes are curated by the team like any other code. Thus all recipes will have the same release tag corresponding to the tag of the amberframework/recipes repo. I think the main thing we have to work out is the mechanism to package a recipe into a zip for distribution.

@damianham Right, we are building another packaging manager. I just wanted to point out the long term consequences of this approach. I personally think this is going to be overwhelming to maintain all the different possible recipes per release and maintaining versioning and packaging, but it looks like this has been blessed by the rest of the amber team, so I will help anyway I can.

@eliasjpr you are correct about a central repository. This is going to get very burdensome very quickly.

[robacarp]

I have a hard time imagining what it would look like to use shards to manage recipes, but as long as we aren’t tracking compressed zips in git.

Would an amber recipe be fetched with a shard install? Perhaps as a development dependency? That sounds like something worth exploring. Unfortunately I don’t think shards don’t have the ability to track more than one dependency in a repository.

Why not just publish the version of a recipe that tracks 1:1 with amber? No code is going to be generated in anything but a dev environment (no version “lock” is needed), and any bugs for a specific version should just be re released under that same version.

Interoperability between recipes becomes a simpler issue too.

There might be some corner cases where it gets weird but I’d prefer to explicitly not deal with version numbers and have a simpiler ecosystem. Perhaps I’m being over simplistic but it seems like that would make everything plain about what recipe tracks what amber version.

[drujensen]

@robacarp I was envisioning something more inline with what @eliasjpr was:

amber new blog -r drujensen/amber-vuejs-recipe

This would point to the shard repo that would include the shard in the dependencies and then when it built bin/amber, it would include the generators. The templates would be downloaded to the lib directory just like any other shard.

We could move all our default generators to amberframework/default-recipe and -r by default would point to that shard.

Developers could simply fork the default-recipe to build their own.

[robacarp]

Okay, that sounds fine to me. The idea of using shards initially freaked me out but I think I overreacted.

Does that leave space to put more than one recipe in a repository? I'd really like to have that.

It's worth mentioning that because of the way that the recipe feature uses liquid, the templates don't have to be included in the build but can be referenced later.

[drujensen]

It's up to the recipe repository what commands are overwritten or added. You could add multiple recipes to a project just by adding their shard as a dependency.

I haven't looked into liquid yet or how these recipes work but good to know it doesn't require compiling them in. I can see how that would make shards less necessary.

I am curious, do we lose compile time checking when we use liquid? If so, we have lost one of the major features of the language imo.

[faustinoaq]

Is nice to see a lot of good comments discussing this issue :smile: :+1:

Currently we have merged on master the recipe feature with support for multiple recipes (.zip files only, we can add source code, though) per repo, so we should coordinate with @damianham what need to be implemented, maybe we can write an status:rfc about this.

I also got a new idea :sweat_smile:

I think the current implementation of recipe feature already cover all our use cases :tada:

let me explain... :wink:

If we already have multiples recipes per repo and I think we can keep the .zip files and source code like @damianham said here https://github.com/amberframework/amber/issues/750#issuecomment-381059716 , then you can use just one .zip file per repo (shard) and you got a nice one (1) file - one (1) management, just like shards.

Optionally like @robacarp said, we can store multiples .zip files per repo, and use external repos, modifying the already existing recipe_source entry in .amber.yml

Finally, we can continuing managing the amberframework/recipes repo with blessed shards like @damianham is doing right now where we have many (*) recipes - one (1) official repo.

@damianham Thank you so much for implementing this feature, you did this really well!!! :100:

[damianham]

@faustinoaq I can see the advantage of storing older versions of recipes for the odd case someone wants to use an older version of amber so storing the zips on the release page would be good for that. I don't understand how the recipe zip files would have different version numbers. If the recipes repo is tagged then all recipes would have the tag so they would all be at the same release. How do we get the latest recipe from the releases packages without specifying a version number ?

@eliasjpr you can use a recipe from a file system path and you can also specify a recipe source in the app's .amber.yml config file so it is easy to work on a recipe and then use it without waiting for it to be merged into the recipes repo. There is one improvement that we could make and that is to allow a URL to be given as the source of the recipe. At the moment it is either a curated recipe in the recipes repo or a file system path only. It should be simple enough if the given recipe name starts with 'http' then download the URL as the recipe.

@drujensen I am not convinced that we need to track versions and dependencies with recipes as the dependencies are specified in the recipe so if you generate an app with a recipe which uses specific version numbers of shards then shards install will install the shard versions according to the version numbers given in the app/shard.yml file. I think that you are thinking that recipes would be installed as a shard into a lib folder. Recipes are external to either amber or a generated application and are not built in to the binary like the existing templates so yes it doesn't require compiling them in and so shards are less necessary. Since it is just copying templates to a generated artifact no end user application code is executed.

A zip file is a super simple delivery method, all we need to do is download and extract the zip using crystal built-in API tools.

If we used another mechanism e.g. shards would it be as simple as that ? I don't think we are talking about creating a new packaging manager, it is simply a process to convert a folder into a zip file to release the recipes as individual zips - and that is probably just a few lines in a bash shell script to find all the app folders (all recipes that are not plugins have an app folder) and create a zip file from the folder contents. Plugins will most likely not have an app folder because they would be applied to an already generated app.

I think at the moment it is a simple system - and we should probably aim to keep it as simple as possible as long as it satisfies the requirements.

[robacarp]

do we lose compile time checking when we use liquid?

Yes, well, sort of. But liquid is used for code generation, not serving up the generated code. The liquid context is compile time checked, but the template itself is not. The output of a liquid recipe is ecr or slang, and the application is compile time checked as always.

[faustinoaq]

@drujensen I think recipes are just amber projects with customized things :sweat_smile:

How do we get the latest recipe from the releases packages without specifying a version number ?

@damianham Oh, never mind, I just realize mi idea is not possible, because I'm tagging all recipes at once, so, even if I just upload the specified files, this will be a head ache to fetch the latest version for a recipe (unless we add versions to the recipe itself). I think a simpler option is storing old recipes and change the name like basic_modular_0.0.1.zip :sweat_smile:

[faustinoaq]

Recipes are external to either amber or a generated application and are not built in to the binary like the existing templates so yes it doesn't require compiling them in and so shards are less necessary. Since it is just copying templates to a generated artifact no end user application code is executed.

@damianham You're right, I think we had some kind of misunderstanding before :sweat_smile:

[faustinoaq]

I think at the moment it is a simple system - and we should probably aim to keep it as simple as possible as long as it satisfies the requirements.

yep, I just realize of that. After reviewing all these comments and testing the recipe feature, I think the current recipe implementation is very nice and simple :+1:

[faustinoaq]

So, The things are missing on the recipe implementation are:

1. plugins (this are a bit different because they generate things inside the project)
2. source code stored beside recipes, (not to download but for maintainability)
3. default recipes: api, react in default/<recipe>.zip folder (currently we're using damianham/<recipe>.zip recipes)

^^^ @amberframework/contributors

[damianham]

If we can agree on a location for the recipe zip files for distribution and the method to publish the zip files to the distribution location I will start to migrate the recipes to the new schema as we discussed here, e.g. top level folders according to a theme with additional default and plugins folders.

[eliasjpr]

@damianham @amberframework/contributors I believe this has been addressed already, if yes let's close this issue

[faustinoaq]

@eliasjpr Yeah, now recipes are tested and deployed automatically, we can try current implementation.

@damianham Maybe in the future we can suggest a new approach like using shards or hierarchical recipes :+1:

[damianham]

@faustinoaq yes we can shift to a shard based implementation. I have implemented a shard based recipe fetcher but I still need to work out how to integrate plugins, other generators (e.g. auth) and also hierarchical recipes. At the moment I bootstrap an app recipe from a github shard for an app called myapp into myapp/.recipe and return the location of the template in that folder. So I still need to work out how an auth generator plugin can be added and also how hierarchical recipes will work. Just a bit busy with my projects at the moment.