Open sisp opened 1 year ago
Seems interesting indeed. As you know, I always try to keep it as simple as possible here. So I've been thinking about a possible solution for your problem that doesn't require a lot of refactoring.
Quoting from https://copier.readthedocs.io/en/stable/#basic-concepts:
Copier doesn't replace the DRY principle... but sometimes you simply can't be DRY and you need a DRYing machine...
So maybe it's time for Copier to DRY itself!
Have you thought about having a metatemplate? A template that generates templates? This way:
copier update
... but from the child template itself.I have made a proof of concept in https://github.com/yajo/copierception. According to my tests, the only problem is that the child template doesn't get the copier.yaml
file created as it should. This is because it's explicitly skipped in Copier for obvious reasons. But we can safely no skip it if there's a _subdirectory
specified, because in that case anything inside of it is put there on purpose.
Seems like an easy solution. What do you think?
That's a great idea! :clap: Also, thanks for the proof of concept! :pray: (And great name copierception
and meme! 😂)
I think this could work indeed. :tada: I've created another proof of concept with a chain of meta-templates to check whether/how this could be done:
flowchart TD
copier-python-meta --> copier-python
copier-python --> python-project
copier-python-meta --> copier-python-django-meta
copier-python-django-meta --> copier-python-django
copier-python-django --> python-django-project
For this to work, I think the workflow looks like this:
copier-python-meta
The filesystem layout of a (simplified) generic Python meta-template might look like this:
copier-python-meta
├── copier.yml
└── meta-template
├── [[ _copier_conf.answers_file ]].j2
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── {{ project_name }}
│ └── __init__.py
└── pyproject.toml.jinja
As you mentioned, copier.yml
needs to be removed from the _exclude
list in /copier.yml
.
copier-python
Using copier-python-meta
, a regular Copier template copier-python
for generic Python projects can be generated:
$ copier copier-python-meta copier-python
No git tags found in template; using HEAD as ref
Copying from template version 0.0.0.post1.dev0+b21a356
create .
create template
create template/{{ project_name }}
create template/{{ project_name }}/__init__.py
create template/pyproject.toml.jinja
create template/{{ _copier_conf.answers_file }}.jinja
create .copier-answers.yml
create copier.yml
And the resulting filesystem layout of the copier-python
template looks like this:
copier-python
├── .copier-answers.yml
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── {{ project_name }}
│ └── __init__.py
└── pyproject.toml.jinja
Using this template, a generic Python project (e.g. python-project
) can be generated as usual.
copier-python-django-meta
Now, let's create a more specific meta-template for a (simplified) Python web project that uses Django based on copier-python-meta
.
The following filesystem layout needs to be created manually:
copier-python-django-meta
├── copier.yml
└── meta-template
└── [[ _copier_conf.answers_file ]].j2
Then, the content of the folder meta-template/
is further populated by the generated output of the copier-python-meta
template:
$ copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
No git tags found in template; using HEAD as ref
Copying from template version 0.0.0.post1.dev0+b21a356
identical .
create template
create template/{{ project_name }}
create template/{{ project_name }}/__init__.py
create template/pyproject.toml.jinja
create template/{{ _copier_conf.answers_file }}.jinja
create .copier-answers-meta.yml
create copier.yml
Now, just for the sake of this example, we could add the file meta-template/template/manage.py.jinja
, that Django uses, to exemplify that the child template is an extension of the copier-python(-meta)
template. The resulting filesystem layout looks like this:
copier-python-django-meta
├── copier.yml
└── meta-template
├── .copier-answers-meta.yml
├── [[ _copier_conf.answers_file ]].j2
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── {{ project_name }}
│ └── __init__.py
├── manage.py.jinja
└── pyproject.toml.jinja
Note the flag --answers-file .copier-answers-meta.yml
which is important to avoid a conflict between the answers file created by copier-python-meta
and the answers file [[ _copier_conf.answers_file ]].j2
that is rendered when generating the copier-python-django
template (see below). On the other hand, .copier-answers-meta.yml
shouldn't be copied into the generated child template copier-python-django
, so it needs to be excluded via _exclude
in /copier.yml
.
When the copier-python-meta
meta-template changes and we would like to update this meta-template accordingly, we would run the following command:
copier --answers-file .copier-answers-meta.yml update copier-python-django-meta/meta-template
Again, note the flag --answers-file .copier-answers-meta.yml
.
copier-python-django
Using copier-python-django-meta
, a regular Copier template copier-python-django
for Python web projects using Django can be generated:
$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref
Copying from template version 0.0.0.post1.dev0+04c5d90
create .
create template
create template/{{ project_name }}
create template/{{ project_name }}/__init__.py
create template/pyproject.toml.jinja
create template/manage.py.jinja
create template/{{ _copier_conf.answers_file }}.jinja
create .copier-answers.yml
create copier.yml
The resulting filesystem layout looks like this:
copier-python-django
├── .copier-answers.yml
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── manage.py.jinja
├── {{ project_name }}
│ └── __init__.py
└── pyproject.toml.jinja
Using this template, a Python web project using Django (e.g. python-django-project
) can be generated as usual.
WDYT, @yajo? If you agree with this extended proof of concept:
copier.yml
file (i.e. remove it from the _exclude
list) in a template when _subdirectory
is used and is not ""
or "."
?We could even create a template for meta-templates that generates this filesystem layout. 🤣
Note the flag
--answers-file .copier-answers-meta.yml
which is important to avoid a conflict between the answers file created bycopier-python-meta
and the answers file[[ _copier_conf.answers_file ]].j2
that is rendered when generating thecopier-python-django
template (see below). On the other hand,.copier-answers-meta.yml
shouldn't be copied into the generated child templatecopier-python-django
, so it needs to be excluded via_exclude
in/copier.yml
.
IIUC this isn't really a big problem because both the meta-template and the sub-template use _subdirectory
. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.
Do you think we should automatically allow copying a
copier.yml
file (i.e. remove it from the_exclude
list) in a template when_subdirectory
is used and is not""
or"."
?
Sure. Actually we should completely remove any default value if using a subdirectory.
Should we add a proper guide in the docs that describes this workflow?
Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.
IIUC this isn't really a big problem because both the meta-template and the sub-template use
_subdirectory
. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.
The conflict is between the answers file of the parent meta-template copier-python-meta
and the (templated) answers file of the child meta-template copier-python-django-meta
. If I ran
-copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
+copier copier-python-meta copier-python-django-meta/meta-template
then copier-python-django-meta
's filesystem layout would be
copier-python-django-meta
├── copier.yml
└── meta-template
- ├── .copier-answers-meta.yml
+ ├── .copier-answers.yml
├── [[ _copier_conf.answers_file ]].j2
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── {{ project_name }}
│ └── __init__.py
├── manage.py.jinja
└── pyproject.toml.jinja
and generating python-copier-django
using this meta-template would show a conflict for .copier-answers.yml
$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref
Copying from template version 0.0.0.post2.dev0+c422bf7
create .
create .copier-answers.yml
create template
create template/{{ project_name }}
create template/{{ project_name }}/__init__.py
create template/pyproject.toml.jinja
create template/manage.py.jinja
create template/{{ _copier_conf.answers_file }}.jinja
conflict .copier-answers.yml
overwrite .copier-answers.yml
create copier.yml
because meta-template/.copier-answers.yml
(which points to the version of the copier-python-meta
meta-template) and meta-template/[[ _copier_conf.answers_file ]].j2
after rendering have identical file names. It looks like meta-template/[[ _copier_conf.answers_file ]].j2
overwrites meta-template/.copier-answers.yml
, but this behavior is ambiguous even if its implementation is deterministic. Do you see my point?
I think it would be great if there was a way to avoid a custom answers file name for pointing to the parent meta-template, but I don't see one at the moment. Any idea?
Do you think we should automatically allow copying a
copier.yml
file (i.e. remove it from the_exclude
list) in a template when_subdirectory
is used and is not""
or"."
?Sure. Actually we should completely remove any default value if using a subdirectory.
:+1: I'd be happy to send a PR next week.
Should we add a proper guide in the docs that describes this workflow?
Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.
:+1: I'd be happy to write a draft and send a PR.
Perhaps a slightly better alternative to the --answers-file .copier-answers-meta.yml
flag in the command
copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
is
copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template
which
copier-python-meta
) to the root folder of the child meta-template (copier-python-django-meta
), which seems better to me.The resulting filesystem layout of copier-python-django-meta
then looks as follows:
copier-python-django-meta
├── .copier-answers.yml
├── copier.yml
└── meta-template
├── [[ _copier_conf.answers_file ]].j2
├── copier.yml
└── template
├── {{ _copier_conf.answers_file }}.jinja
├── {{ project_name }}
│ └── __init__.py
└── pyproject.toml.jinja
I'm trying to understand but I really don't get what's the problem of using just the default copier answers file.
According to my tests, after #941 was merged, the meta-template sample works as expected. Check out this shell session:
$ subtpl=$(mktemp -d)
$ dst=$(mktemp -d)
# Create a sub-template from the meta-template
$ copier -r 84e05b9ae447a4e36e9e678270af8cd3e0cad026 copy https://github.com/yajo/copierception $subtpl
🎤 python (bool)
Yes
🎤 pre_commit (bool)
Yes
Copying from template version 0.0.0.post2.dev0+84e05b9
identical .
create template
create template/{{ _copier_conf.answers_file }}.jinja
create template/{% if pre_commit %}.pre-commit.yaml{% endif %}
create template/pyproject.toml.jinja
create template/README.md.jinja
create copier.yaml
create .copier-answers.yml
$ cd $subtpl
$ git init
Inicializado repositorio Git vacío en /tmp/tmp.eC3Bv4Ijmq/.git/
$ git add -A
$ git commit -m 1
[main (commit-raíz) 6499725] 1
6 files changed, 55 insertions(+)
create mode 100644 .copier-answers.yml
create mode 100644 copier.yaml
create mode 100644 template/README.md.jinja
create mode 100644 template/pyproject.toml.jinja
create mode 100644 template/{% if pre_commit %}.pre-commit.yaml{% endif %}
create mode 100644 template/{{ _copier_conf.answers_file }}.jinja
# Generate a final project from the sub-template
$ copier copy $subtpl $dst
/nix/store/asr4mbhx1va7ha0kn3cvx4n0qlxmg8vk-python3.10-copier-7.1.0a0.dev20230221141200+nix-git-ab72507/lib/python3.10/site-packages/copier/vcs.py:155: ShallowCloneWarning: The repository '/tmp/tmp.eC3Bv4Ijmq' is a shallow clone, this might lead to unexpected failure or unusually high resource consumption.
warn(
No git tags found in template; using HEAD as ref
🎤 project_name
my project
🎤 build_system
poetry
Copying from template version 0.0.0.post1.dev0+6499725
identical .
create .copier-answers.yml
create pyproject.toml
create README.md
# Update the sub-template getting last changes from the meta-template
$ copier update
No git tags found in template; using HEAD as ref
Updating to template version 0.0.0.post3.dev0+fea6c75
No git tags found in template; using HEAD as ref
🎤 python (bool)
Yes
🎤 pre_commit (bool)
Yes
Copying from template version 0.0.0.post3.dev0+fea6c75
identical .
identical template
identical template/{{ _copier_conf.answers_file }}.jinja
identical template/{% if pre_commit %}.pre-commit.yaml{% endif %}
identical template/pyproject.toml.jinja
identical template/README.md.jinja
identical copier.yaml
conflict .copier-answers.yml
overwrite .copier-answers.yml
# Check that the sub-template was properly updated
$ git diff
diff --git a/.copier-answers.yml b/.copier-answers.yml
index bb3d46a..55ab783 100644
--- a/.copier-answers.yml
+++ b/.copier-answers.yml
@@ -1,5 +1,5 @@
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
-_commit: 84e05b9
+_commit: fea6c75
_src_path: https://github.com/yajo/copierception
pre_commit: true
python: true
Thanks for sticking with me and challenging me! :pray: I really appreciate your time invest into this discussion! :pray:
In your shell session example, you have only one meta-template (copierception
) from which you generate the template in $subtpl
from which you generate the project in $dst
. In this scenario, there's no problem.
But I'm considering an extended meta-template scenario in which there are two meta-templates: copier-python-meta
and copier-python-django-meta
where copier-python-django-meta
is based on the generated sub-template from copier-python-meta
(and possibly edited, which is fine because changes in the meta-template copier-python-meta
can be applied by the regular update mechanism). Only this scenario causes a conflict of the answers files.
If you go through the example workflow in https://github.com/copier-org/copier/issues/934#issuecomment-1406398277 and run
copier copier-python-meta copier-python-django-meta/meta-template
instead of
copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
or (as I later improved IMO)
copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template
then you'll see
$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref
Copying from template version 0.0.0.post2.dev0+c422bf7
...
conflict .copier-answers.yml
overwrite .copier-answers.yml
...
which is the result of copier-python-django-meta/meta-template/.copier-answers.yml
and copier-python-django-meta/meta-template/[[ _copier_conf.answers_file ]].j2
conflicting after rendering the latter.
I know it's a brain twist with the many meta-templates and sub-templates generated into other meta-templates. :rofl:
Can you reproduce the problem now? :crossed_fingers:
Haha I need a meta-brain to follow you! 🤭
Could you prototype something and give me a code snippet to reproduce it please? I guess it'll be easier to understand.
Sure, I'll prepare something next week. :slightly_smiling_face:
I've created an example with two meta-templates based on the example above:
copier-python-meta
is a simplified Python base meta-template. You can generate a regular template and a project from the regular template like this:
copier https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python
copier copier-python pyproj
Everything is fine, keep reading. :wink:
copier-python-django-meta
is a simplified Python Django meta-template whose child-template is a (minor) extension of the Python base meta-template's child-template. I have made two releases:
v1.0.0
contains a child-template that was initialized without the --answers-file
flag:
copier https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python-django-meta/meta-template/
So the answers file related to the copier-python-meta
meta-template is meta-template/.copier-answers.yml
. Now, when you generate the regular template, you'll see the following conflict:
$ copier --r v1.0.0 https://gitlab.com/copier-meta-templates-example/copier-python-django-meta copier-python-django
Copying from template version 1.0.0
...
conflict .copier-answers.yml
overwrite .copier-answers.yml
v1.0.1
contains a child-template that was initialized with the --answers-file
flag:
copier --answers-file ../.copier-answers.yml https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python-django-meta/meta-template/
So the answers file related to the copier-python-meta
meta-template is .copier-answers.yml
. Now, when you generate the regular template, you'll see no conflict:
$ copier --r v1.0.1 https://gitlab.com/copier-meta-templates-example/copier-python-django-meta copier-python-django
Copying from template version 1.0.1
create .
create template
create template/{{ project_name }}
create template/{{ project_name }}/__init__.py
create template/pyproject.toml.jinja
create template/manage.py.jinja
create template/{{ _copier_conf.answers_file }}.jinja
create template/README.md.jinja
create .copier-answers.yml
Do you see the difference and why passing the --answers-file
flag (and IMO ideally with the value ../.copier-answers.yml
) is needed (or rather recommended - it works despite overwriting but is sensitive to which file takes precedence)?
This would be useful for FastAPI projects as well. :+1:
Any alternative way to achieve this before this idea gets in?
@Kludex Do you mean a more native/direct way without the meta-template "detour"?
I didn't get your question, but what I want is to be able to create the full structure, and then "components" on subdirectories. Those components can be different copier projects...
IIUC you're going too deep @sisp. As it always has been, with Copier 1 template = 1 repo. If you want to use several levels of meta-templates, each one of them must produce a template by itself, which should be pushed to a separate repo.
So, a meta template of 3 levels would be:
copier.yml
<< _copier_conf.answers_file >>.jinja1
top-template/
copier.yml
[[ _copier_conf.answers_file ]].jinja2
middle-template/
copier.yml
(( _copier_conf.answers_file )).jinja1
final-template/
{{ _copier_conf.answers_file }}.jinja
launch_rocket.py.jinja
Thus you copy it like this:
# Use top-template to generate a middle one and publish it
copier copy https://gitlab.com/example/top-template ./middle
cd middle
git init
git add -A
git commit -m 'middle template'
git remote add origin https://gitlab.com/example/middle-template
git push -u origin main
cd ..
# Use middle-template to generate the final one and publish it
copier copy https://gitlab.com/example/middle-template ./final
cd final
git init
git add -A
git commit -m 'final template'
git remote add origin https://gitlab.com/example/final-template
git push -u origin main
cd ..
# Use final-template to generate a... real project!
copier copy https://gitlab.com/example/final-template ./real
cd real
git init
git add -A
git commit -m 'build: init from template'
git remote add origin https://gitlab.com/example/real
git push -u origin main
./launch_rocket.py
In such scenario, these will always be true:
.copier-answers.yml
provides answers to the parent templatetpl/{{ _copier_conf.answers_file }}.jinja
(or equivalent syntax) instructs copier where to put answers for the next meta-generation.Reaching this point probably means that you're doing something wrong. But it'd work as long as:
_subdirectory
option.I didn't get your question, but what I want is to be able to create the full structure, and then "components" on subdirectories. Those components can be different copier projects...
@Kludex I don't think I understand what you'd like to achieve exactly. Could you provide a simple example?
@yajo I'm not trying to build a meta-template with 3 levels but rather two meta-templates that each generate a regular template but one meta-template extends the other. This is important because it enables meta-templates to extend one another while being developed by independent authors in independent projects. I even think a meta-template with 3 levels covers a different use case than what I'm suggesting.
It seems it's difficult to convey the approach here. Do you think it would make sense to have a brief voice chat about it some time?
@yajo and I had a video chat today, and after a while he suggested to simply fork a parent template and update the child template from the parent template using git merge
/git pull
, so neither a meta-template nor a new Copier feature would be needed. Indeed, this seems to work, but it requires keeping the parent template's history in the child template, otherwise Git isn't able to determine the correct patch. This may be a bit confusing though because it may include commits like "docs: Update changelog for version 0.4.0" which, in this case, contains a reference to a release version of the parent template which has no meaning in the child template. Also, it might make sense to ignore some files in the parent template like CHANGELOG.md
because the child template would always have its own changelog. WDYT, @yajo?
I think a kind of squash-merge approach might be better. But then, since the parent template's history would not be present in the child template's repo, we can't simply perform a pull/merge from the parent template's repo because Git can't determine the correct patch. That's why we'd need to compute the patch manually which means we'd need to keep a record of the parent template's version on which the child template is built. The manual workflow might look like this:
Create a child template based on a parent template, e.g. https://github.com/pawamoy/copier-poetry:
Clone the repo into a folder named after the new project, e.g. copier-django
, and include only the tagged commit:
git clone --depth 1 --branch 0.3.2 https://github.com/pawamoy/copier-poetry.git copier-django
Reinitialize the Git repository to delete its history, tags, etc., and make a new initial commit:
$ rm -rf .git
$ git init
$ git add .
$ git commit -m "chore: initialize with gh:pawamoy/copier-poetry#0.3.2"
Extend the template according to your needs, commit the changes, and make a release:
# Make changes, and then ...
$ git add .
$ git commit
$ git tag v0.1.0
When the parent template evolves, e.g. 0.4.0
gets released, apply the changes to the child template:
Compute the patch for 0.3.2 -> 0.4.0
:
$ tmpdir=$(mktemp -d)
$ git clone --depth 1 https://github.com/pawamoy/copier-poetry.git $tmpdir
$ pushd $tmpdir
$ git fetch --no-tags origin tag 0.3.2 0.4.0
$ git diff 0.3.2 0.4.0 > 0.3.2-0.4.0.patch
$ popd
Apply the patch to the child template:
git apply -3 $tmpdir/0.3.2-0.4.0.patch
Resolve any potential merge conflicts and commit the update:
git commit -m "chore: update to template gh:pawamoy/copier-poetry#0.4.0"
This feels like something Copier should be able to do for us. So Copier could offer the following new commands:
copier template create [-r,--vcs-ref] parent_template_src child_template_destination_path
copier template update [-r,--vcs-ref] child_template_destination_path
And as mentioned above, we'd need to keep a record of the parent template's version (and URL, actually) on which the child template is built. Copier might keep this information in a file like .copier-template.yml
:
src: https://github.com/pawamoy/copier-poetry.git
ref: 0.3.2 # commit hash or tag name
This extension would be non-intrusive because (a) self-contained templates would remain unchanged, only child templates would contain this extra file, and (b) any existing template could serve as a parent template out of the box. WDYT, @yajo?
There's one legal aspect I'm not sure about though. Is it legally allowed (with regard to license and copyright) to effectively squash-merge a parent template (update)? Would it suffice to keep the parent template's license and add the child template's author as an additional copyright holder? In the fork approach, the commit history of the parent template would also exist in the child template's repo, so there is proper attribution of the changes coming from the parent template. When they are squashed, this information is gone. Does anybody know? @yajo @pawamoy
This is another long comment, @yajo. Sorry ... :see_no_evil: :smile:
I think this is a different thing than copier. 😅 Let me explain.
Copier is made to bootstrap projects based on a template and keep them updated. In the Unix philosophy, that's the one thing it does well. 🎯
What you're looking for, though, is a replacement for git pull --squash
.
Let's say we solve it in Copier. I think that all you want is a copy mode
where rendering is disabled. We could solve that with a --render=no
flag
for both copy and update. At first it seems easy, but in reality it goes
against all that copier is.
For example, Copier would need to ignore the copier.yml file in the template, because its instructions can be templated. But we're not rendering, so chances are that the raw copier.yml produces failures. But if we ignore that file, how are you supposed to say that you want to skip CHANGELOG.md if it exists? How are we supposed to infer the default path for the answers file?
The answers file would be simplified because it would contain no answers.
However it still needs the src url, commit and a special new _render: false
answer. However, without rendering, how are we supposed to convert
{{_copier_conf.answers_file}}.jinja
into the answers file?
Even if fixing #983 as has been discussed there, it still makes no sense without reading the copier.yml file. And if we managed somehow to read it, many settings would become confusing. For example, we don't want to use the subdirectory setting, even if found, because we want to clone the full repo always. But that would change the semantics of the template manager internally completely.
Also, why would we have to bother any upstream repo about having a copier.yml file to better support forks? Git pull works just fine with any git repo regardless of its content. Why would a replacement tool be worse than that? This tool should work entirely on the side of the fork, without having to ask for anything upstream. After all, it's the fork who cares about i.e. the CHANGELOG problem, and you could even decide to solve it with a CHANGELOG_DOWNSTREAM.md file. Upstream shouldn't care annoy any of those.
Why would we need copier's complex update diff algorithm at all? There's no rendering, so there's no need to replay last render, right? But then, why do we even call the answers file like that, if it has no answers?
And the list of nonsense (for copier) keeps growing if you think about it. But I think we have enough...
So, my friend, if git pull is still not enough for you, indeed you need something else. But it seems to me that something is not copier. 😄
It's more close to mrchef, but still it isn't the same. I think it's a totally new tool (or maybe it exists already? 🧐).
There's one legal aspect I'm not sure about though. Is it legally allowed
(with regard to license and copyright) to effectively squash-merge a parent template (update)? Would it suffice to keep the parent template's license and add the child template's author as an additional copyright holder? In the fork approach, the commit history of the parent template would also exist in the child template's repo, so there is proper attribution of the changes coming from the parent template. When they are squashed, this information is gone. Does anybody know? @yajo https://github.com/yajo @pawamoy https://github.com/pawamoy
AFAIK FLOSS licenses cover the software source and binary distributions, but not the way it's built or credited. It would be a concern for the user of your new tool I think, not for you. Anyways, IANAL.
Message ID: @.***>
You're not the first one with these needs:
Is your feature request related to a problem? Please describe.
I'd like to create a Copier base template (e.g. a general Python project template) and Copier child templates (e.g. a Python web app project template using FastAPI, a Python ML library template using PyTorch, etc.). The Copier child templates would mostly extend the Copier base template (e.g. add new files, add content to existing files from the Copier base template) but might also overwrite files/folders in the Copier base template (e.g. modify content in existing files from the Copier base template) or even delete files from the Copier base template (although this might be a rare case). Naively, I could create independent Copier templates, but then maintenance wouldn't scale because I'd have to sync the common parts manually. Instead, I would like to update a Copier child template with the changes of its Copier base template in the same way as I can update a generated project with the changes of its associated Copier template.
Describe the solution you'd like
I know this topic has been discussed a few times already (#79, #402, #416), but I haven't found a satisfactory solution and came up with a (to my knowledge) new approach that I'd like to suggest and discuss as it doesn't seem possible out of the box.
To clarify possibly confusing terminology upfront, I'd like to explicitly distinguish between Copier templates (i.e. project templates) and Jinja templates (i.e. content templates used in text files or file/folder names).
So far, I've seen this topic under the term "template inheritance" which (to me) sounds similar to the idea of, e.g., OverlayFS where multiple filesystem sources are merged, possibly enhanced with Jinja inheritance such that an overwritten file may actually be an extension of the base file rather than a complete replacement. I believe #79 tried to implement such an approach, but it was not merged for apparently good reason.
I've been giving this problem some thought for a while and believe that the word "inheritance" might be misleading. Instead of trying to inherit from a Copier base template, I think a Copier child template should be a (modified) copy of the Copier base template containing a reference to the Copier base template similar to how a generated project has a reference to its corresponding Copier template in
.copier-answers.yml
. Then, a very similar update mechanism as for updating a generated project could be used to update a Copier child template with the changes of its associated Copier base template:This approach would offer significant flexibility for customizing a Copier child template because anything can be changed without requiring the developer of the Copier base template to anticipate extension points (i.e. Jinja blocks).
In contrast to generating projects from a Copier template, generating a Copier child template doesn't involve a questionnaire or Jinja templating. Currently, I think only the following settings (from
copier.yml
) remain relevant:answers_file
(But since there are no questions to ask for generating Copier child templates, this setting might be renamed or removed entirely and a hardcoded file name might be used instead.)migrations
min_copier_version
skip_if_exists
tasks
So perhaps a new file, e.g.
copier-template.yml
, should be introduced where the settings for generating Copier child templates are put. The Git rev (_commit: <rev>
) of the Copier base template might be written to a file like.copier-extension.yml
(or a different file name; subject to discussion).Let's make this idea more concrete.
The filesystem layout of a Copier base template might look like this (with
_subdirectory: template
incopier.yml
):And the filesystem layout of a Copier child template (which itself could serve as a Copier base template again) might look like this:
In order to speed-up the adoption of this idea, it might be possible to make
copier-template.yml
optional and assume sane defaults (i.e. no migrations, no tasks, ...) when this file is not present.Finally, two new CLI subcommands would need to be added to the Copier CLI, one for generating a new Copier child template from a Copier base template and one for updating an existing Copier child template from its associated Copier base template:
Describe alternatives you've considered
Both YAML includes of several
copier.yml
files and appyling multiple templates to the same subproject don't solve this problem.I also thought about using Git submodules to include a Copier base template repository in a Copier child template repository, but this approach doesn't work either for several reasons:
Additional context
I think it would be a huge win for Copier to support Copier template hierarchies. If others agree with this idea or we refine it to a promising proposal, I'd be happy to work on a draft PR.