Created new file after copier update seems to be deleted

[nchal]

Describe the problem

After doing a copier update, I cannot commit the incoming changes because the files seems to be deleted

Template

# Configuration
# ----------------------
_envops:
  lstrip_blocks: false
  trim_blocks: false
  keep_trailing_newline: true

_exclude:
  - ".git/**"
  - ".releaserc.yml"
  - "copier.yml"
  - "tests"
  - "tests/**"

_skip_if_exists:
  - "cleyrop/*"

# Questions
# ----------------------
project_name:
  type: str
  help: What is the project name?

initial_codeowners:
  type: str
  help: Who are the codeowners use @gitlab_usernames space limited. If unsure, just use your @gitlab username, including the @ sign.

certmanager_issuer:
  type: str
  choices:
    - letsencrypt
    - customCA

external_oauth:
  type: bool
  default: false
  choices:
    - true
    - false

To Reproduce

copier --vcs-ref=ci/test_remove_copier_task update

Logs

Copying from template version 4.0.0b15.post1.dev0+4e97db3
 identical  .
 identical  .gitleaks.toml
 identical  cleyrop
 identical  cleyrop/README.md
 identical  .tool-versions
 identical  .pre-commit-config.yaml
 identical  .checkov.yaml
 identical  scripts
 identical  scripts/prepare-dev-env.sh
 identical  scripts/update-asdf-version-variables.sh
 identical  scripts/install-asdf-plugins.sh
  conflict  .copier-version
 Overwrite .copier-version? [Y/n]
 identical  .git
 identical  docs
    create  docs/How_to_deal_with_Altas_as_hemera_developer.md
    create  docs/Global_Architecture_Overview.md
    create  docs/How_to_use_skeleton_as_hemera_developer.md
 identical  docs/resources
 identical  docs/resources/global_architecture_overview.drawio.png
    create  docs/resources/global_architecture_overview.drawio
 identical  docs/resources/change_cleyrop_chart_version_in_hemera_developers_env.png
 identical  docs/resources/retrieve_chart_version_in_job.png
 identical  docs/resources/charts_MR_change_values.png
 identical  LICENSE
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml
 identical  ingress-nginx
 identical  ingress-nginx/values.yaml.gotmpl
  conflict  helmfile.yaml
 Overwrite helmfile.yaml? [Y/n]
 identical  loki
    create  loki/develop.yaml.gotmpl
    create  loki/main.yaml.gotmpl
  conflict  loki/values.yaml.gotmpl
 Overwrite loki/values.yaml.gotmpl? [Y/n]
    create  loki/staging.yaml.gotmpl
 identical  README.md
 identical  .gitlab-ci-asdf-versions.yml
 identical  promtail
 identical  promtail/values.yaml.gotmpl
 identical  .yamllint.yaml
 identical  .editorconfig
 identical  prometheus
 identical  prometheus/values.yaml.gotmpl
 identical  .gitlab-ci.yml
 identical  .gitlab
    create  .gitlab/merge_request_templates
    create  .gitlab/merge_request_templates/MEP_Cleyrop.md
 identical  .gitlab/renovate.json
 identical  CODEOWNERS
 identical  CONTRIBUTING.md
 identical  external-dns
 identical  external-dns/values.yaml.gotmpl
  conflict  .gitignore
 Overwrite .gitignore? [Y/n]

Expected behavior

Created files are in my filesystem
BUT the git status show me this

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   .copier-answers.yml
        modified:   .copier-version

Screenshots/screencasts/logs

No response

Operating system

Linux

Operating system distribution and version

Ubuntu 20.04

Copier version

copier 7.0.1

Python version

Python 3.8.10

Installation method

pipx+pypi

Additional context

No response

[pawamoy]

Created files are in my filesystem

So they're not deleted?

If Git only shows you unstaged changes, while the new files are there, I guess it means they're already tracked? Seems a like a Git issue more than a Copier one to me.

Are your repository and template public, so that I can take a look?

[nchal]

Hi, thx for ur answer

I'm pretty sure that git is ok, because since I don't use --override, I can see the files being created by copier step by step and then being delete as soon as the cmd ends

[pawamoy]

and then being delete as soon as the cmd ends

Oh, OK, I thought you were saying that the files were there, in the file system, after the command ended. Does your template use a task that could delete these files? By running a git or rm command for example?

It's hard to help without a reproducible example.

[nchal]

there is no tasks in my copier.yml 😢

and i don't know how to explain this better than that... when I update my project, just before the execution end, some changes seems to be deleted (even if it says created). The current workaround is to avoid using --force and git adding during execution pausing (when it asks for overwrite confirmation)

[nchal]

as u can see, git says : new changes are deletion of the newly created files :

[pawamoy]

Please share your repository URL if you're allowed to, or provide us with a reproducible example. We won't be able to help otherwise :/

[pawamoy]

Check that you didn't previously commit the deletion of these files. Copier would reapply the deletion after each update, explaining this behavior.

[jimilibilibob]

Look like copier just apply the difference between your 2 copier version.

I'm not sure to understand how copying works

Let's imagine this case: I have my project in copy version 1.5 (version specified in my .copier-version) but with a "toto" folder that differs from 1.5 (because I didn't want to apply the modifications). I run my copy update, which triggers the latest version, say 1.6.

What's going to happen?

1. Update my entire project. Both the 1.5 to 1.6 changes and my "toto" folder.
2. Update the differences between 1.5 and 1.6 (not my "toto" folder).

[pawamoy]

Copier merges both your changes and the template changes. If it cannot merge them, you get conflict markers in your files, that you can resolve yourself. There's a diagram here: https://copier.readthedocs.io/en/stable/updating/#how-the-update-works

[jimilibilibob]

Thank you So, that's the option 1. Copier will update "toto" folder

[yajo]

I can see the files being created by copier step by step

In updates, sometimes this list is misleading. See #1156. Is this what was confusing you?

[jimilibilibob]

I've try something.

My .copier-version is v4.0.0-beta.21

If i run :

copier --force  --vcs-ref v4.0.0-beta.23 update

I've only difference between v4.0.0-beta.22 to v4.0.0-beta.23 and i've not modification from v4.0.0-beta.21 to v4.0.0-beta.22.

I need to run command to every command to have all modification.

It's possible to get all the changes when I skip a version

[jimilibilibob]

And I think @nchal have this problem too.

He see all modification comes but modification that doesn't come from last copier skeleton version, so this file seems to be deleted

[pawamoy]

I need to run command to every command to have all modification.

That is not how Copier works, so I doubt that's what happens.

As always, without a clear reproduction of the issue we can only guess.

[yajo]

Hi there.

Thanks for your report, but it's lacking enough information and it cannot be fixed until it's provided. I close it for that reason.

If you provide the requested information later, please reopen it.

Thank you!