Unexpected merge conflict when updating with nested object as argument in data file

thor-matter commented 2 weeks ago

Describe the problem

Thank for a great tool! I have been using Copier for a while now and I am very happy with it.

However, recently I get an unexpected merge conflict when applying an update with a nested object as an argument within a data file.

I have attached a basic template to reproduce the issue.

This may be related to the following Issue regarding hidden merge conflict details in tools. I am experiencing the same problem in pycharm git integration.

https://github.com/copier-org/copier/issues/1833

Template

template-basic.zip

To Reproduce

Steps to reproduce:

$ mkdir test-basic
$ cd test-basic
$ git init

$ copier copy ../template-basic . 
$ git add .
$ git commit -m "Initial commit"

$ copier update --data-file data.yaml 
$ git add .
$ git commit -m "Update items from data file"

$ copier update --data-file data.yaml 

$ git status
On branch master
Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add <file>..." to mark resolution)
    both modified:   document.md

no changes added to commit (use "git add" and/or "git commit -a")

$ git diff
diff --cc document.md
index a552bb2,a552bb2..0000000
--- a/document.md
+++ b/document.md

In addition, I get an unexpected merge conflict when changing the items in the data file.

$ sed -i 's/3/4/g' data.yaml
$ git add .
$ git commit -m "Change items in data file"

$ copier update --data-file data.yaml

$ git status
On branch master
Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add <file>..." to mark resolution)
    both modified:   document.md

no changes added to commit (use "git add" and/or "git commit -a")

$ git diff
diff --cc document.md
index a552bb2,a552bb2..0000000
--- a/document.md
+++ b/document.md
@@@ -7,6 -7,6 +7,10 @@@ Here are our items

          this: 2

++<<<<<<< before updating
          test: 3
++=======
++        test: 4
++>>>>>>> after updating

Logs

No response

Expected behavior

I would not expect any merge conflicts in the described conditions.

Screenshots/screencasts/logs

No response

Operating system

Linux

Operating system distribution and version

Ubuntu 22.04.5 LTS

Copier version

copier 9.4.1

Python version

3.12.2

Installation method

pipx+git

Additional context

No response

pawamoy commented 2 weeks ago

First issue might be fixed by #1813.

For the second issue, when Copier runs the update after you've modified data.yaml:

it regens a new project with this data file (test=4)
meaning document.md has test: 4 in it
it computes the diff between this fresh project and your current project: test: 4 -> test 3!
it updates the current project in-place: document.md now contains test: 4
but the computed diff says it must be test: 3!

I suppose that's how you get the merge conflict. And this behavior seems correct, although maybe not intuitive. IMO it stems from how data.yaml and document.md become de-synchronized when you update only data.yaml. Not sure if this is something Copier should support.

Also, maybe my analysis is wrong.

thor-matter commented 2 weeks ago

First issue might be fixed by #1813.

I'm running the latest version, so I don't think this resolved the issue I'm seeing.

For the second issue, when Copier runs the update after you've modified data.yaml:

it regens a new project with this data file (test=4)

meaning document.md has test: 4 in it

it computes the diff between this fresh project and your current project: test: 4 -> test 3!

it updates the current project in-place: document.md now contains test: 4

but the computed diff says it must be test: 3!

I suppose that's how you get the merge conflict. And this behavior seems correct, although maybe not intuitive. IMO it stems from how data.yaml and document.md become de-synchronized when you update only data.yaml. Not sure if this is something Copier should support.

Maybe I'm confused then. Is the data file not regarded as a kind of answer?

I did some testing where I would store the items provided from the data file in the answers file, and I did see different behavior, but still the confusion persists. It would make sense to me that changes in the data file would apply as an answer change.

To be clear, the file is marked with a conflict, when the template should simply update the value, as it does when changing an answer. Maybe it has something to do with the size of the data structure?

pawamoy commented 2 weeks ago

I'm running the latest version, so I don't think this resolved the issue I'm seeing.

The fix from #1813 was not released yet.

thor-matter commented 2 weeks ago

I'm running the latest version, so I don't think this resolved the issue I'm seeing.

The fix from #1813 was not released yet.

Ah, my bad. I'll do some testing based on the master branch instead.

thor-matter commented 3 days ago

I have tested master and can confirm that the merge bug has been fixed. It would be great to have it released.

The other bug is reported here https://github.com/copier-org/copier/issues/1833

copier-org / copier