Design 3 way diff/merge approach for upstream changes

[jmatsushita]

Part of #8

The typical use case is:

• I re-use content from an upstream provider (typically as a starting point for content adaptation)
• I modify the content.
• Upstream makes changes

The problem

• How do I integrate these changes in my adapted content?

Possible solutions (from how the code people do it)

• 3 way merges?
• recursive or weave merges?

Test

• Use openmentoring DFAK or SIAB upstream content as a test.

The upshot?

• There's already 3 way diff/merge tooling for Markdown that we can reuse.
  • https://github.com/netj/markdown-diff
  • https://github.com/adamarthurryan/dubdiff
• There's huge ecosystem of tools and practices of dealing with this problem in code that can be applied to content.
• There's innovation that could be adapted to (for instance with semantic merge)

Other interesting approaches might include using the markdown structure as the underlying data and use other diffing tools for this.

Interestingly, I couldn't find research literature about applying these tools to natural language text...

[jmatsushita]

Re. Testing : this issue iilab/openmentoring-web#2 is a real life example

[jmatsushita]

Here's a concrete example applied to the case where you need to allow 2 forks of the same repo (one private the other public) to co-evolve and be able to apply changes from the one to the other and vice versa.

1. Initial state

Private file

The private file has an email address.

> cat private.md
Hi there,

Jun.

jun@iilab.org

Public file

The public file is the same but has the email address hidden.

> cat public.md
Hi there,

Jun.

2. Change Private file

Now a change, let’s say to the private.md file:

> cat private2.md
Hi there,

How are you?

Jun.

jun@iilab.org

3. Merge

Now using diff3 the 3-way merge tool in “easy merge” mode:

New Public file

Asking to apply the changes to public.md

> diff3 -m3 public.md private.md private2.md
Hi there,

How are you?

Jun.

The change from private2.md has been applied, but the email address still doesn't show.

4. Change Public File

Make a change to public.md

> cat public2.md
Hi there!

Jun.

5. Merge into private file

We can apply the change to private.md

> diff3 -m3 private.md public.md public2.md
Hi there!

Jun.

jun@iilab.org

It works, the change from public2.md (comma into exclamation mark) is now in the private file too (and therefore shows the email address).

6. Merge changes in both the private and public files

Into the private file

And drum roll, we can also apply the change to private2.md

> diff3 -m3 private2.md public.md public2.md
Hi there!

How are you?

Jun.

jun@iilab.org

Into the public file

> diff3 -m3 public2.md public.md private2.md
Hi there!

How are you?

Jun.

jun@iilab.org

Applying to git

Now I don’t exactly know how to apply that to git, as it does use diff3 under the hood but between BASE (the common parent in the git commit history) LOCAL (the local file) and REMOTE (the remote file). So this would need to be tweaked if we wanted git to do the right thing. I don’t think you can pass the “-3” option (which might not be wanted in all cases) to git either so you would have to use an extra step with a visual merge tool (kdiff3) for instance, which might be good anyway to introduce a manual verification step.

git mergetool and kdiff3

kdiff3 can be added in the git configuration to fire up automatically when git mergetool is called but again, I’m not sure how to tell git to point to the proper BASE/LOCAL/REMOTE that makes sense in our case.

I was able to get kdiff3 to do the right thing (note that BASE needs to be first with kdiff3, rather than in the middle with diff3) with: For the diff into the public repo

> kdiff3 -m private.md public.md private2.md

For the diff into the private repo

> kdiff3 -m public.md private.md private2.md

For the diff into the private repo with a public modification too:

> kdiff3 -m public.md public2.md private2.md 

Next

It might be possible to point git to the right remotes to make this part of a standard git merge process or add a bit of automation to streamline things.

I wonder also how this would work with a versioned content packaging approach...

@seamustuohy @florianap

[jmatsushita]

@critocrito