missing 3-way merge

[kevinhendricks]

I recently found the blog of James Coglin, "The If Works" See: https://blog.jcoglan.com/2017/05/08/merging-with-diff3/

Where he does a wonderful job both explaining how a diff3 style merge works and providing a basic implementation of it in ruby (I think but I have never programmed in ruby so I am unsure).

Using this blog as the basis, I created a diff3merge.py program I would be willing to contribute to dulwich under an MIT license that appears to work.

It does require a diff routine that will output all lines (not just modified lines for ease of calculating the line number offsets for chunks). I also have a python implementation of the myers_diff that does just this but it could easily be replaced by a difflib call to use ndiff while dropping the extra intraline edit hints.

I am sure it still needs work but does pass the simple test cases I have tried. It should provide a good basis for further modification and testing.

Any interest?

[kevinhendricks]

Had some more free time and used your tests/test_merge_base.py as a model and created tests/test_diff3merge.py and also cleaned up a few things in diff3merge.py

Hopefully this will help.

[kevinhendricks]

Any update on merge code? Is there something I could do that would help?

[jelmer]

I'm hoping to spend some more time on this in two weeks time when I have a week off.

[kevinhendricks]

Any updates on getting merge going in current dulwich?

[kevinhendricks]

@jelmer

It has now been 3 years since the primary work on the requirements for simple merge support in dulwich (talked about in this issue) were contributed.

Are there any realistic plans to add support for even a simple 2 branch merge capability to dulwich in the near term (say next 6 months or so?).

If so, I have pulled in your current master to our Sigil-Ebook/dulwich tree and updated the items to get back to the point we left things at back then, just in case it helps:

https://github.com/Sigil-Ebook/dulwich

And again I am willing to volunteer my help and donate my code to dulwich to help bring "merge" about.

If it is not possible in that timeline, would you please let us know so that we at Sigil know to explore alternatives. Not being able to merge multiple versions of the same epub inside Sigil without forcing users to use git itself is a real limitation we simply need to address one way or the other and can delay no longer.

Thank you.

kevinhendricks

[kevinhendricks]

I have made some progress at separating out my merge changes from official dulwich source and making them stand alongside dulwich so that Sigil's merge development processes can be run separately from official dulwich sources.

Our additions to dulwich can now detect merge contents and actually writes out the merged files to the repos working directory. We have also extended the diff capability to include index to working dir, tree to working dir, and tree to tree, etc.

We have created a new repo called dulwich_merge to house the code to try to get better testing and bug reporting by other users needing a merge capability without having to worry about exact structure of the source or interfering with your dev processes or timeline.

See: https://github.com/Sigil-Ebook/dulwich_merge.

Included in that code is a fixed graph.py merge_base routine that is much much more efficient than the one I wrote previously.

See https://github.com/jelmer/dulwich/issues/910 at the bottom for the mode efficient code.

You probably want to incorporate those changes into your own graph.py so can_fast_forward becomes much faster and usable on larger repos.

Hope something here helps.

Kevin

[jelmer]

I've seen your posts; I'll try to comment on this some time later this week.

[kevinhendricks]

In the new standalone dulwich_merge repo (see link previously provided above) branch merge will now collect and return merge results fully, auto stage those files it can while leaving those with chunk conflicts to be hand merged as unstaged. If no hand merging is needed, it will automatically commit the results just like git does.

In addition, a pure python histogram diff module was created by me and added to make merging as close to what git does by default.

All my tests of this code to test merge actual branches from Sigil's git repository have passed.

When you get a free moment, I could really use some feedback about the current code especially what is included in MergeResults, how MergeConflicts vs simple Chunk Conflicts are handled, parameters to pass along to handle options, etc, so that I don't end up veering too far away from something you might want to eventually add to dulwich proper.

As it stands, I plan to add dulwich_merge to Sigil's python plugin code in the near future so that plugins can be written to directly interface with git repos used to version control epubs being authored by multiple developers. This will allow me to implement a pull (and merge) for the cases where fast forward is not possible

Hope this helps.

[clach04]

@kevinhendricks wrote:

Thanks. I will check it out. I have posted the 3-way merge code here:

https://github.com/Sigil-Ebook/dulwich/blob/master/dulwich/diff3merge.py

Fair warning: it is still in very early form...

I only just discovered this, is diff3merge.py available any where @kevinhendricks ? From your comment from last month it sounds like you intend to include in Sigil in the near future. Any chance you could restore your Dulwich fork? It looks like you put significant time/effort into this.

I'm doing research into 3 way merge tools (so this question isn't directly Dulwich related).

[kevinhendricks]

@clach04 I would be happy to post the my version of dulwich with merge capability that I call pyRotter. It is still experimental but it includes all of the diff3merge code and support modules.

Message me at my repo to remind me and I will post it there for you. kevinhendricks@github.com