Bidirectional Mirroring e.g. GitLab Mirroring

[lflare]

• Gitea version (or commit ref): v1.1.3
• Git version: N.A
• Operating system: N.A
• Database (use [x]): N.A
  • [ ] PostgreSQL
  • [ ] MySQL
  • [ ] MSSQL
  • [ ] SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • [x] Yes (provide example URL)
  • [ ] No
  • [ ] Not relevant
• Log gist:

Description

Very simply, I understand that the current Gitea implementation has one-way remote mirroring, with the inability to modify the mirrored repository. However, I wish to propose a mirroring solution akin to that used in GitLab. Is this a feasible proposal? This feature is the only reason I have not switched fully to Gitea after all.

Screenshots

N.A

[lafriks]

How would merge conflicts be resolved?

[lflare]

Like GitLab, it probably shouldn't perform merges. If a fast-forward is not possible, it simply shows the sync error messages. Additionally, if we can set a low sync time and have the repository both push and pull to the upstream mirror, the amount of conflicts would be much lower.

[bkcsoft]

The only way to implement this correctly would be to pull on each push

[lflare]

@bkcsoft, I personally rather that the basic GitLab behaviour be included within the next release than having to wait for the hammering and debate for advanced Gitea-flavour behaviours to be included within the 2020 release.

I would personally be more than happy to work on Gitea again but there are a few reasons barring me from doing so.

[lflare]

I've thought about it for a bit and I think I have a strategy that may work. Essentially, Gitea pulls upstream repository every set interval or pre-push. After pulling, check if current push can be easily fast-forwarded, else reject and force user to pull first? What do you think @bkcsoft?

[lafriks]

@LFlare I think it would really slow down pushing to gitea if it would require pulling remote server

[daviian]

I think I miss the point of this feature. Can someone enlighten me and give use cases where this is useful / necessary?

[lflare]

@lafriks Maybe but also maybe not. Git when not transferring file content data, is relatively lightweight in that it compares checksums.

@daviian My case scenario is that I run Gitea with Github and I would really love to be able to push to my private Gitea and have it mirror the push to Github so I don't have to push twice and potentially forget to push.

[daviian]

@LFlare But why do you prefer pushing to gitea instead of directly pushing to github? What advantages to you see in doing so?

[lafriks]

If nobody is pushing to GitHub directly might be easier to set push git hook to push it further to github

[lflare]

@daviian I have people working on both the private Gitea instance and the Github instance and I personally prefer working on the private Gitea instance but also want the publicity of the repository being published on Github.

@lafriks That's when this feature request comes in, it's easily implementable with a couple of commands but it's incredibly inefficient if manual editing of the githooks are required every time I want to mirror a new repository.

[daviian]

@LFlare How do you manage releases etc.? Do you create every release twice?

[lflare]

@daviian Currently, I don't create releases, I just tag versions. This is off-topic though.

[strk]

What I do is have mirrors on github, with people only contributing via PR, then the PR is merged via commandline instead of via pushing the github button. A mirroring script, triggered by web hook, force-pushes into the github repo.

[pgaskin]

@strk I would also want something like this. I want to use Travis ci with some of my repos for testing on macOS, but it does not work with gitea, so I would want mirroring to GitHub from gitea

[bkcsoft]

Essentially one of the sites has to be the authorative truth.
If that's GitHub pushing to Gitea would look like this:

• On push pre-receive-pack (and pre-commit for gitea UI-updates) runs git pull --force and tells the user to rebase if it's changed (Promotes branching workflow). Pushes the new stuff to GitHub (before writing to the gitea-repo) and fails if that doesn't work.
• On pull pre-upload-pack will git pull --force from GitHub into Gitea, then passes that back to the user.
• On push to GiHub, GitHub needs to call a webhook /api/v1/:owner/:repo/mirror/update that does a git pull --force from GitHub.

For "Gitea is Authorative" the sequence would be

• On push Gitea does git push --force after written to it's repo. On failure tell the user (don't error), and set a "broken mirror"-flag in the UI.
• On pull Gitea does nothing to GitHub.

[strk]

I use Travis with unidirectional mirroring: Gitea triggers mirrorer script which pushes to GitHub (and GitLab) which triggers Travis (and gitlab-ci).

[lflare]

Never mind, I've worked out a solution that I'm happy with in a patch on my fork. Thanks.

[lafriks]

This can still be left open as feature request

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[lflare]

Not stale.

[lflare]

@lunny What does that label mean?

[lafriks]

@lflare just for stale bot not to close this for inactivity

[ghost]

You can't have bidirectional mirrors; you need something to mirror. There is one real object and one or more mirror images. That's just the way it is. There can only be one master.

[lflare]

@lifranc, I respectfully disagree. There's many applications for bidirectional mirroring for repositories. The wonders of Git allows for atomic commit histories and as long as there is a common base for repositories to be merged and mirrored, it can be done.

What is required, is a way for a user to configure merge strategies when a conflict is detected between two bidirectional mirrors, or perhaps a manual selector to choose how a conflict should be resolved. Apart from that, it should just behave like your traditional cloud file storage syncing mechanism.

[ghost]

What is required, is a way for a user to configure merge strategies when a conflict is detected between two bidirectional mirrors, or perhaps a manual selector to choose how a conflict should be resolved. Apart from that, it should just behave like your traditional cloud file storage syncing mechanism.

That's not mirroring. Fine then you want to automate two-way merging. You could perhaps reason about it better if you considered this way. To me a mirror doesnt take actions; it only replicates a master, hence the confusion. I dont see any disagreement, just a different use of words.

PS. Those who are disgruntled by the fact that mirrors can only reflect already existing light are in for a tough life.

PPS. Matrix protocol does something like this to synchronize rooms across the federation. It's not trivial. Much easier to create a mirror of each repo on the other servers, and merge using those, that way there is no conflict blocking the flow of data, and automatic merging can still be used there.

[CodeDoctorDE]

any updates?

[earl-warren]

For the record https://gitea.com/earl-warren/gitea/pulls/2 is the first step to support the Friendly Forge Format (F3) which can eventually be used for bi-directional mirroring including issues, releases etc.

[Thaodan]

For the record #6071 was closed because it was market about a duplicate (or is at least part of the issue). Simple git to git mirroring would be still possible either by overriding everything or synchronizing only specific ref/excluding to sync sync some.

[HuJK]

any updates? We can accpet conflict occurs and resolve conflict manually, but we want this feature. Not only mirroring, but syncing

[delvh]

1. For this feature, there is one immediate dependency that needs to be implemented first: A merge conflict resolution UI. We cannot just allow inconsistent states to be merged.
2. The best way to get a large feature is by creating a PR. Since Gitea is a community-based service, your only other option is to wait for someone else to implement it. Since there are currently >2400 issues open, good luck, might take a while until someone gets around to implement it for you.