Version data repositories not using Git but using sound patch system

[strogonoff]

Considering adopting a version control system based on patches (Pijul/Darcs type).

This would save a lot of duplication and reinventing the wheel:

• Give us powerful primitives for managing revisions of objects—no need to reinvent and maintain our own logic of patching and diffing.
• Save us from having to duplicate object revision management in actual file contents.
• Enable us to reuse file version management for data revision management.
• Save us from having to validate data consistency in case Git ends up silently doing a “bad merge”.

I have brought up using Git layer for versioning object revisions before, but it is suboptimal due to rigid history (changing object history would require a rebase and force-push—as @ronaldtse pointed out, SDOs and others may find such rigid history problematic), and due to unreliable merging algorithms.

The approach suggested in this issue does not have those downsides. Darcs/Pijul algorithms have a huge advantage in that we can e.g. unapply a change made in the past without altering the history and having to force-push, since the identity of a patch once created never changes.

To migrate, we can maintain converters for different repository types between our “YAML-with-revisions” convention and e.g. Pijul repository format. “YAML-with-revisions” style would remain supported as export/backup/interop format.

A downside is that we would not be able to use GitHub to host the “reference” repository versions.

[ronaldtse]

I agree that the Darcs/Pijul property of reversing a patch anywhere to be very attractive. However the popularity of the Git platform, which is used by all our users, remain important where they can deploy and launch their registries...

[strogonoff]

Your comment is about GitHub not Git

On 26 Sep 2020, at 4:09 PM, Ronald Tse notifications@github.com wrote:

 I agree that the Darcs/Pijul property of reversing a patch anywhere to be very attractive. However the popularity of the Git platform, which is used by all our users, remain important where they can deploy and launch their registries...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ronaldtse]

The comment was about Git. While we consider integrating with GitHub, the other platforms like GitLab and BitBucket are also relevant.

[strogonoff]

Sure.

Would you agree if there was a PijulHub, it would not be an issue?

Thus the concern appears to be about managing collaboration-facilitating infrastructure for organizations in our care (i.e. we can’t rely on GitHub for free), not solution as it stands on its own.

Judging on the merits for the solution itself, so far I believe YAML with revisions on top of Git is redundant, fragile and less DRY, losing to YAML on top of a versioning system based on superior patch theory.

It needs to be taken into account. We should ask, what would we do if there was no GitHub? And apply the same logic here.

On 27 Sep 2020, at 10:49 AM, Ronald Tse notifications@github.com wrote:

 The comment was about Git. While we consider integrating with GitHub, the other platforms like GitLab and BitBucket are also relevant.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ronaldtse]

@strogonoff I'm not sure if a PijulHub obsoletes the issue.

• A register has a need to track register items
• Each register item has an individual history
• We want to store a register in a data repository (hopefully those repositories are readily useable by users)

I agree that the Pijul/Darcs model works closest to this. However, the data administrators are familiar with files (in an archive) or Git because they can be considered a portable data archive.

I also agree that implementing revisions on top of Git is redundant and fragile. It would be better if the model of the data operations align with the operations of the register.