Introduce a non-metadata mode for gitfiletree

[ThierryGoubier]

FileTree metadata (or any kind of metadata in .st files, really) is conflict-prone when merging and cherry-picking under git. The gitfiletree merge driver solves the merge issue, but is seen as intimidating to install (and cherry picking does not allow the merge driver to be used).

GitFileTree does not read that metadata anyway, it just writes it for filetree compatibility. Having a way for gitfiletree to generates repositories without metadata could help people who would like to merge without the merge-driver.

[ThierryGoubier]

Decision points for alternatives:

• st chunk format contains metadata, so merge conflicts would happens anyway.
• st chunk format as used by ST/X / Jan Vrany has additional metadata in comments, creating conflicts (https://bitbucket.org/janvrany/stx-libscm).
• filetree without metadata is equivalent to filetree with the appropriate .gitignore.
• filetree has modes to compensate for missing metadata
• some metadata (class definitions) are in fact code and having them conflict is seen as necessary
• filetree one file per method approach has nice benefits for querying the repository compared to one file per class / one file per package.

[dalehenrich]

I am interested in how this goes in Pharo ... it has always been the case that a metadata-less package in filetree can have it's metadata restored by doing an adopt (is the adopt menu item still present in the Pharo monticello browser) ... but I chose not to go that route, because adopt is/was pretty obscure and only folks had to know that it was needed ... Now that there is more overall understanding of git and the filetree metadata issues it might have a better chance of sucess ...

Off the top of my head here are a few more decision points:

• disallow package copy from a metadata-less repository to a metadata-full repository ... allow for an adopt on the fly(?)
• warn(?) when copying a package from a metadata-full repository to a metadata-less repository
• monticello metadata absence should probably be repository wide

[ThierryGoubier]

Yes, adopt is probably complex and would have been frequently necessary. If you consider that gitfiletree never reads the metadata anyway, then the situation becomes a bit different.

For gitfiletree I would consider allowing copy to metadata-less repository, with warning(?) (usually it means recreating an history from the git logs, like copying all versions from another repo to git... not sure about the warning).

And yes, metadata-less would be a repository-wide property. Probably could only be set upon repository creation.

[dalehenrich]

it occurs to me that the method properties should also be counted as monticello meta data ... if you get rid of that than I think that conflicts are real conflicts for everything else ...

[ThierryGoubier]

Yes. Metadata-less for me would leave only properties.json for class and package, and inside monticello.meta, the .st files. I think even monticello.meta/package is redundant with the path.

Conflicts in properties.json for class is something you want to see, if I remember correctly.

[ThierryGoubier]

First try in 6ca3800.

It works, but FileTree can't read it.

[ThierryGoubier]

FileTree able to read (Cypress mode fallback) in b0db19c.

Test repository: Roassal2 (https://github.com/bergel/test2)

@dalehenrich : if you want to have a look.

[ThierryGoubier]

Tests are green, let's run with it. Only apply to new repositories, and no API to change it during the life of the repository.

Merged by pull request #163

[ThierryGoubier]

Enabled, has been in use for a while now, consider this closed.

[estebanlm]

@ThierryGoubier how can I declare a repository metadata-less? (I didn't find a documented way :P)

I'm interested on testing it with https://github.com/pharo-project/pharo-vm

[ThierryGoubier]

It is on purpose that switching an existing gitfiletree/filetree repository to metadata-less mode is not documented. By default, any new repository created by GitFileTree (pharo5 development version) is Metadata-less. And starting yesterday, with the normal version of GitFileTree for Pharo5 as well.

The Metadata-less status of a repository is recorded as a setting inside the .filetree of the repository (for example https://github.com/ThierryGoubier/AltBrowser/blob/pharo5/.filetree).

[dalehenrich]

@ThierryGoubier you realize that by making metadata-less the default you have broken the cross-platform compatibility for FileTree ... either the changes that you've made need to be ported to Squeak or GemStone or you need to make this a concious choice .... I lost several methods in the last week or so when I was making changes in both GemStone and Pharo5.0 and I lost the changes that I made in Pharo5.0, because GemStone did not recognize that changes had been made to the repository ... this is not cool ... Fortunately I noticed that this was going on .... and had to manually look for lost data in a merge with 100 files change and 130 commits!

[dalehenrich]

I like the idea but we have to keep these things in sync or chaos will ensue!

[dalehenrich]

I'm going to guess that my lossage was due to manual edits, but I'm afraid that the same things will happen if all Filetree platforms are not taught about the meta-data-less mode .... it's a real pain to recover these lost edits ...

[dalehenrich]

Naturally, I do not have any time to devote to making chages to the Squeak or GemStone versons of FileTree ;(

[ThierryGoubier]

Hum, I thought that I designed the metadata-less mode in a compatible way with the following effects:

• FileTree will fail on a metadata-less package unless a specific patch is made to FileTree (b0db19c), and this patch isn't present on the Squeak/Gemstone branches. It's not even present on the pharo4.0 and older pharo branches.
• A switch to Metadata-less mode does not look like a hand-edit, but also records the removal of the metadata in the git commit data.
• As of now, unless you hack, it is not possible to switch an existing repository to metadata-less mode.

So, I would expect a metadata-less related error to be very visible in a git diff, because it would include the removal of the metadata.

You are certainly pointing a risk related to hand edits and FileTree, however.

[ThierryGoubier]

Now, it may well happen that merging between a metadata-less branch and a metadata branch may result in the addition of methods without updating the metadata. Reading such a package by FileTree will correctly pick-up the added methods, unless some higher layer decides not to read the package because it has wrongly deduced nothing has changed.

Maybe making the metadata-less mode compatible with normal FileTree was a bad idea, after all.

Or I could add a special version metadata to the metadata-less mode, so that it would create conflicts when merging on a metadata branch? But then it would also create conflicts in metadata-less mode. Or if not, then it may be missed when cherry-picking.

[ThierryGoubier]

Now, it may well happen that merging between a metadata-less branch and a metadata branch may result in the addition of methods without updating the metadata. Reading such a package by FileTree will correctly pick-up the added methods, unless some higher layer decides not to read the package because it has wrongly deduced nothing has changed.

And this could only have happened if someone would have:

• Created a new repo in metadata-less with Pharo5 and a dev version of GitFileTree
• Created a branch in that repository and switched to FileTree in that branch only

Now, it's clear that FileTree does not check if its metadata is on sync with the package on-disk contents, which has made hand-edits a dangerous exercise since the very beginning.

Adding the ability of FileTree to check that the metadata is in sync may be a solution except that, higher above, Metacello may still decide nothing has changed, since there is no API for FileTree to communicate that the package is "dirty" on-disk. Maybe by recreating a pseudo-version and rewriting it on disk when FileTree notices it?

[dalehenrich]

I'm glad you added a "poison pill" to metadata-less mode --- I plan to go meta-dataless sooner or later, but there are just too many things to do :) Sorry for the interruption

[dalehenrich]

Metacello may still decide nothing has changed, since there is no API for FileTree to communicate that the package is "dirty" on-disk.

Actually, I have asupport for meta-dataless packages in Metacello ... Basically with a Cypress project the Monticello version is ignored and the definition differences drives whether or not a package gets loaded ... I haven't used this feature a whole lot, but if you take it for a spin and find it useful, I can take the time to make sure it's part of the standard Metacello release (and eventually get into Pharo) ... oh and I'll fix any bugs that pop up as well :)

Frankly for all disk-based BaselineOfs I think this should be the behavior of Metacello --- in a checkout all of the packages on disk are meant to be loaded together regardless of Monticello version ... would also solve the problem of hand editting ... Doing this is a bit bigger step than I'm prepared to take right now ... but it does keep cropping up as something I could do ...

[ThierryGoubier]

@dalehenrich : I think given the time, this shows the need for two tools / features

• A sanity / synchronisation check for FileTree, ensuring that the on-disk package is correctly synchronized with it's metadata. If not, instead of presenting the stored metadata version, then FileTree present to above a version whose first ancestor is the metadata/version file contents, recreating the necessary metadata (if the changes are valid -- erroneous changes are possible).
• An API telling the relative order of two packages. So that we can encode the following behavior: if the in-image version of a package has the repository newest / only version as an ancestor, then do not upgrade the package. If that is not the case, upgrade the package.
  • Like that, if we end-up with Cypress (no metadata), this rule means "upgrade"
  • In other cases, we can protect from a downgrade.
  • In the first case (FileTree, say a .dkh.7 + hand edits), by providing a temporary version (dkh.8) with the on-disk recorded version (dkh.7) metadata as an ancestor, we can upgrade over the package (dkh.7) without loosing the hand edits. Resaving FileTree would record us a (.dkh.9), with ancestor (.dkh.8), and dkh.8 would have dkh.7 as an ancestor.