matrss
commented
1 year ago The status quo seems to be that gitea includes the symlinks in the tar.gz output, but not the annexed files they point to, while the zip output does not even contain the symlinks (which AFAIK would be supported though, but git archive seems to create the zip files in a way that they don't include symlinks).
The latter regarding symlinks in zip files is true for GitHub as well. I didn't look into how GitHub or Gitea handle git-lfs standalone though, the git-annex repo I checked in GitHub uses git-lfs as a special remote.
The git archive command does not seem to be flexible enough to automatically resolve symlinks or include the whole annex in the output or similar things.
In the case of tar we could just add the whole annex to the archive in a second step, but that would include files that are not present in the exported branch/tree-ish and the resulting archive provides a suboptimal experiences in my archive viewer, since I needed to unpack the archive first before the symlinks became usable. This would also not work for the current zip output, because of the missing symlinks.
Therefore, the most straightforward approach I can think of is creating the zip or tar archive using git archive as it's currently done and then in a second step adding all the annexed files directly to that, replacing the (possibly) existing symlinks. We can get a list of all the annexed files in a branch/tree-ish, even those not present on the server, using git annex find --branch <ref> --anything and resolve them to files in the annex using the symlinks tracked in git.
An alternative would be the approach taken by DataLad's datalad export-archive, which does not use git archive and instead reimplements the entire export routine. I think there is some value in keeping the git archive usage though, since it would mean that nothing changes for repos that do not use git-annex.
With this I still have two open questions:
- How would we handle missing annexed files? Should gitea refuse to export, show a warning but provide an archive with what it has or not show a warning at all?
- Currently, gitea can export to a "bundle" format as well. This seems to be some git specific way of exporting the entire repo including history as a file. It looks like it would be more complicated to get all the data into that. Should we even keep that option? I would be inclined to remove it all-together, since I do not see a use case for it that wouldn't be handled by a normal clone anyway.
What do you think about this? I could try implementing the "create archive with git archive, then add annexed files" approach, if you are open to a PR for that. I am new to Go though, so I cannot guarantee perfectly idiomatic code.
kousu
The "Releases" page lets you download an archive (from
git-archive?). It would be handy if this worked with git-annex datasets too, since then we could share people a link like https://data.neuropoly.org/neuropoly/some-project/archive/1.0.0.zip and they could get a versioned the data without having to install or learn git-annex (with gitea,GET /neuropoly/some-project/archive/1.0.0.zipdownloads assome-project-1.0.0.zip, preserving the version number, so long as the receiver uses a standard browser orcurl -LJO).It's good if people do want to learn and use git-annex, but some applications -- imagine, deploying training sets to clusters, simple exploratory work, etc -- don't need the extra headache.
I'll note that we currently can't do this with e.g. https://github.com/spine-generic/data-multi-subject/releases/tag/r20230223; there, the download links give a small <1MB .zip containing mostly annex pointers.
Plan