search

johnmehr / gitup

A minimalist, dependency-free FreeBSD program to clone/pull Git repositories.
BSD 2-Clause "Simplified" License
50 stars 9 forks source link

gitup no longer able to synchronize src on 13-STABLE #69

Closed Math2 closed 3 years ago

Math2 commented 3 years ago

I believe that this is following this commit: https://cgit.freebsd.org/src/commit/?id=04c4bd7f7b525b1627304894e3413693e945bc0d

gitup started erroring out when trying to synchronize the repo. I tried to delete /var/db/gitup/src and with an empty /usr/src directory but the same thing happens.

$ gitup src
# Scanning local repository...
# Host: git.freebsd.org
# Port: 443
# Repository Path: /src.git
# Target Directory: /usr/src
# Want: cb96c6ba30cb917bc1223247caa93d135818199a
# Branch: stable/13
# Action: clone
gitup: load_object: local file for object 3331601f6dc50ef2c9779c1656218701b48b276c -- /usr/src/sys/contrib/openzfs/scripts/zfs-images not found: No such file or directory

That's with gitup 0.94 and this config file:

# $FreeBSD$
#
# Default configuration options for gitup.conf.
{
    "defaults" : {
        "host"           : "git.freebsd.org",
        "port"           : 443,
        "verbosity"      : 1,
        "work_directory" : "/var/db/gitup",
    },

    "ports" : {
        "repository" : "/ports.git",
        "branch"     : "quarterly",
        "target"     : "/usr/ports",
        "ignores"    : [
            "distfiles",
            "packages",
            "INDEX-13",
        ]
    },

    "src" : {
        "repository" : "/src.git",
        "branch"     : "stable/13",
        "target"     : "/usr/src",
    },

}
vpushkar commented 3 years ago

Same here, trying to fetch fresh 13-STABLE and it fails. Trace attached gdb.txt

DeepCoreDK commented 3 years ago

Same here, also fails for current. Looks like the last openzfs import did something that getup doesn't like.

wesnm commented 3 years ago

Specifically, it looks like the problem is a remote repository module:

diff --git a/sys/contrib/openzfs/.gitmodules b/sys/contrib/openzfs/.gitmodules
index d400f10a7e63..9eaa2b0495dc 100644
--- a/sys/contrib/openzfs/.gitmodules
+++ b/sys/contrib/openzfs/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "scripts/zfs-images"]
    path = scripts/zfs-images
-   url = https://github.com/zfsonlinux/zfs-images
+   url = https://github.com/openzfs/zfs-images

The stack trace shows the file type (0160000) as "whiteout" (S_IFWHT)? Never heard of that before now. "git" with "--depth 1" just shows it as an empty directory. Seems like the proper thing to do would be the same.

emaste commented 3 years ago

Some more info: https://lists.freebsd.org/archives/freebsd-git/2021-June/000015.html

We are probably going to just delete sys/contrib/openzfs/scripts/zfs-images in the FreeBSD tree

Math2 commented 3 years ago

Specifically, it looks like the problem is a remote repository module:

diff --git a/sys/contrib/openzfs/.gitmodules b/sys/contrib/openzfs/.gitmodules
index d400f10a7e63..9eaa2b0495dc 100644
--- a/sys/contrib/openzfs/.gitmodules
+++ b/sys/contrib/openzfs/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "scripts/zfs-images"]
  path = scripts/zfs-images
- url = https://github.com/zfsonlinux/zfs-images
+ url = https://github.com/openzfs/zfs-images

The stack trace shows the file type (0160000) as "whiteout" (S_IFWHT)? Never heard of that before now. "git" with "--depth 1" just shows it as an empty directory. Seems like the proper thing to do would be the same.

Interesting.

$ git ls-tree 584b4410bb044e4d67454b8daaefa9321f5d88ae
100644 blob 5621a6e147a0e3d934f1f35cd6ae71f0a65dbd4b    .gitignore
100644 blob f59826fe6bc87241063e25e8f53873f308bead01    Makefile.am
100755 blob 71cf521666a69ed4300e4b0e3542ae472c6e32c5    commitcheck.sh
100644 blob 8268315b336187326d712c787e8f3dd4118c31b8    common.sh.in
100755 blob d19718ecf443a1591fb414c4db79f46f5dea4cca    cstyle.pl
100755 blob 8649b93183a2bc81b0e8cdecd1e85ff198f22af1    dkms.mkconf
100755 blob 81de475877dddd3ff1e5db48886168af0b8761f4    dkms.postbuild
100755 blob 5112cc807f678055135bb046a3f67a474ce94c2e    enum-extract.pl
100755 blob 240cde3106a787ea1f0a861ad860daf077a6c718    kmodtool
100755 blob f5c743b137991ff269435c611c1db786e3cbc25a    make_gitrev.sh
100755 blob 186d94639a5651d2ff2c26365ae936b79731b655    man-dates.sh
100755 blob 87e817500b1b2e55205eae8e6014fd29e5b76d08    paxcheck.sh
100755 blob 02b4922006f270933e4771867db3aa2f080caa8a    zfs-helpers.sh
160000 commit 3331601f6dc50ef2c9779c1656218701b48b276c  zfs-images
100755 blob 45e08473d35cf5340664f0d906b22444c32446e8    zfs-tests.sh
100755 blob 2f5f3f8fdb54d452693cd1c8e0ff7dc342b1b838    zfs.sh
100755 blob 99824d6dd4afbb2c44e57e43d1aaecc8f2efcbce    zfs2zol-patch.sed
100755 blob 56dfbadae47b7a4207864c58018b4a728ed5f853    zimport.sh
100755 blob bbe326aa07e08347870adb8dd7041819191503ce    zloop.sh
100755 blob bb6d9faac450e88319106e093af33c2b135f2601    zol2zfs-patch.sed

The tree object directly points to a commit! I didn't know git could do that. I thought only commits would point to commits (and trees) and trees would point to trees and blobs.

That's a so-called "gitlink" apparently. I think that this is how git remembers which specific revision of a submodule was checked out. And git encodes it with a file mode that matches our S_IFWHT "whiteouts".

/*               
 * A "directory link" is a link to another git directory.
 *                    
 * The value 0160000 is not normally a valid mode, and
 * also just happens to be S_IFDIR + S_IFLNK
 */
#define S_IFGITLINK     0160000
#define S_ISGITLINK(m)  (((m) & S_IFMT) == S_IFGITLINK)

And yeah I think that unless you specifically ask to checkout submodules (which gitup doesn't do and probably doesn't need to do), you can just ignore this tree entry.

emaste commented 3 years ago

git encodes it with a file mode that matches our S_IFWHT

Well, that is curious. Presumably git itself always interprets that as a special case and will never pass such a mode to system calls, so it's only 3rd party implementations (or git bugs) that might encounter this issue.

checkout submodules (which gitup doesn't do and probably doesn't need to do)

FreeBSD's current stance is we won't use submodules in the upstream tree. It's unlikely, but possible, that this will change at some point in the future.

Math2 commented 3 years ago

Well, that is curious. Presumably git itself always interprets that as a special case and will never pass such a mode to system calls, so it's only 3rd party implementations (or git bugs) that might encounter this issue.

Yeah it's special-cased all over the git source and comments say it was chosen because it's an unused value on Linux. And it is sometimes just treated as if it were a directory.

johnmehr commented 3 years ago

I just committed a partial fix to get past the current breakage (but it's not a complete fix for supporting submodules -- still working on that).

Math2 commented 3 years ago

I just committed a partial fix to get past the current breakage (but it's not a complete fix for supporting submodules -- still working on that).

This seems to work perfectly now. Thanks!

BTW, while testing I noticed some line ending differences in the files checked out by gitup and those in a git working directory for the same branch. They're the files listed in contrib/bc/.gitattributes. The .gitattributes got added fairly recently with a bc update. But it's not causing any problems.

johnmehr commented 3 years ago

I'm also seeing discrepancies with

contrib/bc/bc.vcxproj contrib/bc/bc.vcxproj.filters contrib/bc/bcl.vcxproj contrib/bc/bcl.vcxproj.filters

in the packfiles I'm testing with so there's something awry. Investigating...

Math2 commented 3 years ago

I'm also seeing discrepancies with

contrib/bc/bc.vcxproj contrib/bc/bc.vcxproj.filters contrib/bc/bcl.vcxproj contrib/bc/bcl.vcxproj.filters

in the packfiles I'm testing with so there's something awry. Investigating...

IIUC it's just newline normalization. The .gitattribute file tells git to switch the CRLFs into LFs before checking in and turn them back into CRLFs when checking out.

grahamperrin commented 3 years ago

https://github.com/johnmehr/gitup/issues/69#issuecomment-869225041

… partial fix to get past the current breakage …

Thanks, can it make it way to the ports collection?

I get gitup: load_object: local file for object 3331601f6dc50ef2c9779c1656218701b48b276c -- /usr/src/sys/contrib/openzfs/scripts/zfs-images not found: No such file or directory on FreeBSD 14.0-CURRENT.

johnmehr commented 3 years ago

Sorry for the delay. I just pushed out 0.95 -- how does it look?

emaste commented 3 years ago

Note that the zfs-images submodule has since been removed: https://github.com/freebsd/freebsd-src/commit/79e6eb5c018d44ee59c5b9b10755d116c860c366

edinilsonjs commented 3 years ago

Sorry for the delay. I just pushed out 0.95 -- how does it look?

Works great!!! Thank you very much! Regards..