search

serpent-os / boulder-d-legacy

Replaced by Rust tooling
https://serpentos.com
21 stars 7 forks source link

git: Submodule checkout not working #80

Closed ReillyBrogan closed 1 year ago

ReillyBrogan commented 1 year ago

I'm trying to package qt5-declarative which has the following upsteam: git|https://invent.kde.org/qt/qt/qtdeclarative.git : 05c3f4921d81fb1b0eb497515c24ae56221d1e0a

This fails with the following error message (and also hangs requiring ctrl+c):

[13:48:33] INFO      Stage success: clean-root
[13:48:33] INFO      Stage success: create-root
Cloning into bare repository '/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative.git'...
remote: Enumerating objects: 462449, done.
remote: Counting objects: 100% (6431/6431), done.
remote: Compressing objects: 100% (2720/2720), done.
remote: Total 462449 (delta 4436), reused 5349 (delta 3673), pack-reused 456018
Receiving objects: 100% (462449/462449), 156.31 MiB | 20.88 MiB/s, done.
Resolving deltas: 100% (374549/374549), done.
Cloning into '/var/cache/boulder/upstreams/git/invent.kde.org/qt/qt/qtdeclarative.git'...
done.
HEAD is now at 05c3f4921d Revert "Fix missing glyphs when using NativeRendering"
Submodule 'tests/auto/qml/ecmascripttests/test262' (/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative-testsuites.git) registered for path 'tests/auto/qml/ecmascripttests/test262'
fatal: repository '/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative-testsuites.git' does not exist
fatal: clone of '/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative-testsuites.git' into submodule path '/var/cache/boulder/upstreams/git/invent.kde.org/qt/qt/qtdeclarative.git/tests/auto/qml/ecmascripttests/test262' failed
Failed to clone 'tests/auto/qml/ecmascripttests/test262'. Retry scheduled
fatal: repository '/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative-testsuites.git' does not exist
fatal: clone of '/var/cache/boulder/upstreams/staging/git/invent.kde.org/qt/qt/qtdeclarative-testsuites.git' into submodule path '/var/cache/boulder/upstreams/git/invent.kde.org/qt/qt/qtdeclarative.git/tests/auto/qml/ecmascripttests/test262' failed
Failed to clone 'tests/auto/qml/ecmascripttests/test262' a second time, aborting
[13:48:45] ERROR     Exception: Failed to fetch/checkout submodules
[13:48:45] ERROR     Stage failure: fetch-upstreams

However I can use git to do this manually and it works:

git clone https://invent.kde.org/qt/qt/qtdeclarative.git
Cloning into 'qtdeclarative'...
remote: Enumerating objects: 461615, done.
remote: Counting objects: 100% (6328/6328), done.
remote: Compressing objects: 100% (2707/2707), done.
remote: Total 461615 (delta 4340), reused 5257 (delta 3583), pack-reused 455287
Receiving objects: 100% (461615/461615), 156.10 MiB | 20.67 MiB/s, done.
Resolving deltas: 100% (373918/373918), done.

cd qtdeclarative/
git checkout 05c3f4921d81fb1b0eb497515c24ae56221d1e0a
Note: switching to '05c3f4921d81fb1b0eb497515c24ae56221d1e0a'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 05c3f4921d Revert "Fix missing glyphs when using NativeRendering"

git submodule update --init --recursive
Submodule 'tests/auto/qml/ecmascripttests/test262' (https://invent.kde.org/qt/qt/qtdeclarative-testsuites.git) registered for path 'tests/auto/qml/ecmascripttests/test262'
Cloning into '/home/reilly/snekpit/src/qtdeclarative/tests/auto/qml/ecmascripttests/test262'...
Submodule path 'tests/auto/qml/ecmascripttests/test262': checked out '6b0c42c63c2492bd0a7a96d3179d122b5f71793f'
ReillyBrogan commented 1 year ago

CC @GZGavinZhao

GZGavinZhao commented 1 year ago

I think this is caused by a faulty .gitmodules file, which tries to fetch a submodule outside of the project root instead of an actual "remote":

[submodule "tests/auto/qml/ecmascripttests/test262"]
    path = tests/auto/qml/ecmascripttests/test262
    url = ../qtdeclarative-testsuites.git
    branch = snapshot-20180312-3c69133-based

I think it may be better to just use HTTP source, as that is the single git submodule qtdeclarative has. Is there a reason you're using Git source?

GZGavinZhao commented 1 year ago

Hmm I see what's going on. The remote's relative url would point to https://invent.kde.org/qt/qt/qtdeclarative-testsuites.git, but we're doing two cloning. The second cloning (which is when submodules are fetched) clones from the staging path into the final path, which cause the testsuite's url to evaluate to a local path. Maybe an option should be added to clone into final path directly.

ReillyBrogan commented 1 year ago

I think it may be better to just use HTTP source, as that is the single git submodule qtdeclarative has. Is there a reason you're using Git source?

The KDE folk have specifically asked that the functionality for generating a tarball from a git commit not be used for qt5 as it causes a greater amount of load on their systems compared to doing a git clone. There's otherwise no http source for qt5 as it's only maintained as a git branch and doesn't see releases (you pretty much just use whatever the latest commit on the kde/5.15 branch is at the time of building). Note that if you look into that repo you'll see tags but those are deceiving as those are the upstream tags while the repo is maintained as a series of backports on top of the upstream QT tag.

TLDR this unfortunately does need to work with git cloning otherwise I'm going to need to host my own converted git -> http source which is icky.

ReillyBrogan commented 1 year ago

Maybe an option should be added to clone into final path directly.

If I understand you right this might be the best option.

GZGavinZhao commented 1 year ago

Maybe an option should be added to clone into final path directly.

If I understand you right this might be the best option.

I think so too. I'll do that then.

GZGavinZhao commented 1 year ago

Also the hanging issue will be gone when the libgit2 back-end is used. That code is all finished and tested, I just need to clean it up.

ReillyBrogan commented 1 year ago

How do I actually use this? I've tried defining my source as the following and it's not working (guessing based off of the code changes, but this should really be documented):

upstreams   :
    - git|https://invent.kde.org/qt/qt/qtdeclarative.git:
        ref: 05c3f4921d81fb1b0eb497515c24ae56221d1e0a
        # Relative submodules
        staging: false
GZGavinZhao commented 1 year ago

Not in front of the computer, but I recall that in my testing I used staging: no. Your inference is correct, I think it's just some yaml weirdness.