purpleidea / mgmt

Next generation distributed, event-driven, parallel config management!
https://purpleidea.com/tags/mgmtconfig/
GNU General Public License v3.0
3.67k stars 315 forks source link

git submodule can't have trailing slash #602

Open NiceGuyIT opened 4 years ago

NiceGuyIT commented 4 years ago

Versions:

Description:

git clone --recursive https://github.com/purpleidea/mgmt fails to clone submodules that have trailing slashes. The complete output is in the spoiler below.

diff --git a/.gitmodules b/.gitmodules
index 7a73a8f..578c2a7 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,6 +1,6 @@
 [submodule "vendor/github.com/coreos/etcd"]
        path = vendor/go.etcd.io/etcd
-       url = https://github.com/coreos/etcd/
+       url = https://github.com/coreos/etcd
 [submodule "vendor/google.golang.org/grpc"]
        path = vendor/google.golang.org/grpc
        url = https://github.com/grpc/grpc-go
@@ -15,7 +15,7 @@
        url = https://github.com/purpleidea/go-systemd
 [submodule "vendor/honnef.co/go/augeas"]
        path = vendor/honnef.co/go/augeas
-       url = https://github.com/dominikh/go-augeas/
+       url = https://github.com/dominikh/go-augeas
 [submodule "vendor/github.com/grpc-ecosystem/go-grpc-prometheus"]
        path = vendor/github.com/grpc-ecosystem/go-grpc-prometheus
        url = https://github.com/grpc-ecosystem/go-grpc-prometheus
git clone --recursive https://github.com/purpleidea/mgmt ```text Cloning into 'mgmt'... remote: Enumerating objects: 17, done. remote: Counting objects: 100% (17/17), done. remote: Compressing objects: 100% (11/11), done. remote: Total 11529 (delta 8), reused 15 (delta 6), pack-reused 11512 Receiving objects: 100% (11529/11529), 4.29 MiB | 9.90 MiB/s, done. Resolving deltas: 100% (7992/7992), done. Submodule 'vendor/github.com/purpleidea/distribution' (https://github.com/purpleidea/distribution) registered for path 'vendor/github.com/docker/distribution' Submodule 'vendor/github.com/purpleidea/docker' (https://github.com/purpleidea/docker) registered for path 'vendor/github.com/docker/docker' Submodule 'vendor/github.com/purpleidea/go-connections' (https://github.com/docker/go-connections) registered for path 'vendor/github.com/docker/go-connections' Submodule 'vendor/github.com/grpc-ecosystem/go-grpc-prometheus' (https://github.com/grpc-ecosystem/go-grpc-prometheus) registered for path 'vendor/github.com/grpc-ecosystem/go-grpc-prometheus' Submodule 'vendor/github.com/grpc-ecosystem/grpc-gateway' (https://github.com/grpc-ecosystem/grpc-gateway) registered for path 'vendor/github.com/grpc-ecosystem/grpc-gateway' Submodule 'vendor/github.com/hashicorp/go-multierror' (https://github.com/hashicorp/go-multierror) registered for path 'vendor/github.com/hashicorp/go-multierror' Submodule 'vendor/github.com/ugorji/go' (https://github.com/ugorji/go) registered for path 'vendor/github.com/ugorji/go' Submodule 'vendor/github.com/coreos/etcd' (https://github.com/coreos/etcd/) registered for path 'vendor/go.etcd.io/etcd' Submodule 'vendor/google.golang.org/grpc' (https://github.com/grpc/grpc-go) registered for path 'vendor/google.golang.org/grpc' Submodule 'vendor/gopkg.in/fsnotify.v1' (https://gopkg.in/fsnotify.v1) registered for path 'vendor/gopkg.in/fsnotify.v1' Submodule 'vendor/honnef.co/go/augeas' (https://github.com/dominikh/go-augeas/) registered for path 'vendor/honnef.co/go/augeas' Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/docker/distribution'... remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 19756 (delta 0), reused 0 (delta 0), pack-reused 19755 Receiving objects: 100% (19756/19756), 12.77 MiB | 6.07 MiB/s, done. Resolving deltas: 100% (10679/10679), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/docker/docker'... remote: Enumerating objects: 249678, done. remote: Total 249678 (delta 0), reused 0 (delta 0), pack-reused 249678 Receiving objects: 100% (249678/249678), 133.72 MiB | 7.95 MiB/s, done. Resolving deltas: 100% (167395/167395), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/docker/go-connections'... remote: Enumerating objects: 38, done. remote: Counting objects: 100% (38/38), done. remote: Compressing objects: 100% (31/31), done. remote: Total 8779 (delta 11), reused 23 (delta 7), pack-reused 8741 Receiving objects: 100% (8779/8779), 2.06 MiB | 6.39 MiB/s, done. Resolving deltas: 100% (369/369), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/grpc-ecosystem/go-grpc-prometheus'... remote: Enumerating objects: 21, done. remote: Counting objects: 100% (21/21), done. remote: Compressing objects: 100% (18/18), done. remote: Total 468 (delta 3), reused 10 (delta 2), pack-reused 447 Receiving objects: 100% (468/468), 137.15 KiB | 2.54 MiB/s, done. Resolving deltas: 100% (222/222), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/grpc-ecosystem/grpc-gateway'... remote: Enumerating objects: 40, done. remote: Counting objects: 100% (40/40), done. remote: Compressing objects: 100% (31/31), done. remote: Total 8984 (delta 19), reused 14 (delta 8), pack-reused 8944 Receiving objects: 100% (8984/8984), 3.70 MiB | 7.75 MiB/s, done. Resolving deltas: 100% (6229/6229), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/hashicorp/go-multierror'... remote: Enumerating objects: 38, done. remote: Counting objects: 100% (38/38), done. remote: Compressing objects: 100% (27/27), done. remote: Total 167 (delta 15), reused 30 (delta 11), pack-reused 129 Receiving objects: 100% (167/167), 55.40 KiB | 1.91 MiB/s, done. Resolving deltas: 100% (77/77), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/github.com/ugorji/go'... remote: Enumerating objects: 37, done. remote: Counting objects: 100% (37/37), done. remote: Compressing objects: 100% (28/28), done. remote: Total 4235 (delta 15), reused 17 (delta 9), pack-reused 4198 Receiving objects: 100% (4235/4235), 4.26 MiB | 11.91 MiB/s, done. Resolving deltas: 100% (3308/3308), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/go.etcd.io/etcd'... ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of 'https://github.com/coreos/etcd/' into submodule path '/home/user/go/src/github.com/purpleidea/mgmt/vendor/go.etcd.io/etcd' failed Failed to clone 'vendor/go.etcd.io/etcd'. Retry scheduled Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/google.golang.org/grpc'... remote: Enumerating objects: 62, done. remote: Counting objects: 100% (62/62), done. remote: Compressing objects: 100% (56/56), done. remote: Total 20237 (delta 13), reused 15 (delta 1), pack-reused 20175 Receiving objects: 100% (20237/20237), 10.59 MiB | 9.14 MiB/s, done. Resolving deltas: 100% (12553/12553), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/gopkg.in/fsnotify.v1'... remote: Enumerating objects: 1383, done. remote: Total 1383 (delta 0), reused 0 (delta 0), pack-reused 1383 Receiving objects: 100% (1383/1383), 363.31 KiB | 934.00 KiB/s, done. Resolving deltas: 100% (887/887), done. Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/honnef.co/go/augeas'... ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of 'https://github.com/dominikh/go-augeas/' into submodule path '/home/user/go/src/github.com/purpleidea/mgmt/vendor/honnef.co/go/augeas' failed Failed to clone 'vendor/honnef.co/go/augeas'. Retry scheduled Cloning into '/home/user/go/src/github.com/purpleidea/mgmt/vendor/go.etcd.io/etcd'... ERROR: Repository not found. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. fatal: clone of 'https://github.com/coreos/etcd/' into submodule path '/home/user/go/src/github.com/purpleidea/mgmt/vendor/go.etcd.io/etcd' failed Failed to clone 'vendor/go.etcd.io/etcd' a second time, aborting ```
purpleidea commented 4 years ago

What git version are you using git --version that causes this? It seems to work fine in CI.

NiceGuyIT commented 4 years ago

I found the problem. My ~/.gitconfig forces all GitHub URLs to use SSH instead of HTTPS. This was in response to go get using HTTPS for private repos and asking for the credentials. Golang's FAQ explains Why does "go get" use HTTPS when cloning a repository? I chose the second solution by adding this to my ~/.gitconfig.

[url "ssh://git@github.com/"]
    insteadOf = https://github.com/

With that configuration, it's impossible to add a submodule with a trailing slash. Using SSH explicitly, a trailing slash fails.

$ git submodule add git@github.com:NiceGuyIT/sub-child-2.git/ child-with-slash-4
Cloning into '/home/user/Projects/sub-parent-3/child-with-slash-4'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:NiceGuyIT/sub-child-2.git/' into submodule path '/home/user/Projects/sub-parent-3/child-with-slash-4' failed

Removing the trailing slash works.

$ git submodule add git@github.com:NiceGuyIT/sub-child-2.git child-with-slash-3
Cloning into '/home/user/Projects/sub-parent-3/child-with-slash-3'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (4/4), done.

One could argue git could be improved by removing the trailing slash when the https:// scheme is replaced by the ssh:// scheme.

As for this issue, it's up to you if you want to remove the slash or not. Considering this repo uses go get and some of your user base could have the same workaround mentioned previously, I vote to remove the trailing slashes.

NiceGuyIT commented 4 years ago

This was reported to git's mailing list but haven't received a response. I'm leaving it open because while I don't see any cons, there is one small edge case it helps.

purpleidea commented 4 years ago

On Mon, Jun 22, 2020 at 1:36 PM David Randall notifications@github.com wrote:

This was reported to git's mailing list but haven't received a response. I'm leaving it open because while I don't see any cons, there is one small edge case it helps.

Nice work, let us know what you find out if I miss following the thread!

Thanks, James