search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.16k stars 17.69k forks source link

cmd/go: special case for chiselapp.com interferes with '.fossil' extension #31014

Open urandom2 opened 5 years ago

urandom2 commented 5 years ago

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build279953534=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to clone a fossil repo from chiselapp using the .fossil to test self-hosted fossil repos. It looks like the chiselapp.com/user/kyle/repository/fossilgg format is handled separately in cmd/go/internal/get/vcs.go.

$ go get chiselapp.com/user/kyle/repository/fossilgg.fossil

What did you expect to see?

The go get succeeds and documentation can be listed via go doc for confirmation:

$ go doc -u -all -src fossilgg
package fossilgg.fossil // import "chiselapp.com/user/kyle/repository/fossilgg.fossil"

CONSTANTS

const greeting = "Hello from fossil!"

FUNCTIONS

func Hello() string {
        return greeting
}

What did you see instead?

The fossil clone fails:

# cd .; fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil /home/user/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil/.fossil
redirect limit exceeded
server returned an error - clone aborted
package chiselapp.com/user/kyle/repository/fossilgg.fossil: exit status 1

This also leaves behind an empty fossilgg.fossil directory in $GOPATH that has the side effect of preventing future calls to go get:

$ go get chiselapp.com/user/kyle/repository/fossilgg.fossil
can't load package: package chiselapp.com/user/kyle/repository/fossilgg.fossil: no Go files in /home/user/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil
bcmills commented 5 years ago

Are you sure that the repository can be cloned in general? If so, what command did you use to verify that, and in what way (if any) does it differ from the fossil clone command printed by go get?

How does the behavior change, if at all, in module mode (with GO111MODULE=on and the working directory within a module)?

ksshannon commented 5 years ago

@arnottcr you need to supply the path to the repository with out the .fossil extension:

go get chiselapp.com/user/kyle/repository/fossilgg

urandom2 commented 5 years ago

@kamanson: I am aware that the clone works without the .fossil suffix, however per the spec I should be able to clone with either, right?

@bcmills: while the output is not identical, modules appear to preserve the failing behaviour:

$ GO111MODULE=on go get chiselapp.com/user/kyle/repository/fossilgg.fossil 
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go get chiselapp.com/user/kyle/repository/fossilgg.fossil: fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil .fossil in /tmp/go/pkg/mod/cache/vcs/351aec3555812167b2e4d08a2027fe8df20a3e623d2d5ffa60cfba15a9da2437: exit status 1:
        redirect limit exceeded
        server returned an error - clone aborted
$ GO111MODULE=off go get chiselapp.com/user/kyle/repository/fossilgg.fossil
# cd .; fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil /tmp/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil/.fossil
redirect limit exceeded
server returned an error - clone aborted
package chiselapp.com/user/kyle/repository/fossilgg.fossil: exit status 1

It looks like an underlying fossil clone failure may be the root cause:

$ fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil fossilgg.fossil
redirect with status 302 to http://chiselapp.com/notfound
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect limit exceeded
Clone done, sent: 6998  received: 13314  ip: 2607:f1c0:84b:4b02:68e8:7a3f:2812:3fc0
server returned an error - clone aborted
$ fossil clone https://chiselapp.com/user/kyle/repository/fossilgg bare
Round-trips: 2   Artifacts sent: 0  received: 16
Clone done, sent: 572  received: 14731  ip: 10.0.1.187
Rebuilding repository meta-data...
  100.0% complete...
Extra delta compression... 
Vacuuming the database... 
project-id: e85b3a05650ffbb291dd1f398ab5b602d1859efb
server-id:  199d504c2d1c6d90f045d5f4e906fb93ff0b526d
admin-user: user (password is "password")

For completeness, I have included output for the successful go get calls without vcs suffix:

$ GO111MODULE=off go get chiselapp.com/user/kyle/repository/fossilgg


$ GO111MODULE=on go get chiselapp.com/user/kyle/repository/fossilgg
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding chiselapp.com/user/kyle/repository/fossilgg latest
go: downloading chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: extracting chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db



Further investigation shows that this vcs suffix does work from some endpoints:




however this one uses a meta tag, so I am unsure if it counts:
<meta name="go-import" content="fossil.boisestate.edu/fossilgg fossil https://fossil.boisestate.edu/fossilgg">


$ GO111MODULE=on go get fossil.boisestate.edu/fossilgg.fossil
go: finding fossil.boisestate.edu/fossilgg.fossil latest
go: downloading fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db
go: extracting fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db



$ GO111MODULE=off go get fossil.boisestate.edu/fossilgg.fossil


$ GO111MODULE=on go get fossil.boisestate.edu/fossilgg
go: finding fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db
go: finding fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding fossil.boisestate.edu/fossilgg latest
go: downloading fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db
go: extracting fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db



$ GO111MODULE=off go get fossil.boisestate.edu/fossilgg


            

        

            

            

                ksshannon
                commented
                 5 years ago            

            

                
@arnottcr can you point to 




@kamanson: I am aware that the clone works without the .fossil suffix, however per the spec I should be able to clone with either, right?




Chisel can handle redirects however it wants, same as any other code host.  It doesn't work with git either:


$ go get github.com/golang/text.git
go get github.com/golang/text.git: invalid version control suffix in github.com/ path


Can you point me to the spec you are referring to?
            

        

            

            

                bcmills
                commented
                 5 years ago            

            

                


Can you point me to the spec you are referring to?




I'm guessing the section in https://tip.golang.org/cmd/go/#hdr-Remote_import_paths starting at “For code hosted on other servers […]”.


But note that that is an either/or, not both: in general we expect each repository to have one canonical path, not multiple variations. You should import via only that canonical path.
            

        

            

            

                urandom2
                commented
                 5 years ago            

            

                
Yeah, that was the paragraph I was citing as spec, but my reading was that the package/module owner had the choice of either option, not the entire hosting provider. So if a vcs suffix was provided, it would trump the http/meta driven resolution.


I guess I do not see the benefit of preventing a package/module owner from using github.com/user/repo.git/pkg if they want to? But if this is working as expected, I can understand that, it just feels like a hidden rule set[0], that only serves to confuse users.


0] This rule only applies for GitHub and JazzHub, and leverages get.noVCSSuffix.
            

        

            

            

                bcmills
                commented
                 5 years ago            

            

                


my reading was that the package/module owner had the choice of either option, not the entire hosting provider




chiselapp.com is literally a special case:
https://github.com/golang/go/blob/df557fe3ea94ff8888abd6a6ab4cc5d5351d77a6/src/cmd/go/internal/get/vcs.go#L1023-L1029


We should probably try to migrate all of those special cases over to go-import meta tags, but it doesn't seem critical enough to do right now.
            

        

            

            

                urandom2
                commented
                 5 years ago            

            

                
sgtm; would you mind if I worked on a patch, or is there a larger refactor in the works that would nullify such efforts? (e.g. modules)
            

        

            

            

                ksshannon
                commented
                 5 years ago            

            

                
If you add the .fossil extension, fossil itself fails with the redirect error:


fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil tmp.fossil



Click...
redirect with status 302 to http://chiselapp.com/notfound
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/



redirect limit exceeded
Clone done, sent: 7018  received: 13314  ip: 74.208.146.128
server returned an error - clone aborted


I don't understand how this can be fixed.  Would we drop the .fossil extension?  This seems like even more of a special case to me.  Fossil itself is hosted at https://fossil-scm.org/, which can't have an extension.
            

        

            

            

                bcmills
                commented
                 5 years ago            

            

                
@arnottcr, I think that deprecating any (or all) of the cmd/go special-cases (presumably replacing them with go-import tags) would need to go through the proposal committee.


In particular, we would need buy-in from folks who currently maintain or use those hosting services, since either they would need to add the go-import tags for their servers or folks would need to migrate off of those services.
            

        

            

            

                bcmills
                commented
                 5 years ago            

            

                
I'd like to take a step back, though: what's the concrete problem you're trying to address using the .fossil extension?
            

        

            

            

                urandom2
                commented
                 5 years ago            

            

                
I was looking into .fossil support for gddo, but was unable to find non-chiselapp fossil hosting[0]. Thus, I was hoping that chiselapp.com/user/<user>/repository/<repo>.fossil would be a decent substitute, however it failed to go get so I opened a ticket.


I do not have a workflow that is broken by this issue, and realistically it would probably be more useful to implement gddo support as a provider/special case like github or bitbucket, since chisel has somewhat standard APIs.


I was not aware when I opened this ticket that github.com/user/repo.git also failed, so I am happy to leave this to languish until such a time that it makes sense to clean up all, or at least most, the special cases.


0] I think this is because fossil is intended as a github in a binary, so most people run their own, however I was trying to find something official.