cmd/go: support self-hosted fossil in 'go get'

[schollz]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/zns/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/zns/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build451328689=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am self-hosting fossils using a standalone server. I have a very simple "hello, world" example here: https://fossil.schollz.com/hello-world which I am trying to go get. This is a functional fossil, as you can clone and run it:

> fossil clone https://fossil.schollz.com/hello-world hello-world.fossil
> fossil open hello-world.fossil
> go run main.go
hello, world

I then tried to go get this fossil.

What did you expect to see?

I expected to be able to run

> go get fossil.schollz.com/hello-world.fossil

which would automatically download hello-world as a fossil into my $GOPATH.

What did you see instead?

Despite it being run using the canonical fossil server command, I am unable to go get it. Here are the attempts:

>  go get fossil.schollz.com/hello-world
package fossil.schollz.com/hello-world: unrecognized import path "fossil.schollz.com/hello-world" 
(parse https://fossil.schollz.com/hello-world?go-get=1: no go-import meta tags ())
> go get fossil.schollz.com/hello-world.fossil
package fossil.schollz.com/hello-world.fossil: cannot download, fossil.schollz.com/hello-world 
uses insecure protocol

It seems that the current fossil support only supports chiselapp.com - I refer to this change that added support.

Is there not yet support for self-hosted fossils?

[AlexRouSg]

The correct command is go get -insecure fossil.schollz.com/hello-world.fossil Your host is defaulting to http and go get doesn't like that so to force it to use http you need to pass the -insecure flag

[schollz]

Thanks for your help. It actually still doesn't work for me though.

> go get -insecure fossil.schollz.com/hello-world.fossil
can't load package: package fossil.schollz.com/hello-world.fossil: 
no Go files in /home/zns/go/src/fossil.schollz.com/hello-world.fossil

I don't think it's processing the Go files correctly, as there is a go file in the main branch at the root folder.

But it does seem to create a directory, although there is also nothing in it (including the fossil repo):

> tree $GOPATH/src/fossil.schollz.com
/home/zns/go/src/fossil.schollz.com
└── hello-world.fossil

1 directory, 0 files

[AlexRouSg]

Using raw fossil commands results in errors, is the repo setup correctly and can you clone manually with fossil?

[AlexRouSg]

Sorry, I haven't used fossil before and was using the wrong url. Looks like go get is too.

cc @ksshannon @ianlancetaylor

go get is calling fossil clone fossil.schollz.com/hello-world however the correct usage would be fossil clone http://fossil.schollz.com/hello-world or fossil clone https://fossil.schollz.com/hello-world

[ksshannon]

@schollz, there could be a few thinks going on here. If you have a public URL I can test against, it'd probably be easier. You should set the tag to something like:

<meta name="go-import" content="fossil.schollz.com/hello-world fossil https://fossil.schollz.com">

or something similar.

This is done a little differently depending on if your fossil is before or after the drafts enhancement was implemented.

I'd be glad to dig into it, but I need to be able to replicate. For posterity, please report your fossil version as well. We'll get it figured out.

[AlexRouSg]

@ksshannon The URL is public btw, ohh and I tried 2.6 direct download from https://www.fossil-scm.org/xfer/uv/download.html

[ksshannon]

@AlexRouSg, I believe it's the head metadata. It works the same with git hosting.

[AlexRouSg]

@ksshannon The metadata error would explain the error with go get -insecure fossil.schollz.com/hello-world, however go get -insecure fossil.schollz.com/hello-world.fossil doesn't complain about metadata and progress until fossil errors out.

[AlexRouSg]

go get -insecure -v fossil.schollz.com/hello-world.fossil
# cd .; fossil 
Usage: fossil COMMAND ...
   or: fossil help           -- for a list of common commands
   or: fossil help COMMAND   -- for help with the named command

Commands and filenames may be passed on to fossil from a file
by using:

    fossil --args FILENAME ...

Each line of the file is assumed to be a filename unless it starts
with '-' and contains a space, in which case it is assumed to be
another flag and is treated as such. --args FILENAME may be used
in conjunction with any other flags.
# cd .; fossil 
Usage: fossil COMMAND ...
   or: fossil help           -- for a list of common commands
   or: fossil help COMMAND   -- for help with the named command

Commands and filenames may be passed on to fossil from a file
by using:

    fossil --args FILENAME ...

Each line of the file is assumed to be a filename unless it starts
with '-' and contains a space, in which case it is assumed to be
another flag and is treated as such. --args FILENAME may be used
in conjunction with any other flags.
fossil.schollz.com/hello-world.fossil (download)
# cd .; fossil clone fossil.schollz.com/hello-world /home/asdf/go/src/fossil.schollz.com/hello-world.fossil/.fossil
unknown repository: fossil.schollz.com/hello-world
package fossil.schollz.com/hello-world.fossil: exit status 1

[ksshannon]

@AlexRouSg go get -insecure fossil.schollz.com/hello-world.fossil is a 404, same as wget http://fossil.sholiz.com/hello-world.fossil so I may need to clean up some of the std out dump, but I think that is a different issue. I'll look further if the metadata tag doesn't solve this issue, and then we'll move on.

[AlexRouSg]

@ksshannon Yes it's a 404, but looking at the -v output, it calls fossil clone fossil.schollz.com/hello-world so go get is stripping the .fossil.

This is going into undocumented territory ... someone should clarify whether go get myurl.com/repo.vcs needs to actually exist or if it's just a way to denote vcs and the url used is myurl.com/repo

If it's the latter then the issue is fossil clone myurl.com/repo is not a valid fossil command, it requires the protocol part as well and should instead run fossil clone http://myurl.com/repo or fossil clone https://myurl.com/repo

[ksshannon]

@AlexRouSg No offense, but I'd like to solve @schollz 's issue first. We can file more issues after if we need to fix fossil support. I'm sure there are issues, but I'm not sure they should fall under this one.

[schollz]

@ksshannon Thanks for helping too.

> fossil version
This is fossil version 2.6 [9718f3b078] 2018-05-04 12:56:42 UTC

The URL (https://fossil.schollz.com/hello-world) is indeed live and a working fossil. You can try it yourself with:

> fossil clone https://fossil.schollz.com/hello-world hello-world.fossil
> fossil open hello-world.fossil
> go run main.go
hello, world

Let me know if I can provide any other information, or if you can point me in the right direction I can try digging too. I'm not so familiar with the Go source yet though.

[ksshannon]

@schollz read the docs at:

https://golang.org/cmd/go/#hdr-Remote_import_paths

And set the meta tag it mentions for you repository. You should be able to add it at:

https://fossil.schollz.com/hello-world/setup_skinedit

[schollz]

@ksshannon Awesome. It works.

Here's the header I added:

<meta name="go-import" content="fossil.schollz.com/hello-world fossil https://fossil.schollz.com/hello-world">

It is now go get-able:

> go get -u -v fossil.schollz.com/hello-world
Fetching https://fossil.schollz.com/hello-world?go-get=1
Parsing meta tags from https://fossil.schollz.com/hello-world?go-get=1 (status code 200)
get "fossil.schollz.com/hello-world": found meta tag get.metaImport{Prefix:"fossil.schollz.com/hello-world", VCS:"fossil", RepoRoot:"https://fossil.schollz.com/hello-world"} at https://fossil.schollz.com/hello-world?go-get=1
fossil.schollz.com/hello-world (download)
> tree $GOPATH/src/fossil.schollz.com
/home/zns/go/src/fossil.schollz.com
hello-world
  main.go
  README.md

1 directory, 2 files

Thanks for your help @ksshannon and @AlexRouSg