Are "go get" vanity domain redirects possible with existing Caddyfile directives?

tildeleb commented 8 years ago

Has anyone been able to handle go-get vanity domain redirects with existing Caddyfile directives?

See here: https://docs.google.com/document/d/1jVFkZTcYbNLaTxXD9OcGfn7vYv5hWtPx9--lTx1gPMs/edit https://golang.org/cmd/go#hdr-Remote_import_paths (Sorry I don't know why the fragment doesn't work, search for "Remote import paths")

Example: You issue the command go get leb.io/hrff and it get's redirected (in my case) to go get github.com/tildleb/hrff

The url sent to the web server at leb.io looks like:

http://leb.io/package?go-get=1

I want to respond with:

var redir = []string{
    "<!DOCTYPE html>\n",
    "<head>\n",
        "<meta http-equiv=\"refresh\" content=\"0; URL='http://godoc.org/leb.io/%s'\">\n",
        "<meta name=\"go-import\" content=\"leb.io/%s git https://github.com/tildeleb/%s\">\n",
    "</head>\n",
}

where %s is the package/path name less the leading /. This is actual code from hacked up web server I've tested it with.

I think template directives come closest, but there are two issues:

The match needs to be on go-get=1 and not the path.
There is no file extension

If a match can be made I think the placeholder ${file} will capture the package name?

Assuming there isn't a way to handle this today, what's the right approach? I am willing to write a new directive for it, but templates seem like the right approach to me.

mholt commented 8 years ago

Do you mean that you only want to show that output if the query string == ?go-get;1? (Also, don't you mean go-get=1?

tildeleb commented 8 years ago

Here is a proposal.

Most of caddy directives match on path. My proposal is to add a more flexible match syntax that uses the keyword match.
A url will be parsed into a path, query, and fragment. In all places were a path is now accepted for matching you can use the keyword match instead. match looks like a function call with the following arguments:
```
match(path, query, fragment)  
```
where all the parameters are regular expressions and all must match (i.e. "and"). So given what I want to do the line in the caddy file would look like:
```
template match(,"go-get=1")
```
I propose that the extensions used the template be extended to also allow regular expressions. So something like:
```
template match(,"go-get=1") ".*" // this matches any extension and a path with no extension
```

or maybe

     template match(,"go-get=1") extensions(".*") // this matches any extension and a path with no extension

tildeleb commented 8 years ago

yes only show the output when the query string is "go-get=1". Sorry for the typo above. Yes the query string is "go-get=1". I updated my other comments to correct this.

mholt commented 8 years ago

Okay, so with regards to your question, I think this (or something like it) should do what you need.

$ caddy templates

<!DOCTYPE html>
<head>
    <meta http-equiv="refresh" content="0; URL='https://godoc.org/leb.io{{.URL.Path}}'">
    <meta name="go-import" content="leb.io git https://github.com/tildeleb{{.URL.Path}}">
</head>

I read the docs you linked to, and I'm not sure that it's imperative to only add the meta tag to the site's root if that query string is present; but if it is, I suppose you can use something like

{{if eq .URL.RawQuery "go-get=1"}} ... {{end}}

tildeleb commented 8 years ago

Thanks. I understand the template but not how to configure caddy to use it. I still don't understand how the url is matched.

What template directive goes on the Caddyfile?
What should the name of the template file be and what extension if any?

mholt commented 8 years ago

I would make a file at whatever the package path is, and plop that HTML in there. It doesn't need a file extension. For the Caddyfile, you can just say templates or if you want to be more specific you can say templates /package/path or whatever.

tildeleb commented 8 years ago

I think you are missing the crux of the problem. I make lots of packages in go. I don't want to have to add an html file or edit the Caddyfile every time I create a new package at "leb.io". What I want Caddy to do is match any http/https request to the leb.io domain that has a query of go-get=1 and return the template file that you describe above with the approrpriate substitutions.

I don't think Caddy as it is currently constructed can do this because all directive matching occurs only on the path part of the url. That is why I proposed a more comprehensive matching system above that can match on path, url, and fragment.

As this issue demonstrates not every request can be matching on path.

If you want I could file an issue requesting an enhancement to the existing matching algorithm for directives.

mholt commented 8 years ago

I think the query string is irrelevant here; as far as I understand, you want to to render that HTML template every time a request is made to a package path; whether it has go-get=1 is in the query string or not? I infer this because this line in the template:

<meta http-equiv="refresh" content="0; URL='https://godoc.org/leb.io{{.URL.Path}}'">

only works in browsers (AFAIK the go get client doesn't redirect to the godoc page).

Query strings are supposed to convey data with a request, not change the resource being requested - that's the job of the path. (Also, specifying the user agent is the job of a request header, not a query string. It seems strange that would be the only way to detect go get?) Thus I am hesitant to allow matching on query strings... I am not against the idea of making Caddy more useful, but I want to make sure it doesn't bend over backward and violate some basic, standard conventions in the process.

captncraig commented 8 years ago

I have a bit of experience doing this, and the go get tool is a bit finicky. I have done this for our packages which are bosun.org/** but resolve to github.com/bosun-monitor/bosun/**. What we were doing is simply including the meta tag in our base template and always serving it (even on 404s). This generally works since any package resolves to the same repo. If you have separate repos on the domain you may have other problems so this may not apply.

The problem I have is that when the go get tool fetches over https it does not parse meta tags, so the get fails. This required us to use generate to make an html page for each package in our repo. It kinda sucked. There is an open issue on the go project about this and rsc acknowledged it should be fixed, but who knows really.

If I had a perfect world I would want a directive that did the following:

vanity github.com/myname/mypackage myvanitydomain.com

this would on startup (and maybe periodically) scan a list of packages under that hierarchy. If it gets a request for a valid package with or without the go-get query it would serve a page with the meta tag, and a reload to godoc.org. See https://bosun.org/collect for an example of what we serve.

That sounds like a lot of work though. If serving over http, just add the meta tag to your core template. If over https, unfortunately you have fewer options.

mholt commented 8 years ago

Can this be resolved with new features added to the rewrite directive? It can match based on more than just path.

tildeleb commented 8 years ago

Sorry I was gone for a few months but now I am back on this. Will investigate rewrite this week and post the results.

jwkohnen commented 8 years ago

@tildeleb Any news? I have the same use case and could help out. I've tried to configure this with rewrite a while ago, but didn't get anywhere. In doubt I'd implement a special middleware for this.

captncraig commented 8 years ago

I think this may require a custom plugin to handle all possible cases and repo layouts.

Consider packages:

my.tld/foo
my.tld/bar
my.tld/somethingelse/foo

It is possible these both link to the same repository (subpackages), or different ones. How do I know if I should point at github.com/myname/someOtherRepo/foo or github.com/myname/foo?

captncraig commented 8 years ago

Config could be something like:

#single package to a static repository
goget /foo github.com/myname/myrepo

#multiple packages with a simple convention of "mydomain.tld/repo = github.com/myname/repo"
goget /:repo github.com/myname/:repo

#multiple subpackages all in a single repo.
goget * github.com/myname/somerepo

Not sure what kind of syntax would be preferrable on the wildcardy bits there.

mholt commented 8 years ago

@captncraig You may be right that a plugin would be the best way to do this. They're easy to write and easy to get, and it's a pretty specific need that most web server users don't have.

@tildeleb I'm closing this now since, after discussion, it's not really an issue in Caddy per-se (and I'm trying to get through the backlog), but I have added a link to it on this thread on the community forum so that other developers who are interested in writing a plugin for Caddy can see this as an idea. There's lots of good stuff here and I think someone who wants to put in a few hours can make it happen :+1:

davidlazar commented 8 years ago

I've described how I setup Caddy to support custom import domains here:

https://davidlazar.org/blog/003-golang-custom-import-domain-with-caddy.txt

mholt commented 8 years ago

@davidlazar Thanks! Mind if I quote it here for future readers in case it ever disappears?

Source: https://davidlazar.org/blog/003-golang-custom-import-domain-with-caddy.txt?

https://vuvuzela.io is served by Caddy, a webserver that handles LetsEncrypt certificates automatically.

I'd like to setup Caddy so that Go programs can use the onionbox package from Github (https://github.com/vuvuzela/crypto) by typing:

import "vuvuzela.io/crypto/onionbox"

I added these lines to my Caddyfile:

templates
rewrite {
    to {path} {path}/ /gopkg.html?path={path}
}

This enables Go-style templates and routes requests to the gopkg.html file if the requested path does not exist in the filesystem.

The gopkg.html file is a template that returns the required tag:

{{- $path := .URL.Query.Get "path" -}}
{{- if len $path | ne 0 -}}
{{- $repoRoot := index (.Split $path "/") 1 -}}
<!doctype html>
<html>
<head>
  <meta name="go-import" content="vuvuzela.io/{{$repoRoot}} git https://github.com/vuvuzela/{{$repoRoot}}">
  {{/*<meta http-equiv="refresh" content="0; url=https://godoc.org/vuvuzela.io{{$path}}">*/}}
</head>
<body>
  Are you looking for a Go package?
  Maybe it exists on <a href="https://github.com/vuvuzela/{{$repoRoot}}">Github</a>,
  or on <a href="https://godoc.org/vuvuzela.io{{$path}}">GoDoc</a>.
</body>
</html>
{{end}}

It assumes that the first path component ("crypto" in "vuvuzela.io/crypto/onionbox"), is the name of the repository that should be cloned from Github. This mimics the behavior of the go-import-redirector tool.

The second <meta> line can be uncommented to redirect browsers to the GoDoc page for that package.

References:

July 2016

mholt commented 7 years ago

This is now implemented with a plugin: https://caddyserver.com/docs/http.gopkg

caddyserver / caddy

Are "go get" vanity domain redirects possible with existing Caddyfile directives? #358