search

gomods / athens

A Go module datastore and proxy
https://docs.gomods.io
MIT License
4.41k stars 496 forks source link

404 from @latest and @v/list when pre-filling the disk cache #1416

Open brian73286 opened 4 years ago

brian73286 commented 4 years ago

Describe the bug I get 404 from @v/list and @latest endpoints for modules when using athens on a computer that has never cached internet resources, after following instructions for pre-filling the disk cache.

Error Message When I GET $HOST:$PORT/github.com/julienschmidt/httprouter/@v/list my client receives 404 and kubectl logs shows:

time="2019-10-09T12:34:56Z" level=info msg="exit status 1: go list -m: github.com/julienschmidt/httprouter@latest: git ls-remote -q origin in /tmp/athens<DECIMAL>/pkg/mod/cache/vcs/<HEXADECIMAL>: exit status 128:\n\tfatal: unable to access 'https://github.com/julienschmidt/httprouter/': Could not resolve host: github.com\n" http-method=GET http-path=/github.com/julienschmidt/httprouter/@v/list http-url=/github.com/julienschmidt/httprouter/@v/list kind="Not Found" module= operation=download.ListHandler ops="[download.ListHandler pool.List protocol.List vcsLister.List]" version=

When I GET $HOST:$PORT/github.com/julienschmidt/httprouter/@latest my client receives 404 and kubectl logs shows:

time="2019-10-09T12:34:56Z" level=info msg="exit status 1: go list -m: github.com/julienschmidt/httprouter@latest: git ls-remote -q origin in /tmp/athens<DECIMAL>/pkg/mod/cache/vcs/<HEXADECIMAL>: exit status 128:\n\tfatal: unable to access 'https://github.com/julienschmidt/httprouter/': Could not resolve host: github.com\n" http-method=GET http-path=/github.com/julienschmidt/httprouter/@latest http-url=/github.com/julienschmidt/httprouter/@latest kind="Not Found" module= operation=download.LatestHandler ops="[download.LatestHandler pool.Latest protocol.Latest vcsLister.List]" version=

To Reproduce Using a computer that has never connected to the internet, follow the instructions at https://github.com/gomods/athens/pull/1414 as they were on 2019-10-08 using the v0.2.0 tag of github.com/plexsystems/pacmod to pack up the v1.3.0 tag of github.com/julienschmidt/httprouter for athens. That creates 3 files: go.mod, v1.3.0.info, v1.3.0.zip. Rename v1.3.0.zip to source.zip. Using kubernetes run the image gomods/athens:v0.6.1 with persistent storage mounted at /opt/athens/storage and overwrite the image's included config by mounting yours at /config/config.toml. Your config.toml contains:

StorageType = "disk"
[Storage]
    [Storage.Disk]
        RootPath = "/opt/athens/storage"

using kubectl cp copy the 3 files, one of was renamed, to be under /opt/athens/storage/github.com/julienschmidt/httprouter/v1.3.0/ and see they default to 0640 file permissions. It doesn't seem to make a different but restart the pod for good measure in case the files need to exist in persistent storage before the pod launches.

Expected behavior I expect output similar to what is documented at https://docs.gomods.io/intro/protocol/

Environment (please complete the following information):

Additional context These work perfectly fine: GET $HOST:$PORT/github.com/julienschmidt/httprouter/@v/v1.3.0.info returns valid JSON with sane values GET $HOST:$PORT/github.com/julienschmidt/httprouter/@v/v1.3.0.mod returns text that matches the contents of the module's go.mod file even though the Content-Type is application/json GET $HOST:$PORT/github.com/julienschmidt/httprouter/@v/v1.3.0.zip serves the zip archieve for download and when downloaded and extracted the files & directory hierarchy matches the module.

This entire issue was hand-typed so feel free to ask if anything in the error messages is a typo.

brian73286 commented 4 years ago

I'm guessing either athens is failing to do some sort of logic or I need my config.toml to specify GOPRIVATE, GOPROXY, and/or GONOPROXY in athen's GoBinaryEnvVars property. If it is the latter then that should also be documented.

jpreese commented 4 years ago

I've made a couple comments on PR #1414 relating to the guide, and my gut says it's because of the step that has you renaming from v1.3.0 to source.

When you run go mod download, it returns a .zip that is named the same as the version.

brian73286 commented 4 years ago

I just tried renaming /opt/athens/storage/github.com/julienschmidt/httprouter/v1.3.0/source.zip to /opt/athens/storage/github.com/julienschmidt/httprouter/v1.3.0/v1.3.0.zip. The @v/list and @latest endpoints continue to return 404 and now GET $HOST:$PORT/github.com/julienschmidt/httprouter/@v/v1.3.0.zip also returns 404 instead of offering a file for download. I've reverted this rename.

jpreese commented 4 years ago

Interesting. I'll keep an eye on this thread, because if its supposed to be source, then I'd rather pacmod generate a file called source.zip so users aren't stuck renaming it.

I was just going off of the fact that when you use the Go toolchain, it tosses a v1.3.0.zip file in your mod cache.

arschles commented 4 years ago

if its supposed to be source, then I'd rather pacmod generate a file called source.zip so users aren't stuck renaming it.

@jpreese iirc naming it source.zip was an arbitrary decision. We can of course have Athens look for a $VERSION.zip on the backend, but we'll need to retain backward compatibility too, so it doesn't sound worth it to me.

On the other hand, Instead of changing pacmod completely, what do you think about adding a new command called pacmod import-athens or something similar? That comment could do the same thing as packmod pack and then rename & move the files into the disk storage root.

jpreese commented 4 years ago

I'm not opposed to that. I just wanted to make sure we weren't missing anything with the naming of source.zip.

Because the go mod toolchain uses $VERSION.zip, I think it would be worthwhile for Athens to do the same, if even at a major version bump.

arschles commented 4 years ago

if even at a major version bump

We haven't done one of these. We haven't made the plunge to a v1.0.0 yet either 😉

That said, would you be up for submitting a PR that does the backward compatible $VERSION.zip support in the disk driver? No pressure if not - I'll pick it up.

Also, if you're open to it, I'm totally open to adding a pacmod import-athens command. Let me know.

jpreese commented 4 years ago

I can tackle that @arschles.

For pacmod, my gut would be to add an output / name sort of parameter that would allow the user to name their artifacts. I think that gives us what we need without needing to create a special purpose command for Athens.

Though, if we update Athens to support both $VERSION and source (for backwards shenanigans), I don't think we'd need it, right? We'd just let pacmod continue to archive modules as $VERSION.zip, and Athens would be able to handle it.

brian73286 commented 4 years ago

I just verified this issue also affects the new v0.7.0 release in the same ways.

I tried copying source.zip to v1.3.0.zip, then restarting the container so it is running with four files under /opt/athens/storage/github.com/julienschmidt/httprouter/v1.3.0/

Where the last two are copies of each other. The hope was that instead of providing a file named one way or the other, try giving it both. This did not fix the 404's I get for /@v/list and /@latest. I think the problem may be deeper than the naming of that file.

jpreese commented 4 years ago

I don't seem to be having the same issues that you're running into, @brian73286. The list command should be working fine as that just looks at the folders. The name of the artifacts shouldn't matter.

This was how I was spinning up the container:

docker run -e "ATHENS_STORAGE_TYPE=disk" -e "ATHENS_DISK_STORAGE_ROOT=tmp" --rm -it -v C:\Users\jpree\Desktop\athens:/tmp -p 3000:3000 gomods/athens

Are your storage type and storage path vars set?

jpreese commented 4 years ago

@arschles I took some time today to look into some of the internals of how Athens handles different filesystems, and I don't necessarily see a problem with how Athens is storing the modules.

I think it's a little confusing that it doesn't mimic the same file structure as the Go tooling, but from an API point of view, the file structure shouldnt/doesn't matter. I was able to spin up a local Athens proxy and run the commands fine, including having my archive named source.zip.

arschles commented 4 years ago

@brian73286 I haven't been active in this issue for some time, sorry about that. If the issue is still happening, let me know and I'll help debug some more.

kmalloc commented 4 years ago

I am experiencing same issue, and the reproducing step is exactly the same as described in this thread.

following works fine:

GET $HOST:$PORT/github.com/kmalloc/l5/@v/v1.0.0.info GET $HOST:$PORT/github.com/kmalloc/l5/@v/v1.0.0.mod GET $HOST:$PORT/github.com/kmalloc/l5/@v/v1.0.0.zip

however, @latest and @v/list return 404.

athens I am using is 0.7.2. go 1.12.5

image

arschles commented 4 years ago

@brian73286 @jpreese @kmalloc I couldn't reproduce this again on v0.7.2. Here is how I ran Athens:

ecomaz :: ~/Desktop/athens-volume » docker run -v $ATHENS_STORAGE:/var/lib/athens \ 
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \ 
   -p 3000:3000 \
   gomods/athens:v0.7.2

I did an export ATHENS_STORAGE=$PWD before I ran this. Forgot to copy that in above.

That command runs Athens with the default config.toml file, so it's using disk storage and the directory is on my host.

Here are my logs for what I did:

ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@latest      
{"Version":"v1.3.0","Time":"2019-09-29T23:21:22Z"}
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@list  
404 page not found
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/list
v1.0.0
v1.1.0
v1.2.0
v1.3.0%                                                                                                                          
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/v1.3.0.mod
module github.com/julienschmidt/httprouter

go 1.7
ecomaz :: ~/Desktop/athens-volume » ls -al
total 0
drwxr-xr-x   3 aaronschlesinger  staff   96 Mar 20 10:32 .
drwx------@ 26 aaronschlesinger  staff  832 Mar 20 10:23 ..
drwxr-xr-x@  3 aaronschlesinger  staff   96 Mar 20 10:32 github.com

After these commands, Athens has downloaded the module into my $PWD/athens-volume directory. It looks like this:

ecomaz :: ~/Desktop/athens-volume » ls github.com/julienschmidt/httprouter/v1.3.0/                                         127 ↵
go.mod      source.zip  v1.3.0.info

So it's certainly important that the source code is called source.zip. Athens won't work with it being v1.3.0.zip. Same goes for go.mod instead of v1.3.0.mod.

@kmalloc I think that the issue you mentioned is a bit different because github.com/kmalloc/l5 doesn't seem to exist. Did you load that code into your Athens storage using pacmod?

Let me know if you'd like logs for Athens, happy to share them.

Finally, I tried the docs at https://docs.gomods.io/configuration/prefill-disk-cache/ and noticed some things are off:

Let me know if you have any ideas regarding the second bullet

kmalloc commented 4 years ago

@brian73286 @jpreese @kmalloc I couldn't reproduce this again on v0.7.2. Here is how I ran Athens:

ecomaz :: ~/Desktop/athens-volume » docker run -v $ATHENS_STORAGE:/var/lib/athens \ 
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \ 
   -p 3000:3000 \
   gomods/athens:v0.7.2

I did an export ATHENS_STORAGE=$PWD before I ran this. Forgot to copy that in above.

That command runs Athens with the default config.toml file, so it's using disk storage and the directory is on my host.

Here are my logs for what I did:

ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@latest      
{"Version":"v1.3.0","Time":"2019-09-29T23:21:22Z"}
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@list  
404 page not found
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/list
v1.0.0
v1.1.0
v1.2.0
v1.3.0%                                                                                                                          
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/v1.3.0.mod
module github.com/julienschmidt/httprouter

go 1.7
ecomaz :: ~/Desktop/athens-volume » ls -al
total 0
drwxr-xr-x   3 aaronschlesinger  staff   96 Mar 20 10:32 .
drwx------@ 26 aaronschlesinger  staff  832 Mar 20 10:23 ..
drwxr-xr-x@  3 aaronschlesinger  staff   96 Mar 20 10:32 github.com

After these commands, Athens has downloaded the module into my $PWD/athens-volume directory. It looks like this:

ecomaz :: ~/Desktop/athens-volume » ls github.com/julienschmidt/httprouter/v1.3.0/                                         127 ↵
go.mod      source.zip  v1.3.0.info

So it's certainly important that the source code is called source.zip. Athens won't work with it being v1.3.0.zip. Same goes for go.mod instead of v1.3.0.mod.

@kmalloc I think that the issue you mentioned is a bit different because github.com/kmalloc/l5 doesn't seem to exist. Did you load that code into your Athens storage using pacmod?

Let me know if you'd like logs for Athens, happy to share them.

Finally, I tried the docs at https://docs.gomods.io/configuration/prefill-disk-cache/ and noticed some things are off:

  • pacmod pack $VERSION . doesn't work. It needs to be pacmod pack $MODULE_NAME $VERSION. I'll fix that in a PR shortly
  • Something is off either on permissions or the file structure. I suspect it's permissions because the directory structure in the docs looks the same as what Athens shows. I need to look into more detail

Let me know if you have any ideas regarding the second bullet

I changed my repo later, please refere to the attached screenshot in my previous reply. the issue is the same.

kmalloc commented 4 years ago

@brian73286 @jpreese @kmalloc I couldn't reproduce this again on v0.7.2. Here is how I ran Athens:

ecomaz :: ~/Desktop/athens-volume » docker run -v $ATHENS_STORAGE:/var/lib/athens \ 
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \ 
   -p 3000:3000 \
   gomods/athens:v0.7.2

I did an export ATHENS_STORAGE=$PWD before I ran this. Forgot to copy that in above. That command runs Athens with the default config.toml file, so it's using disk storage and the directory is on my host. Here are my logs for what I did:

ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@latest      
{"Version":"v1.3.0","Time":"2019-09-29T23:21:22Z"}
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@list  
404 page not found
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/list
v1.0.0
v1.1.0
v1.2.0
v1.3.0%                                                                                                                          
ecomaz :: ~/Desktop/athens-volume » curl localhost:3000/github.com/julienschmidt/httprouter/@v/v1.3.0.mod
module github.com/julienschmidt/httprouter

go 1.7
ecomaz :: ~/Desktop/athens-volume » ls -al
total 0
drwxr-xr-x   3 aaronschlesinger  staff   96 Mar 20 10:32 .
drwx------@ 26 aaronschlesinger  staff  832 Mar 20 10:23 ..
drwxr-xr-x@  3 aaronschlesinger  staff   96 Mar 20 10:32 github.com

After these commands, Athens has downloaded the module into my $PWD/athens-volume directory. It looks like this:

ecomaz :: ~/Desktop/athens-volume » ls github.com/julienschmidt/httprouter/v1.3.0/                                         127 ↵
go.mod      source.zip  v1.3.0.info

So it's certainly important that the source code is called source.zip. Athens won't work with it being v1.3.0.zip. Same goes for go.mod instead of v1.3.0.mod. @kmalloc I think that the issue you mentioned is a bit different because github.com/kmalloc/l5 doesn't seem to exist. Did you load that code into your Athens storage using pacmod? Let me know if you'd like logs for Athens, happy to share them. Finally, I tried the docs at https://docs.gomods.io/configuration/prefill-disk-cache/ and noticed some things are off:

  • pacmod pack $VERSION . doesn't work. It needs to be pacmod pack $MODULE_NAME $VERSION. I'll fix that in a PR shortly
  • Something is off either on permissions or the file structure. I suspect it's permissions because the directory structure in the docs looks the same as what Athens shows. I need to look into more detail

Let me know if you have any ideas regarding the second bullet

I changed my repo later, please refere to the attached screenshot in my previous reply. the issue is the same.

I double check the file/directory permission, it seems fine, where can i get the log file of athens?

arschles commented 4 years ago

@kmalloc Athens logs to stdout but you posted some of them from your setup. Do you want to see the Athens logs from when I tried to reproduce?

jdsatava127 commented 4 years ago

We are experiencing the same issues in Docker version v0.9.0. Our Athens is running in a completely offline environment and we have pre-filled the disk cache. Calls for specific versions like go get github.com/gorilla/websocket@1.4.2 work fine. But calls for @latest result in the 404 Not Found error for https://athens/github.com/gorilla/websocket/@v/list. Athens throws this error:

INFO[2:57PM]: exit status 1: go list -m module github.com/gorilla/websocket: Get "https://proxy.golang.org/github.com/gorilla/websocket/@v/list"

We have Athens configured with the following to try to prevent proxying upstream

ATHENS_DOWNLOAD_MODE=none
ATHENS_GO_BINARY_ENV_VARS="GOPROXY=off; GOSUMDB=off; GOPRIVATE=*; GONOPROXY=*"
ATHENS_GONOSUM_PATTERNS=*

Is there a way to have Athens serve the @v/list endpoint instead of it trying to reach upstream?

danieloleynyk commented 3 years ago

We are experiencing the same issues in Docker version v0.9.0. Our Athens is running in a completely offline environment and we have pre-filled the disk cache. Calls for specific versions like go get github.com/gorilla/websocket@1.4.2 work fine. But calls for @latest result in the 404 Not Found error for https://athens/github.com/gorilla/websocket/@v/list. Athens throws this error:

INFO[2:57PM]: exit status 1: go list -m module github.com/gorilla/websocket: Get "https://proxy.golang.org/github.com/gorilla/websocket/@v/list"

We have Athens configured with the following to try to prevent proxying upstream

ATHENS_DOWNLOAD_MODE=none
ATHENS_GO_BINARY_ENV_VARS="GOPROXY=off; GOSUMDB=off; GOPRIVATE=*; GONOPROXY=*"
ATHENS_GONOSUM_PATTERNS=*

Is there a way to have Athens serve the @v/list endpoint instead of it trying to reach upstream?

same issue over here :/

drewwells commented 3 years ago

~Latest appears to work now~. Latest being unavailable is a bummer. I also see errors around timestamp evaluations on the poll worker option.

Error: handlePollIndex("/poll"): index.Client.GetVersions(ctx, 0001-01-01 00:00:00 +0000 UTC, 10): decoding JSON: json: cannot unmarshal string into Go value of type internal.IndexVersion
ericsonj commented 2 years ago

Hi, check config.toml file:

# Modes:
# 1. strict: merge VCS versions with storage versions, but fail if either of them fails.
# 2. offline: only get storage versions, never reach out to VCS.
# 3. fallback: only return storage versions, if VCS fails. Note this means that you may
# see inconsistent results since fallback mode does a best effort of giving you what's
# available at the time of requesting versions.
NetworkMode = "fallback"
DrPsychick commented 1 year ago

linking issues #1583