Metadata mismatch between `Release` and `Packages.bz`

Thanks for writing and sharing the apt repository plugin.

What are you trying to do? Provide a persistent proxy of upstream apt repositories for ubuntu which doesn't delete old packages. Today we ran into an issue with a proxy repository for http://dl.google.com/linux/chrome/deb/. The file metadata/dists/stable/main/Release was stale. It was "last verified" on 2018-03-21. It contained an invalid hash sum for metadata/dists/stable/main/binary-amd64/Packages.bz2. That file was "last verified" on 2018-03-23.

Workstations configured with the nexus server in /etc/apt/sources.list were failing to apt-get update with:

Err:26 https://nexus.redacted.com/repository/apt-dl.google.com-chrome-stable stable/main amd64 Packages
Hash Sum mismatch

I manually deleted the Release file through the nexus UI and after doing an apt-get update from a workstation this file was pulled into nexus at the correct (newer) version.

What feature or behavior is this required for? Reproducible system image building
How could we solve this issue? (Not knowing is okay!) I'm not sure exactly but it seems that possibly metadata files are cached with a policy that ignores the relationship between them. Without knowing the full mechanisms of the debian repository format, my suggestion would be to consider the file Release and all of the Packages.bz-like files it points to as a single "unit". Update them all together, or none at all.
Anything else? I don't know exactly how the plugin manages the metadata cache and I don't understand the meaning of the "maximum metadata age" parameter. Does the plugin just exactly proxy the file contents from the upstream server? Or does it parse the upstream file and build a local database?

If it just proxies the upstream server files, is it not possible to install old packages that are currently cached in nexus but have been removed from the upstream?

This is a bit of a tricky issue. As you've discovered, there is certainly the possibility for this to happen. The problem is that a Release file generally references a lot of package files, many of which won't be used and are rather large. So the options are:

1.) Whenever we fetch a Release file, force a download of all the corresponding Package files as well, which can take a long time

2.) Whenever we fetch a Release file, invalidate the metadata cache, and hope that the client is going to fetch the metadata files it cares about in the next couple of seconds

Right now we do option 2 (https://github.com/sonatype-nexus-community/nexus-repository-apt/blob/master/src/main/java/net/staticsnow/nexus/repository/apt/internal/proxy/AptProxyFacet.java#L80) since for the use cases I cared about when I wrote this code the assumption in option 2 was fine.

If you take a snapshot, that will actually do option 1: download a snapshot of the Release and Packages files, it just takes longer than perhaps we would want to block the Release request for.

If someone wanted to create a PR that added an option to force download of package indexes whenever the Release file was fetched that would be a great addition I think.

If it just proxies the upstream server files, is it not possible to install old packages that are currently cached in nexus but have been removed from the upstream?

It does just proxy and store whatever files are fetched. If you want to freeze the manifests at a point in time you can create a snapshot.

Oh I see. Now that I understand the problem, I think the snapshot feature makes more sense to me (I didn't really understand how it worked). Since the snapshot feature does #1 I think that is probably what I want to be using for my use case then.

I've followed the instructions on the README of how to create a snapshot, but how does one use a snapshot. After creating curl -u <redacted>:<redacted> -X MKCOL https://nexus.<redacted>.com/repository/<redacted>/snapshots/20180410

Then I try adding snapshots/20180410 to the end of the deb line in etc/apt/sources.list but apt-get update exits with 404 not found

I think you need to add the snapshot to the end of the url in sources.list, not the end of the line. The url part of the line should look exactly like the MKCOL url.

Ah, yes sorry, that is what I mean (end of the URL). Trying to dig in a little, for example I am able to fetch InRelease for the main URL but not for the snapshot URL:

josh@cookie:tmp$ wget https://nexus.<redacted>.com/repository/<repo>/dists/xenial/InRelease
--2018-04-10 09:27:33--  https://nexus.<redacted>.com/repository/<repo>/dists/xenial/InRelease
Resolving nexus.<redacted>.com (nexus.<redacted>.com)... 54.201.101.27
Connecting to nexus.<redacted>.com (nexus.<redacted>.com)|54.201.101.27|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2738 (2.7K) [text/plain]
Saving to: ‘InRelease’

InRelease                                            100%[=====================================================================================================================>]   2.67K  --.-KB/s    in 0s      

2018-04-10 09:27:33 (210 MB/s) - ‘InRelease’ saved [2738/2738]

josh@cookie:tmp$ wget https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/dists/xenial/InRelease
--2018-04-10 09:27:43--  https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/dists/xenial/InRelease
Resolving nexus.<redacted>.com (nexus.<redacted>.com)... 54.201.101.27
Connecting to nexus.<redacted>.com (nexus.<redacted>.com)|54.201.101.27|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-04-10 09:27:43 ERROR 404: Not Found.

When I put https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410/ as the URL in my sources.list

deb [arch=amd64,i386] https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410/ xenial main

I get the following from apt-get update:

josh@cookie:sonatype_nexus$ python apt_nexus_test.py --arch amd64 --suite xenial /tmp/rootfs cuda-toolkit-9.0
Ign:1 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial InRelease
Ign:2 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial Release
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons
Err:3 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 Packages
  404  snapshot/20180410/dists/xenial/main/binary-amd64/Packages.xz
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshot/20180410 xenial/main DEP-11 64x64 Icons

Try adding a / to the end of the URL, both when you issue the MKCOL and in the sources list. Also, it's snapshots, not snapshot

Ah, whoops on shapshot. Still doesn't seem to do the trick. I appreciate your help so far. Any thoughts on what else I could be doing wrong?

josh@cookie:Notes$ curl -u admin:<redacted> -X MKCOL https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/
josh@cookie:Notes$ echo $?
0

josh@cookie:Notes$ wget https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/dists/xenial/InRelease
--2018-04-10 15:14:19--  https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/dists/xenial/InRelease
Resolving nexus.<redacted>.com (nexus.<redacted>.com)... 54.201.101.27
Connecting to nexus.<redacted>.com (nexus.<redacted>.com)|54.201.101.27|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2018-04-10 15:14:19 ERROR 404: Not Found.

apt-get update output with

deb [arch=amd64,i386] https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/ xenial main

Ign:1 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial InRelease
Ign:2 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial Release
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Ign:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Err:3 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 Packages
  404  Not Found
Ign:4 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main i386 Packages
Ign:5 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main all Packages
Ign:6 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en_US
Ign:7 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main Translation-en
Ign:8 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main amd64 DEP-11 Metadata
Ign:9 https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial/main DEP-11 64x64 Icons
Reading package lists... Done
W: The repository 'https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410 xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/dists/xenial/main/binary-amd64/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used

For reference, if I remove snapshots/20180410/ I get the following:

Get:1 https://nexus.<redacted>.com/repository/<repo> xenial InRelease [2,738 B]
Get:2 https://nexus.<redacted>.com/repository/<repo> xenial/main amd64 Packages [22.8 kB]
Get:3 https://nexus.<redacted>.com/repository/<repo> xenial/main all Packages [1,723 B]
Fetched 27.2 kB in 0s (42.5 kB/s)    
Reading package lists... Done
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'https://nexus.<redacted>.com/repository/<repo> xenial InRelease' doesn't support architecture 'i386'
Reading package lists... Done
Building dependency tree... Done

Oh!

Even though curl exits with 0, I think that MKCOL is failing:

josh@cookie:Notes$ curl -v -u admin:<redacted> -X MKCOL https://nexus.<redacted>.com/repository/<repo>/snapshots/20180410/
*   Trying 54.201.101.27...
* Connected to nexus.<redacted>.com (54.201.101.27) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: nexus.<redacted>.com (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: CN=nexus.<redacted>.com
*    start date: Thu, 22 Feb 2018 14:40:30 GMT
*    expire date: Wed, 23 May 2018 14:40:30 GMT
*    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*    compression: NULL
* ALPN, server accepted to use http/1.1
* Server auth using Basic with user 'admin'
> MKCOL /repository/<repo>/snapshots/20180410/ HTTP/1.1
> Host: nexus.<redacted>.com
> Authorization: Basic <redacted>
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.10.3 (Ubuntu)
< Date: Wed, 11 Apr 2018 14:43:38 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
* Authentication problem. Ignoring this.
< WWW-Authenticate: BASIC realm="Sonatype Nexus Repository Manager"
< 
* Connection #0 to host nexus.<redacted>.com left intact

The response is HTTP/1.1 401 Unauthorized. I tried changing to 20180411 and it still failed with Unauthorized. Perhaps there is some permission that I didn't add to admin that I need to add? Also, I don't think it matters but nexus is behind an nginx reverse proxy. The nginx configuration includes

  # nexus repositories, credentials managed by nexus itself
  location /repository {
    auth_request off;
    proxy_pass http://localhost:8081;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

  }

and the response authentication realm is "Sonatype Nexus Repository Manager" so I think the nginx setup isn't getting in the way.

Also, I tried the following directly on the machine itself:

curl -v -u admin:<redacted> -X MKCOL http://localhost:8081/repository/<repo>/snapshots/20180411/

and got

*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8081 (#0)
* Server auth using Basic with user 'admin'
> MKCOL /repository/<repo>/snapshots/20180411/ HTTP/1.1
> Host: localhost:8081
> Authorization: Basic <redacted>
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Date: Wed, 11 Apr 2018 14:54:49 GMT
< Server: Nexus/3.9.0-01 (OSS)
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
* Authentication problem. Ignoring this.
< WWW-Authenticate: BASIC realm="Sonatype Nexus Repository Manager"
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

Notabily, the request.log entry does not show a username:

127.0.0.1 - - [11/Apr/2018:21:57:10 +0000] "MKCOL /repository/<repo>/snapshots/20180411 HTTP/1.1" 401 0 4 "curl/7.47.0"

for other entries in this log, the second dash is a username like "admin" or "josh"

Hm... it seems to be some kind of interaction with the "Remote User Token" configuration. If I add -H "<my-RUT>: admin" to the curl command, then it succeeds with:

< HTTP/1.1 201 Created
< Date: Wed, 11 Apr 2018 22:30:15 GMT
< Server: Nexus/3.9.0-01 (OSS)
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
< X-Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
< Content-Length: 0

Is that somehow intended? Posting files to the repository does not require adding the RUT header (normal -u <user>:<pass> works just fine with curl).

Except that for some repositories it yields a null pointer exception:

2018-04-11 22:50:03,783+0000 WARN  [qtp2003082976-2648]  admin org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: MKCOL /repository/apt-packages.cloud.google.com-xenial/snapshots/201804011
java.lang.NullPointerException: null
    at net.staticsnow.nexus.repository.apt.internal.snapshot.AptSnapshotFacetSupport.collectSnapshotItems(AptSnapshotFacetSupport.java:126)
    at net.staticsnow.nexus.repository.apt.internal.snapshot.AptSnapshotFacetSupport.createSnapshot(AptSnapshotFacetSupport.java:65)
    at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
    at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
    at net.staticsnow.nexus.repository.apt.internal.snapshot.AptSnapshotHandler.handleSnapshotAdminRequest(AptSnapshotHandler.java:81)
    at net.staticsnow.nexus.repository.apt.internal.snapshot.AptSnapshotHandler.handle(AptSnapshotHandler.java:67)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:42)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
    at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:211)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:173)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
    at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
    at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:530)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:748)

@DarthHater - Any idea on what could cause the "Remote User Token" problem? I don't use that, and the plugin doesn't do anything specific with it, so pretty clueless on that.

As far as the exception: I looked at the code, and it seems like this happens if there's no Release or InRelease file in the repo you're trying to snapshot. Is it possible that those repos have a misconfigured upstream URL, or are empty hosted repos? In any case, the error reporting should be better here, I'll create an issue to improve it.

Here's the metadata tree for "packages.cloud.google.com" as viewed on our nexus deployment.

So it has InRelease but not Release. One thing I'm noticing is that the folder under dists is cloud-sdk-xenial. Must this folder match the "distribution" field in the apt repository configuration? I currently have that field set to `xenial':

My goal today is to try to disable RUT and see if snapshot creation works using curl -u. I will update with results of that test.

Ok yeah, I disabled RUT and now curl -u works too. I think I may be observing another issue, however. I have a proxy setup for http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu/. I took a snapshot today and got:

however InRelease contains references that weren't captured as part of the snapshot. In particular:

9d5ab4c9278e71d6a2a9ce6ad700ae573253b80a             4457 main/i18n/Translation-en.gz
 1a291bffa51d59dae0e0032e47add3491a4176e1              201 main/i18n/Index
 a6f07a9351280efc916e446e31b84c62c7c4a1a9             4224 main/i18n/Translation-en.xz
 4fb1336b21c044c8c78ed2306ceae239dd81552b            35052 main/i18n/Translation-en
 aac2ae8b37e657882d890d921d2bd82714d933dc            21441 main/source/Sources

So when I add the following to my etc/apt/sources.list:

deb [arch=amd64] https://nexus.<redacted>.com/repository/apt-ppa.launchpad.net-graphics-drivers-xenial/snapshots/20180412/ xenial main

and run apt-get update, I get

Ign:21 https://nexus.<redacted>.com/repository/apt-ppa.launchpad.net-graphics-drivers-xenial/snapshots/20180412 xenial/main Translation-en
Err:21 https://nexus.<redacted>.com/repository/apt-ppa.launchpad.net-graphics-drivers-xenial/snapshots/20180412 xenial/main Translation-en
  404  Not Found
Fetched 53.4 kB in 1s (29.9 kB/s)
Reading package lists... Done
E: Failed to fetch https://nexus.<redacted>.com/repository/apt-ppa.launchpad.net-graphics-drivers-xenial/snapshots/20180412/dists/xenial/main/i18n/Translation-en  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

I appreciate all the help you've provided on this thread so far. To summarize I think we've identified (potentially) three issues so far:

HTTP basic authentication doesn't work for snapshot creation if RUT is enabled
The repository layout of packages.cloud.google.com leads to a null pointer exception (or it's layout is wrong)
snapshots do not capture all of the files required by apt (or my configuration is wrong)

I would be happy to break these out into separate issue reports on github if you'd like. At this point they appear to be unrelated to each other.

Yeah, definitely think they're unrelated.

For point 2, I filed an issue to improve the logging, but the root cause is almost certainly what you mentioned: the distribution in your config does not match the folder name in dists. The snapshot code only looks for files to snapshot in the directory named by the configuration, and in this case finds none, so there's an error.

For point 3: it's expected. The i18n and translations are huuuuuge, and I wanted to keep snapshots as small as possible. For my use cases these files are unneeded, and the install works fine without them. Seems like a reasonable thing to put behind a config (i.e. include translations in snapshots and include sources in snapshots) if it's important to your use case to have these files.

Ok great. For point 2: I fixed the "Distribution" field and now the snapshot works for that one. Thanks! For point 3: Oh, i see. Is there some way to configure apt to not require these files then? Maybe that's a question for google. I'll see what I can find to convince apt to ignore them. I think the config option sounds like a great idea.

Edit: disable apt translations: https://askubuntu.com/a/74663

I guess appstream (DEP-11) files are also not snapshotted? I needed to do this as well in order to use a snapshot of a the proxy for us.archive.ubuntu.com.

Alright, I've successfully implemented apt configurations using snapshots for both workstation configurations and container/image builds... so I think I'm pretty much covered for my use case here. Thanks so much for writing the plugin and for taking the time to help me through these issues. I'm comfortable closing this issue unless you wish to keep it open for some reason.

I know that @DarthHater mentioned updating documentation for this plugin. I think there were a few "gotchas" covered in this thread that might be worth adding to docs:

metadata files are fetched and cached as plain-old-files. See info above about InRelease or Release fetches invalidate Packages. This can lead to stale / mismatched metadata files for some use cases.
Use snapshots to get a complete metadata tree at a particular point in time and this can be used for reproduce-able executions of apt-get
If RUT is enabled then basic auth can't be used for snapshot creation
Only the "distribution" directory is snapshotted (perhaps that is the only meaning of this configuration field really?)
Snapshots don't include some large metadata like translations or DEP-11 files (see links above for how to deal with this fact)

APT is now part of Nexus Repository Manager. Version 3.17.0 includes the APT plugin by default. If this is still an issue if using 3.17.0 or later please file an issue at https://issues.sonatype.org/. Links to the new source code location are in the top level README.md

sonatype-nexus-community / nexus-repository-apt

Metadata mismatch between `Release` and `Packages.bz` #46