RPC Failed due to transport error when pushing images with DOCKER_CONTENT_TRUST=true

[blackduck-serv-builder]

This problem started yesterday (I can see successful pushes yesterday morning ~1pm EST and failures when pushing at ~7pm EST).

When pushing docker images to our hub.docker.com organization, we use DOCKER_CONTENT_TRUST=true, and have secured passwords for the repository and root. These have not changed (and I can verify different results with 'bad' passwords (it just keeps asking for the password until it's correct)).

When pushing with the correct passwords, and content trust enabled, we get the following error:

docker push myorganization/my-image:1.0.0
The push refers to a repository [docker.io/myorganization/my-image]
abcd1234: Layer already exists 
efgh5678: Layer already exists 
1.0.0: digest: sha256:xyzabc123456 size: 1361
Signing and pushing trust metadata
Enter passphrase for repository key with ID z123456: 
Failed to sign "docker.io/myorganization/my-image":1.0.0 - An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure
An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure

When pushing with content trust disabled, this process completes successfully. I believe the issue is within the content trust.

[sonrics11]

I am having this exact same problem:

09:36:47 latest: digest: sha256:93b74c5de5ee7c4cb256f42ed20ef4e74bc1397691a11b55d9d931e1a901781e size: 8215 09:36:47 Signing and pushing trust metadata 09:36:48 failed to sign docker.io/ibmzcontainers/isolated_vm_icp:latest: An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure

[jmwong]

Sorry for the inconvenience. We’re investigating.

[jmwong]

@blackduck-serv-builder @sonrics11 to help us diagnose, can you provide the output of ‘docker version’ and ‘docker info’? Thanks

[sonrics11]

`jenkins@s215bchain17:~$ docker version Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:24:34 2018 OS/Arch: linux/s390x Experimental: false

Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:34 2018 OS/Arch: linux/s390x Experimental: false jenkins@s215bchain17:~$ `

[sonrics11]

`jenkins@s215bchain17:~$ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 18.06.1-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 15 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 69663f0bd4b60df09991c08812a60108003fa340 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-134-generic Operating System: Ubuntu 16.04.5 LTS OSType: linux Architecture: s390x CPUs: 12 Total Memory: 78.54GiB Name: s215bchain17 ID: GJV2:AWDL:GS5J:4EVQ:PW76:EYEE:OWCH:SNTY:BCGM:QEAC:KLSO:XQNT Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Username: sj98ta Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support jenkins@s215bchain17:~$ `

[vennb]

Seeing this too

[drnybble]

Maybe related?

[root@paw-build1 ~]#  DOCKER_CONTENT_TRUST=1 docker pull debian:stretch
WARN[0001] Error while downloading remote metadata, using cached timestamp - this might not be the latest version available remotely
ERRO[0001] Metadata for timestamp expired
WARN[0001] Error while downloading remote metadata, using cached timestamp - this might not be the latest version available remotely
ERRO[0001] Metadata for timestamp expired
Error: remote repository docker.io/library/debian out-of-date: timestamp expired at Wed Oct 3 15:40:21 UTC 2018

[makimat]

I'm seeing the same too. Posted a stackoverflow question about how to debug this kind of issues: https://stackoverflow.com/questions/52628096/how-to-debug-failing-docker-image-signing-with-the-docker-hub-registry-notary

I think the question would be relevant even after this issue has been resolved as currently the error message given by failed signing is not helpful and there's not much documentation available about how to debug the root cause.

My env:

user@machine:~$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

user@machine:~$ docker info
Containers: 7
 Running: 0
 Paused: 0
 Stopped: 7
Images: 3585
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1021-gcp
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 2.439GiB
Name: machine
ID: 2JO6:E3V6:DJB5:5ZRU:NSDZ:KUHH:F6IK:3BJR:UWXW:ARSI:Y3OB:4BKN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: sgsupport
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[sonrics11]

I don't know if someone is working on this, but my signing "DOCKER_CONTENT_TRUST=1" started working.

[jmwong]

Hi all, sorry for the delayed response. We did identify an issue yesterday with Docker Hub Notary and have resolved the issue as of 5.44am PT today: https://status.docker.com/pages/history/533c6539221ae15e3f000031. I apologize for the prolonged outage. We are looking into improving our processes to avoid this in the future.

Please reach out to us at support@docker.com if you're still seeing issues with this.

[sonrics11]

@jmwong Thank you for all the support !!!

[makimat]

Would you mind checking my question about what to do about debugging similar problems in the future? The RPC-related error message given by the push command was not that helpful.

[mhart]

I'm seeing this now too – exact same error with no more details.

failed to sign docker.io/<image>: An error occurred during validation: rpc error: code = 14 desc = grpc: RPC failed fast due to transport failure

[mhart]

The issue seems to be fixed for me now. Was broken for many hours though – and I didn't change anything on my end. Just started working.

Again, would be INCREDIBLY helpful to have some more detail in these error messages so we can at least track down what the issue might be.