search

notaryproject / notary

Notary is a project that allows anyone to have trust over arbitrary collections of data
Apache License 2.0
3.23k stars 508 forks source link

Fatal: Client is offline #1161

Closed brandongohwh closed 7 years ago

brandongohwh commented 7 years ago

I'm new to Docker Notary and followed the guide for installing the server, however I run into the error where I cannot get notary to connect to the server.

This output is from my computer:

root@brandon-Inspiron-5459:/home/brandon/Desktop# ./notary -D -s https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443 -d ~/.notary list docker.io DEBU[0000] Using the following trust directory: /root/.notary DEBU[0000] Trusting 1 certs
ERRO[0000] could not reach https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443: Get https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443/v2/: read tcp 10.217.242.29:39934->54.169.113.36:4443: read: connection reset by peer INFO[0000] continuing in offline mode
DEBU[0000] No yubikey found, using alternative key storage: no library found

As shown, I can ping and get an acknowledgement from the server but the connection resets when I use notary to query. All TCP and UDP ports for EC2 are open.

Using the same command from EC2 yields no problems:

ubuntu:~$ ./notaryex -s https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443 list docker.io/trust -D DEBU[0000] Configuration file not found, using defaults DEBU[0000] Using the following trust directory: /home/ubuntu/.notary DEBU[0000] No yubikey found, using alternative key storage: no library found DEBU[0000] received HTTP status 404 when requesting root.

These are the docker containers currently on the (clean) server:

root@ip-172-31-23-92:/home/ubuntu# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 04f1a4f11d75 notary_server "/usr/bin/env sh -..." 22 minutes ago Up 22 minutes 0.0.0.0:4443->4443/tcp, 0.0.0.0:32775->8080/tcp notary_server_1 f53874e1dd60 notary_signer "/usr/bin/env sh -..." 22 minutes ago Up 22 minutes notary_signer_1 699a27332b67 mariadb:10.1.10 "/docker-entrypoin..." 22 minutes ago Up 22 minutes 3306/tcp notary_mysql_1

And both computers have the same notary client version (Since everything was downloaded and compiled today):

root@ip-172-31-23-92:/home/ubuntu# ./notaryex version notary Version: 0.4.3 Git commit: 9211198

(FYI) I am also using a self-signed rootCA, notary-signer and notary-server certificate

Here is the configuration file for config.json:

{ "remote_server": { "url": "https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443", "root_ca": "root-ca.crt" } }

Please help as I need to get notary server up for my research work :)

brandongohwh commented 7 years ago

I solved the connection reset problem since it was a timeout issue with the network that I'm using getting the same error when setting up a collection.

On my computer it shows the following output: root@brandon-Inspiron-5459:/home/brandon/Desktop# ./notary -s https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443 init ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test -D DEBU[0000] Using the following trust directory: /root/.notary DEBU[0000] Trusting 1 certs
ERRO[0000] could not reach https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443: Get https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443/v2/: read tcp 10.217.242.29:49574->54.169.113.36:4443: read: connection reset by peer INFO[0000] continuing in offline mode
DEBU[0000] No yubikey found, using alternative key storage: no library found DEBU[0000] No yubikey found, using alternative key storage: no library found Root key found, using: 57de0ee9caf820ae9f47e6f695a2dbca12e7be23bf39dbd60d24957fe08d9886 DEBU[0000] No yubikey found, using alternative key storage: no library found Enter passphrase for root key with ID 57de0ee: DEBU[0004] generated ECDSA key with keyID: 6acca7b65a63164a9da175769be5100c11736618f072603ee94e67b48fd46846 DEBU[0004] generated new ecdsa key for role: targets and keyID: 6acca7b65a63164a9da175769be5100c11736618f072603ee94e67b48fd46846 Enter passphrase for new targets key with ID 6acca7b (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): Repeat passphrase for new targets key with ID 6acca7b (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): DEBU[0009] generated ECDSA key with keyID: 3591157b80344964c5c0f0ff843f82e19f8d3a383596a2f696b9b990e4e7839d DEBU[0009] generated new ecdsa key for role: snapshot and keyID: 3591157b80344964c5c0f0ff843f82e19f8d3a383596a2f696b9b990e4e7839d Enter passphrase for new snapshot key with ID 3591157 (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): Repeat passphrase for new snapshot key with ID 3591157 (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test):

On the client CLI on the server this shows up:

root@ip-172-31-23-92:/home/ubuntu# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 04f1a4f11d75 notary_server "/usr/bin/env sh -..." 2 hours ago Up 2 hours 0.0.0.0:4443->4443/tcp, 0.0.0.0:32775->8080/tcp notary_server_1 f53874e1dd60 notary_signer "/usr/bin/env sh -..." 2 hours ago Up 2 hours notary_signer_1 699a27332b67 mariadb:10.1.10 "/docker-entrypoin..." 2 hours ago Up 2 hours 3306/tcp notary_mysql_1 root@ip-172-31-23-92:/home/ubuntu# ./notaryex -s https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443 init ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test Root key found, using: ec5a4185b2e09c69985bca41ba343a67f5ab25c9eb5c812597b98a46f73ba009 Enter passphrase for root key with ID ec5a418: root@ip-172-31-23-92:/home/ubuntu# ./notaryex -s https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443 init ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test -D DEBU[0000] Using the following trust directory: /root/.notary DEBU[0000] Trusting 1 certs
DEBU[0000] No yubikey found, using alternative key storage: no library found DEBU[0000] No yubikey found, using alternative key storage: no library found Root key found, using: ec5a4185b2e09c69985bca41ba343a67f5ab25c9eb5c812597b98a46f73ba009 DEBU[0000] No yubikey found, using alternative key storage: no library found Enter passphrase for root key with ID ec5a418: DEBU[0001] generated ECDSA key with keyID: 82da78750c084c37e96eaa23e6fa77192f5cbf3b70fad30b96b4d51fb04632a1 DEBU[0001] generated new ecdsa key for role: targets and keyID: 82da78750c084c37e96eaa23e6fa77192f5cbf3b70fad30b96b4d51fb04632a1 Enter passphrase for new targets key with ID 82da787 (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): Repeat passphrase for new targets key with ID 82da787 (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): DEBU[0005] generated ECDSA key with keyID: 48b012d41803a4727ef8b5d370e64a963983c75338f820f3c224a821ef564bb0 DEBU[0005] generated new ecdsa key for role: snapshot and keyID: 48b012d41803a4727ef8b5d370e64a963983c75338f820f3c224a821ef564bb0 Enter passphrase for new snapshot key with ID 48b012d (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test): Repeat passphrase for new snapshot key with ID 48b012d (ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test):

cyli commented 7 years ago

Would you happen to have the notary server/signer logs on your compute instance (docker logs notary_server_1 and docker logs notary_signer_1? It looks like the client is connecting, but the server is 500'ing, at least when you run the CLI command on your server.

brandongohwh commented 7 years ago

This is the logs for notary_server_1:

root@ip-172-31-23-92:/home/ubuntu# docker logs notary_server_1 no change notaryserver database migrated to latest version {"level":"info","msg":"Version: 0.5.0, Git commit: 9ecd6ebe","time":"2017-05-16T02:30:59Z"} {"level":"debug","msg":"Trusting 1 certs","time":"2017-05-16T02:30:59Z"} {"level":"info","msg":"Using remote signing service","time":"2017-05-16T02:30:59Z"} {"level":"info","msg":"Using mysql backend","time":"2017-05-16T02:30:59Z"} 2017/05/16 02:30:59 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 172.19.0.2:7899: getsockopt: connection refused"; Reconnecting to {notarysigner:7899 } {"level":"info","msg":"Starting Server","time":"2017-05-16T02:30:59Z"} {"level":"info","msg":"Enabling TLS","time":"2017-05-16T02:30:59Z"} {"level":"info","msg":"Starting on :4443","time":"2017-05-16T02:30:59Z"} 2017/05/16 02:31:00 Failed to dial notarysigner:7899: connection error: desc = "transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner"; please retry. {"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T02:31:09Z"} 2017/05/16 02:31:26 http: TLS handshake error from 54.169.113.36:47236: read tcp 172.18.0.4:4443->54.169.113.36:47236: read: connection reset by peer {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"08cc883c-70f0-4c28-98fa-27a2bc2b9981","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47242","http.request.uri":"/v2/","http.request.useragent":"Go-http-client/1.1","http.response.duration":"41.856µs","http.response.status":200,"http.response.written":2,"level":"info","msg":"response completed","time":"2017-05-16T02:33:52Z"} {"docker.io/trust":"gun","go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"de5938f0-4f11-4575-8111-bb0665b2f8e9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47244","http.request.uri":"/v2/docker.io/trust/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2017-05-16T02:33:52Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"de5938f0-4f11-4575-8111-bb0665b2f8e9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47244","http.request.uri":"/v2/docker.io/trust/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2017-05-16T02:33:52Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"de5938f0-4f11-4575-8111-bb0665b2f8e9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47244","http.request.uri":"/v2/docker.io/trust/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"3.695679ms","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2017-05-16T02:33:52Z"} {"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T02:33:59Z"} 2017/05/16 02:36:19 http: TLS handshake error from 192.122.131.41:4391: read tcp 172.18.0.4:4443->192.122.131.41:4391: read: connection reset by peer 2017/05/16 02:40:37 http: TLS handshake error from 192.122.131.41:2424: read tcp 172.18.0.4:4443->192.122.131.41:2424: read: connection reset by peer 2017/05/16 02:41:34 http: TLS handshake error from 192.122.131.41:7183: read tcp 172.18.0.4:4443->192.122.131.41:7183: read: connection reset by peer 2017/05/16 02:42:45 http: TLS handshake error from 192.122.131.41:4350: read tcp 172.18.0.4:4443->192.122.131.41:4350: read: connection reset by peer 2017/05/16 02:43:28 http: TLS handshake error from 192.122.131.41:6956: read tcp 172.18.0.4:4443->192.122.131.41:6956: read: connection reset by peer 2017/05/16 02:43:35 http: TLS handshake error from 192.122.131.41:2225: read tcp 172.18.0.4:4443->192.122.131.41:2225: read: connection reset by peer 2017/05/16 02:43:55 http: TLS handshake error from 192.122.131.41:2557: read tcp 172.18.0.4:4443->192.122.131.41:2557: read: connection reset by peer 2017/05/16 02:47:33 http: TLS handshake error from 192.122.131.41:3490: read tcp 172.18.0.4:4443->192.122.131.41:3490: read: connection reset by peer {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"ca30ef19-8fdc-42d9-a733-8f0b08ef146f","http.request.method":"GET","http.request.remoteaddr":"70.39.157.194:31180","http.request.uri":"/v2/:","http.request.useragent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: \u003cnil\u003e","time":"2017-05-16T02:58:57Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"ca30ef19-8fdc-42d9-a733-8f0b08ef146f","http.request.method":"GET","http.request.remoteaddr":"70.39.157.194:31180","http.request.uri":"/v2/:","http.request.useragent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"191.247µs","http.response.status":404,"http.response.written":104,"level":"info","msg":"response completed","time":"2017-05-16T02:58:57Z"} 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:1854: read tcp 172.18.0.4:4443->192.122.131.41:1854: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:1409: read tcp 172.18.0.4:4443->192.122.131.41:1409: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:1967: read tcp 172.18.0.4:4443->192.122.131.41:1967: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:2554: read tcp 172.18.0.4:4443->192.122.131.41:2554: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:2582: read tcp 172.18.0.4:4443->192.122.131.41:2582: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:2709: read tcp 172.18.0.4:4443->192.122.131.41:2709: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:2879: read tcp 172.18.0.4:4443->192.122.131.41:2879: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:3093: read tcp 172.18.0.4:4443->192.122.131.41:3093: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:3121: read tcp 172.18.0.4:4443->192.122.131.41:3121: read: connection reset by peer 2017/05/16 03:17:09 http: TLS handshake error from 192.122.131.41:3692: read tcp 172.18.0.4:4443->192.122.131.41:3692: read: connection reset by peer 2017/05/16 03:22:04 http: TLS handshake error from 192.122.131.41:2594: read tcp 172.18.0.4:4443->192.122.131.41:2594: read: connection reset by peer 2017/05/16 03:22:46 http: TLS handshake error from 192.122.131.41:1716: read tcp 172.18.0.4:4443->192.122.131.41:1716: read: connection reset by peer 2017/05/16 03:24:11 http: TLS handshake error from 192.122.131.41:7626: read tcp 172.18.0.4:4443->192.122.131.41:7626: read: connection reset by peer 2017/05/16 03:25:35 http: TLS handshake error from 192.122.131.41:4973: read tcp 172.18.0.4:4443->192.122.131.41:4973: read: connection reset by peer 2017/05/16 03:27:01 http: TLS handshake error from 192.122.131.41:3045: read tcp 172.18.0.4:4443->192.122.131.41:3045: read: connection reset by peer 2017/05/16 03:28:34 http: TLS handshake error from 192.122.131.41:2013: read tcp 172.18.0.4:4443->192.122.131.41:2013: read: connection reset by peer {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"9127c750-a6f4-4119-aa3e-4a2e804c3d58","http.request.method":"GET","http.request.remoteaddr":"119.56.100.104:34466","http.request.uri":"/v2/","http.request.useragent":"Go-http-client/1.1","http.response.duration":"47.264µs","http.response.status":200,"http.response.written":2,"level":"info","msg":"response completed","time":"2017-05-16T03:30:21Z"} {"docker.io":"gun","go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"014ec779-831a-4df1-9a83-824b78e23fa8","http.request.method":"GET","http.request.remoteaddr":"119.56.100.104:34468","http.request.uri":"/v2/docker.io/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET root role","time":"2017-05-16T03:30:22Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"014ec779-831a-4df1-9a83-824b78e23fa8","http.request.method":"GET","http.request.remoteaddr":"119.56.100.104:34468","http.request.uri":"/v2/docker.io/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: No record found","time":"2017-05-16T03:30:22Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"014ec779-831a-4df1-9a83-824b78e23fa8","http.request.method":"GET","http.request.remoteaddr":"119.56.100.104:34468","http.request.uri":"/v2/docker.io/_trust/tuf/root.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"803.61µs","http.response.status":404,"http.response.written":116,"level":"info","msg":"response completed","time":"2017-05-16T03:30:22Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"3d06977a-ada2-4631-ad80-786de65d4928","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47248","http.request.uri":"/v2/","http.request.useragent":"Go-http-client/1.1","http.response.duration":"40.671µs","http.response.status":200,"http.response.written":2,"level":"info","msg":"response completed","time":"2017-05-16T04:48:23Z"} {"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:48:29Z"} {"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test":"gun","go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"4badabe8-2949-4d40-9f23-c96570bd7ee9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47250","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","level":"error","msg":"500 GET timestamp key: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:48:33Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"4badabe8-2949-4d40-9f23-c96570bd7ee9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47250","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","level":"error","msg":"unknown: unknown error: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:48:33Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"4badabe8-2949-4d40-9f23-c96570bd7ee9","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47250","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"905.745µs","http.response.status":500,"http.response.written":70,"level":"info","msg":"response completed","time":"2017-05-16T04:48:33Z"} 2017/05/16 04:49:37 http: TLS handshake error from 192.122.131.41:5278: read tcp 172.18.0.4:4443->192.122.131.41:5278: read: connection reset by peer {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"36bfffb3-4152-4678-a73b-5ebf9dbf85a6","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47252","http.request.uri":"/v2/","http.request.useragent":"Go-http-client/1.1","http.response.duration":"49.167µs","http.response.status":200,"http.response.written":2,"level":"info","msg":"response completed","time":"2017-05-16T04:51:05Z"} {"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:51:09Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"a1a65348-dad4-46ce-845a-86cade5c603b","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47254","http.request.uri":"/v2/","http.request.useragent":"Go-http-client/1.1","http.response.duration":"41.645µs","http.response.status":200,"http.response.written":2,"level":"info","msg":"response completed","time":"2017-05-16T04:51:09Z"} {"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test":"gun","go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"e17a7ca3-2424-4a8b-a944-346c577b11d4","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47256","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","level":"error","msg":"500 GET timestamp key: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:51:18Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"e17a7ca3-2424-4a8b-a944-346c577b11d4","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47256","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","level":"error","msg":"unknown: unknown error: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T04:51:18Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"e17a7ca3-2424-4a8b-a944-346c577b11d4","http.request.method":"GET","http.request.remoteaddr":"54.169.113.36:47256","http.request.uri":"/v2/ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/test/_trust/tuf/timestamp.key","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"642.125µs","http.response.status":500,"http.response.written":70,"level":"info","msg":"response completed","time":"2017-05-16T04:51:18Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"f3700b39-25b2-463f-9967-38a1cdcaf540","http.request.method":"GET","http.request.remoteaddr":"4.79.123.0:32552","http.request.uri":"/v2/:","http.request.useragent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)","level":"info","msg":"metadata not found: You have requested metadata that does not exist.: \u003cnil\u003e","time":"2017-05-16T04:52:25Z"} {"go.version":"go1.8.1","http.request.host":"ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443","http.request.id":"f3700b39-25b2-463f-9967-38a1cdcaf540","http.request.method":"GET","http.request.remoteaddr":"4.79.123.0:32552","http.request.uri":"/v2/:","http.request.useragent":"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"125.775µs","http.response.status":404,"http.response.written":104,"level":"info","msg":"response completed","time":"2017-05-16T04:52:25Z"}

Everything in between was omitted as the whole log is about 125 pages long and it is a repeat of this:

{"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner\"","time":"2017-05-16T08:12:09Z"}

And here is the log for notary_signer_1:

root@ip-172-31-23-92:/home/ubuntu# docker logs notary_signer_1 error: dial tcp 172.18.0.2:3306: getsockopt: connection refused waiting for mysql://signer@tcp(mysql:3306)/notarysigner to come up. no change notarysigner database migrated to latest version {"level":"info","msg":"Version: 0.5.0, Git commit: 9ecd6ebe","time":"2017-05-16T02:30:59Z"} {"level":"debug","msg":"Trusting 1 certs","time":"2017-05-16T02:30:59Z"} {"level":"debug","msg":"Default Alias: timestamp_1","time":"2017-05-16T02:30:59Z"} 2017/05/16 02:31:00 grpc: Server.Serve failed to complete security handshake from "172.19.0.3:58756": remote error: tls: bad certificate

brandongohwh commented 7 years ago

I am not sure whether this will help but I noticed in another thread (https://github.com/moby/moby/issues/21741) that multiple network interfaces will interfere with docker ip bindings (Amazon EC2):

root@ip-172-31-23-92:/home/ubuntu/notary/fixtures# ifconfig br-01d91b24fa49 Link encap:Ethernet HWaddr 02:42:b3:f0:81:d7 inet addr:172.18.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:b3ff:fef0:81d7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:648 errors:0 dropped:0 overruns:0 frame:0 TX packets:629 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:55814 (55.8 KB) TX bytes:61893 (61.8 KB)

br-f61327c607cc Link encap:Ethernet HWaddr 02:42:31:fc:fc:1d inet addr:172.19.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:31ff:fefc:fc1d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1072 (1.0 KB) TX bytes:648 (648.0 B)

docker0 Link encap:Ethernet HWaddr 02:42:fb:e0:f9:5e inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:fbff:fee0:f95e/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:3731 errors:0 dropped:0 overruns:0 frame:0 TX packets:15214 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:203637 (203.6 KB) TX bytes:48351867 (48.3 MB)

eth0 Link encap:Ethernet HWaddr 06:61:dc:15:87:9f inet addr:172.31.23.92 Bcast:172.31.31.255 Mask:255.255.240.0 inet6 addr: fe80::461:dcff:fe15:879f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:297268 errors:0 dropped:0 overruns:0 frame:0 TX packets:82939 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:403584091 (403.5 MB) TX bytes:9865085 (9.8 MB)

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:160 errors:0 dropped:0 overruns:0 frame:0 TX packets:160 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:11840 (11.8 KB) TX bytes:11840 (11.8 KB)

veth082dc31 Link encap:Ethernet HWaddr ae:c2:c8:9e:18:47 inet6 addr: fe80::acc2:c8ff:fe9e:1847/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5829 errors:0 dropped:0 overruns:0 frame:0 TX packets:7133 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:623433 (623.4 KB) TX bytes:840979 (840.9 KB)

veth2f81e8f Link encap:Ethernet HWaddr ce:43:e7:7c:b1:f9 inet6 addr: fe80::cc43:e7ff:fe7c:b1f9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:22 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:648 (648.0 B) TX bytes:1776 (1.7 KB)

veth524b959 Link encap:Ethernet HWaddr 26:e3:38:2a:2e:62 inet6 addr: fe80::24e3:38ff:fe2a:2e62/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:21 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:648 (648.0 B) TX bytes:1686 (1.6 KB)

veth77ed7ed Link encap:Ethernet HWaddr b6:c0:5b:3f:96:ec inet6 addr: fe80::b4c0:5bff:fe3f:96ec/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1109 errors:0 dropped:0 overruns:0 frame:0 TX packets:1343 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:105182 (105.1 KB) TX bytes:142863 (142.8 KB)

vethecbf30b Link encap:Ethernet HWaddr 72:e0:db:53:fb:06 inet6 addr: fe80::70e0:dbff:fe53:fb06/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7806 errors:0 dropped:0 overruns:0 frame:0 TX packets:6332 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:918763 (918.7 KB) TX bytes:667005 (667.0 KB)

riyazdf commented 7 years ago

@brandongohwh thanks for the report!

for this error:

{"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = "transport: x509: certificate is valid for ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, not notarysigner"","time":"2017-05-16T08:12:09Z"}

What's the CN/SAN in your notary server/signer certificates? It looks like you might have a mismatch between what the server reports as its hostname and the certificate info itself. This seems to be causing the notary server and signer to not properly connect with each other, as @cyli points out with the 500s.

brandongohwh commented 7 years ago

I regenerated my certificates to include the respective SAN but it is still unable to connect (Same error 500). Here is the self-signed certificate (rootCA) output:

root@ip-172-31-23-92:/home/ubuntu# openssl x509 -in certgen/rootCA.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12112607019469252621 (0xa8189fe8ab5e280d)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=<value>, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 01:30:22 2017 GMT
            Not After : Jun 16 01:30:22 2017 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:d4:62:49:ce:70:75:6e:08:a5:5a:98:6a:3c:26:
                    5d:f9:e3:af:1c:31:2d:9f:0f:f5:32:3c:7a:e8:ce:
                    88:68:2c:e5:32:df:f1:29:17:2d:fc:c1:c4:d0:e4:
                    d3:be:0c:c0:c6:83:1f:42:cf:7e:e2:5a:cb:27:de:
                    8b:af:8e:63:fe:05:4a:e1:7a:4f:19:12:9b:87:07:
                    1f:7a:a8:2c:a6:9d:f9:8d:17:c8:c0:a9:ea:f3:d3:
                    c4:c6:9e:f2:62:09:53:5f:0e:33:56:74:c0:d4:3d:
                    98:3d:cd:58:1e:58:50:19:46:2d:d1:fb:18:51:d3:
                    5d:1e:70:b2:66:66:62:67:d5:91:5e:b5:30:8e:bb:
                    6e:4b:05:3c:01:3c:ae:6e:10:e1:e5:8d:22:d0:3b:
                    d2:e3:e8:db:17:c3:a8:fc:fa:48:04:5d:b6:ef:60:
                    31:2b:be:c3:4e:d4:3c:f8:e3:bb:49:5f:73:f6:ba:
                    4d:1c:1f:1a:c6:06:4d:5b:b7:0d:1f:48:ad:99:94:
                    4b:f8:d3:ce:54:3a:e3:72:30:7b:26:e6:ff:f5:01:
                    65:64:c6:66:50:4e:3e:e7:cb:8f:29:cd:02:45:ba:
                    a9:12:1a:3e:f7:2e:f8:b3:cd:f2:e3:1c:7e:a8:0e:
                    a0:ad:2e:69:57:ba:f6:ee:2e:d7:25:11:f0:94:41:
                    b2:9b:7c:cd:e5:a9:5c:a4:05:50:54:cb:06:4c:31:
                    93:6b:24:41:4b:c4:b5:57:3b:d8:8a:18:4b:98:cb:
                    34:eb:34:15:46:10:c2:56:4f:45:19:7a:bc:67:dd:
                    12:7c:35:9a:8c:fb:d1:3e:a3:7c:58:b7:40:18:cd:
                    41:cc:d5:c3:1d:6c:91:f5:53:cb:cd:77:8b:72:f6:
                    56:58:8d:12:fa:b5:d4:be:62:6a:c1:cb:0c:08:b8:
                    ce:e8:0b:c5:a1:a0:ca:df:da:c6:84:8c:cd:ce:dd:
                    19:c9:d6:bd:6e:a4:7a:8c:37:e2:21:4b:82:4d:e0:
                    30:75:84:fe:01:77:ea:05:da:45:47:45:01:10:30:
                    38:c6:15:4f:ab:c9:0f:a2:5a:38:b3:25:b6:7b:5a:
                    e1:fb:82:86:37:94:74:fc:51:35:7b:cd:f5:eb:e9:
                    0f:66:f4:65:58:6f:a1:ab:17:48:cc:21:7d:05:96:
                    24:03:b6:c4:2f:4a:a3:5f:83:9b:34:1e:8b:0e:d0:
                    27:1b:ef:68:ae:ea:7e:bd:92:3b:80:9c:3c:c2:4d:
                    32:58:3b:35:8a:42:5d:bf:77:3f:78:28:13:40:67:
                    1f:7f:4f:35:9a:59:33:64:30:7f:83:61:53:e1:be:
                    7e:25:a0:08:ff:d3:df:09:d6:48:3d:f2:73:8b:ad:
                    11:03:ab
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, DNS:notaryserver, DNS:notary-server, DNS:notarysigner
            X509v3 Subject Key Identifier: 
                05:E1:F6:7F:6E:7B:E0:2E:36:D8:BC:25:EA:57:95:DC:39:0F:A0:53
            X509v3 Authority Key Identifier: 
                keyid:05:E1:F6:7F:6E:7B:E0:2E:36:D8:BC:25:EA:57:95:DC:39:0F:A0:53

            X509v3 Basic Constraints: 
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         af:9b:a2:de:e3:e0:08:5c:8c:b4:cc:da:f7:3a:71:59:4e:94:
         e1:3f:ce:96:52:67:fd:e9:09:2d:70:7d:d6:c8:91:46:82:ef:
         13:75:5d:90:c2:b3:e4:30:91:65:1f:b5:ad:ee:6a:2c:ef:bd:
         91:06:d2:14:17:74:34:61:0d:de:87:0c:16:87:1d:2d:e0:95:
         a5:3f:f2:31:fb:37:5c:56:a0:ae:bd:95:c6:d8:3d:da:18:31:
         fb:b1:30:40:33:42:f0:19:ea:e0:b1:67:53:fc:e6:ae:8f:2a:
         9e:a2:64:4f:82:2a:92:d2:ab:91:f9:36:44:23:70:f6:3f:d5:
         a5:4e:41:ed:90:f5:71:a4:24:d0:f1:bd:6c:87:79:da:a7:5c:
         3f:ac:91:d6:c5:93:d0:e4:1d:76:96:96:99:2e:44:e4:94:22:
         ae:7f:a3:a9:ee:8f:5d:a5:cf:f8:e4:61:3a:57:f0:6a:fa:80:
         e3:97:68:e0:dc:d8:46:4c:79:ce:d9:f0:59:aa:69:0b:3f:cb:
         ea:0f:66:de:c2:f3:2d:0a:3a:be:d6:46:e1:f6:06:eb:3f:ec:
         f4:11:e3:d3:b7:4f:80:b2:60:01:e9:c8:72:e8:36:52:6f:d2:
         d3:0f:27:a3:5f:7c:48:14:3b:22:cf:ca:3e:92:a8:84:42:62:
         2c:1c:33:b6:05:98:89:25:7f:44:b6:e6:9a:26:b7:f2:b5:cf:
         0b:de:bd:cb:b4:92:80:73:b4:a1:ad:d4:ae:9b:6e:ae:a8:a8:
         44:cb:55:c4:dc:99:a7:fe:47:24:27:75:a6:9d:a1:5d:1d:66:
         41:4a:3b:51:75:89:0d:b0:ed:07:66:c8:1f:0d:86:96:09:06:
         fc:d4:73:bf:fd:b2:a4:f4:ae:45:aa:64:27:c1:c5:df:da:a3:
         1e:61:ba:bd:7a:b4:bf:c5:da:50:b2:ec:14:e4:9a:68:86:6d:
         51:e7:63:61:fe:da:04:00:73:64:d5:f7:92:3d:5b:dd:4c:de:
         6f:a7:77:24:67:cb:d6:9f:6b:17:0b:64:7d:6f:dd:00:2c:da:
         9c:f1:49:6e:59:ea:dd:ca:1d:03:7d:38:6b:d2:73:3b:33:cf:
         e1:43:f7:12:23:14:44:6c:a8:3a:04:ce:af:ae:06:ef:b8:aa:
         cf:e7:88:cf:c0:86:cf:b3:cc:57:99:3f:0c:4d:c9:41:e3:6c:
         07:dd:44:7a:2c:ff:a5:b0:93:03:c3:46:31:51:87:98:95:7d:
         68:4f:95:b2:fb:9a:4f:57:fc:25:14:cc:26:d3:00:66:84:ed:
         e3:d5:53:0f:d1:a9:aa:da:11:c3:e7:5a:04:72:8a:2a:bd:6a:
         9a:ba:d1:2b:02:c3:c6:2a
cyli commented 7 years ago

@brandongohwh That is the CA certificate. What is the leaf certificate being used for the notary signer and notary server? Do they both have the same SANs?

brandongohwh commented 7 years ago

I fixed the SANs on notary-client and notary-server certificates using the guide from this link: http://stackoverflow.com/questions/21297139/how-do-you-sign-certificate-signing-request-with-your-certification-authority

So currently all my certificates and keys are in pem format but it points to the correct files (respective certificates and keys). Error 500 still persists but now with this error

{"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority\"","time":"2017-05-17T07:18:39Z"}

I copied the root.pem file into ca-certificates and renamed it into .crt format so it is currently in the trusted certificates list.

I changed the configuration file for Notary CLI and Notary Server to remove the root_ca field, which solves the bad certificate problem.

Notary CLI configuration:

{
        "remote_server": {
                "url": "https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443"
        }
}

Notary server configuration:

{
        "server": {
                "http_addr": ":4443",
                "tls_key_file": "./notary-serverkey.pem",
                "tls_cert_file": "./notary-server.pem"
        },
        "trust_service": {
                "type": "remote",
                "hostname": "notarysigner",
                "port": "7899",
                "key_algorithm": "rsa",
                "tls_client_cert": "./notary-server.pem",
                "tls_client_key": "./notary-serverkey.pem"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "server@tcp(mysql:3306)/notaryserver?parseTime=True"
        }
}

Notary signer configuration: 

{
        "server": {
                "grpc_addr": ":7899",
                "tls_cert_file": "./notary-signer.pem",
                "tls_key_file": "./notary-signerkey.pem",
                "client_ca_file": "./notary-server.pem"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "signer@tcp(mysql:3306)/notarysigner?parseTime=True"
        }
}

The updated logs for notary_server_1 are as follows:

root@ip-172-31-23-92:/home/ubuntu/notary# docker logs notary_server_1
no change
notaryserver database migrated to latest version
{"level":"info","msg":"Version: 0.5.0, Git commit: 9ecd6ebe","time":"2017-05-17T07:40:53Z"}
{"level":"info","msg":"Using remote signing service","time":"2017-05-17T07:40:53Z"}
{"level":"info","msg":"Using mysql backend","time":"2017-05-17T07:40:53Z"}
2017/05/17 07:40:53 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 172.19.0.2:7899: getsockopt: connection refused"; Reconnecting to {notarysigner:7899 <nil>}
{"level":"info","msg":"Starting Server","time":"2017-05-17T07:40:53Z"}
{"level":"info","msg":"Enabling TLS","time":"2017-05-17T07:40:53Z"}
{"level":"info","msg":"Starting on :4443","time":"2017-05-17T07:40:53Z"}
2017/05/17 07:40:54 Failed to dial notarysigner:7899: connection error: desc = "transport: x509: certificate signed by unknown authority"; please retry.

Logs for notary_signer_1:

root@ip-172-31-23-92:/home/ubuntu/notary# docker logs notary_signer_1
error: dial tcp 172.18.0.2:3306: getsockopt: connection refused
waiting for mysql://signer@tcp(mysql:3306)/notarysigner to come up.
no change
notarysigner database migrated to latest version
{"level":"info","msg":"Version: 0.5.0, Git commit: 9ecd6ebe","time":"2017-05-17T07:40:54Z"}
{"level":"debug","msg":"Trusting 1 certs","time":"2017-05-17T07:40:54Z"}
{"level":"debug","msg":"Default Alias: timestamp_1","time":"2017-05-17T07:40:54Z"}
2017/05/17 07:40:54 grpc: Server.Serve failed to complete security handshake from "172.19.0.3:60000": remote error: tls: bad certificate

Here is the re-generated notary-server.pem certificate:

root@ip-172-31-23-92:/home/ubuntu/notary# openssl x509 -in ../certgen/notary-server.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3 (0x3)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 06:36:04 2017 GMT
            Not After : May 17 06:36:04 2018 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ae:f7:06:e4:91:37:ea:0c:00:e6:18:01:33:53:
                    ad:7a:67:dd:4c:25:2d:54:dd:39:04:11:ca:63:b0:
                    96:c8:39:c1:2f:24:6f:f9:5a:4e:b2:39:09:b4:6f:
                    59:be:20:c6:41:00:14:54:82:04:81:1b:1f:3b:c5:
                    6b:5f:86:88:b7:6c:b8:e3:a7:d0:02:66:75:bf:f8:
                    22:cc:58:4b:75:b0:46:ed:27:5c:70:13:57:aa:f9:
                    09:20:a2:b7:23:96:ac:78:f3:f9:b5:13:18:2c:ff:
                    9b:32:1f:b6:96:77:7e:b6:ca:bd:9a:05:4a:97:8b:
                    08:51:d8:a1:44:e0:77:b6:c7:85:3e:bd:70:df:df:
                    cd:c7:cd:8f:91:8e:12:95:5b:51:4e:b1:2c:ca:ba:
                    3a:17:8f:89:24:2c:20:a5:21:6a:7e:37:77:37:5d:
                    f2:97:56:72:f8:f5:18:28:be:f2:8c:95:b6:74:70:
                    75:1a:dc:ed:82:35:11:a8:21:82:f4:6e:ee:64:c1:
                    50:41:d1:92:7e:48:96:fa:c3:69:70:4a:ed:55:1d:
                    2b:38:e3:fa:e9:70:54:2b:55:26:39:8e:13:95:e6:
                    d9:0f:a5:28:c3:1b:5b:51:55:09:8d:56:3e:83:bc:
                    09:40:6b:9b:2a:eb:58:8b:19:60:34:8b:c4:6b:33:
                    9a:95
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                7D:25:52:4C:55:BA:59:8C:53:6F:80:3A:74:BD:34:6F:4E:9B:7A:E6
            X509v3 Authority Key Identifier: 
                keyid:02:93:16:36:23:B5:F8:41:EB:6C:E2:31:20:76:0D:E8:93:38:CB:17

            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, DNS:notary-server, DNS:notaryserver, DNS:notary-signer, DNS:notarysigner
    Signature Algorithm: sha256WithRSAEncryption
         1a:69:c0:f8:2c:fd:25:bf:46:e5:7b:b7:1b:a4:ad:6f:69:09:
         86:ab:73:6d:51:de:6f:52:4b:b6:f5:97:06:88:31:06:d8:6f:
         c8:8a:72:a6:22:94:84:4f:78:aa:5d:fd:ae:87:70:a4:a2:58:
         38:d1:38:7b:33:11:b4:35:13:1e:65:10:cf:d4:d5:af:9f:96:
         a1:33:7d:69:31:04:d2:f1:b6:a5:17:66:95:64:3f:c7:70:7a:
         9f:5c:5b:26:72:ee:b2:47:13:55:ac:36:f1:9b:78:36:58:2b:
         7d:67:a6:77:1b:34:a5:4c:b8:24:c7:8c:9b:3a:df:d4:94:0c:
         ae:d5:a6:1b:fe:fe:2e:63:f0:60:5c:84:62:85:98:69:7d:eb:
         f7:20:75:ee:15:78:35:f0:58:d7:b3:1f:e7:6e:c2:68:a9:e0:
         5d:1a:82:b5:91:b2:3d:4b:56:de:0e:c9:79:dd:12:fb:c1:19:
         cf:1e:28:c2:93:30:b5:a9:42:7a:15:de:8d:8a:8b:d3:23:63:
         82:c3:01:91:84:16:17:45:2c:07:22:b1:94:08:f4:79:8c:f9:
         db:f0:f5:cc:ad:cf:3c:9a:62:ab:b6:11:37:fe:9b:93:8b:09:
         5e:14:24:53:79:94:98:75:f2:e6:38:c8:41:56:1f:5a:35:22:
         7e:8e:4b:9e:29:a3:2c:50:38:6b:83:c4:ec:ba:b7:5a:c0:76:
         c4:bd:59:b5:d7:23:15:84:5d:41:e0:a3:d7:f9:e5:2d:d0:47:
         40:c7:bd:22:cc:9e:63:61:e0:ea:b9:c3:e6:2e:c1:fa:c9:81:
         34:8f:ab:2b:07:e9:99:ed:4d:9a:00:50:01:6a:0d:fd:c3:df:
         46:18:dd:f4:83:04:a6:48:c3:25:34:09:ed:34:6c:b2:b2:92:
         b4:43:79:5d:78:01:4e:32:19:ae:0e:70:eb:61:7d:ca:07:65:
         6e:64:47:fa:95:6a:b8:96:be:d2:c2:f3:31:08:f3:af:d6:4d:
         d4:10:ec:60:96:c8:4c:8b:7c:b2:35:a4:75:df:e3:8f:e6:39:
         22:92:40:55:f9:b8:61:2f:d8:e7:bd:0a:97:da:d4:6e:a8:5f:
         86:c0:4d:79:7a:9e:5e:2a:ef:1b:8e:e6:79:63:31:b4:ae:35:
         09:b9:32:b8:c7:39:92:7a:7b:53:a0:04:14:08:1d:21:3d:59:
         8c:c9:3d:76:6c:3a:d7:56:a9:00:db:60:ad:01:a9:a6:dc:d5:
         ec:fb:40:8b:65:7e:e6:b0:2d:7d:8a:92:f6:61:76:45:0d:55:
         ed:e8:13:68:16:34:68:e6:5d:6d:20:ba:73:4e:54:33:99:13:
         d4:7d:9b:87:36:f2:45:07

and notary-signer.pem certificate:

root@ip-172-31-23-92:/home/ubuntu/notary# openssl x509 -in ../certgen/notary-signer.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6 (0x6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 06:39:52 2017 GMT
            Not After : May 17 06:39:52 2018 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, CN=ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e7:f9:89:f2:70:d4:ec:29:46:7c:90:b7:85:a7:
                    eb:22:75:03:72:79:2b:5e:0a:98:0a:d3:b6:64:77:
                    65:c0:f9:c4:44:17:f6:ea:b6:68:d8:00:0c:9d:56:
                    08:98:b7:6a:af:95:5f:36:c4:b1:57:5d:d3:7a:6b:
                    56:96:94:ca:c4:4b:4f:61:0b:2f:c1:f0:b0:ec:b7:
                    a9:33:52:39:a2:03:41:48:61:26:e9:ae:1d:ff:20:
                    aa:38:84:97:99:fd:db:94:29:82:6a:43:d3:e2:b0:
                    d0:ed:f1:a5:15:95:73:8b:9b:dc:e5:18:cf:93:31:
                    4d:e6:78:ac:63:98:b0:81:fc:82:ab:fe:61:6f:92:
                    51:d0:67:1f:42:03:86:04:64:cf:8a:dd:1e:05:9d:
                    fd:f6:07:cc:b6:d3:a3:f5:f5:fb:e2:52:47:47:6f:
                    44:31:03:0c:23:bd:d1:7c:f0:9b:b8:3c:01:01:66:
                    c3:fa:ef:33:65:a5:ab:f2:8a:77:9a:79:7a:03:5f:
                    43:e1:92:5b:b3:88:07:1c:7c:7c:7c:36:56:2c:2a:
                    56:75:d8:86:c1:fd:2c:74:08:3c:74:ef:eb:d1:ca:
                    07:75:71:8c:2b:52:9c:08:dd:04:a7:75:9f:4a:5e:
                    b1:d4:05:06:39:40:69:fe:85:f3:ac:c0:15:50:00:
                    f3:f5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                4A:DE:88:8C:C3:FB:38:E3:6F:EA:2E:02:B5:C5:98:BA:13:24:A2:F9
            X509v3 Authority Key Identifier: 
                keyid:02:93:16:36:23:B5:F8:41:EB:6C:E2:31:20:76:0D:E8:93:38:CB:17

            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com, DNS:notary-server, DNS:notaryserver, DNS:notary-signer, DNS:notarysigner
    Signature Algorithm: sha256WithRSAEncryption
         87:2f:8d:fe:6f:59:77:20:0f:f6:c8:56:4d:6d:3b:d4:d1:52:
         04:7c:dc:8a:0e:f2:ac:f1:f2:4e:6e:cc:ed:76:a6:e0:d3:c3:
         b8:d8:8f:66:f3:be:7a:d3:6c:6d:0d:7e:fb:1b:b6:b8:00:e8:
         c8:f3:c3:ae:80:04:a1:0e:14:60:a8:a3:61:70:db:ba:0d:34:
         ef:9d:00:6e:5e:e3:bb:f7:4e:02:5d:a8:87:20:96:ec:21:b4:
         62:f9:ab:17:c9:ec:59:06:dc:29:79:37:8f:a8:4a:98:07:19:
         44:42:b5:72:89:3e:e5:3d:bb:0c:f5:75:fc:8d:1b:14:4c:71:
         bd:90:e0:10:7f:27:e0:86:ce:ed:38:b4:a1:db:05:f8:d0:5a:
         34:9b:60:a5:af:89:13:53:ab:37:fd:ed:f2:0a:4e:17:a5:ff:
         0d:d3:fb:61:75:6e:90:46:c5:70:7b:47:77:8b:5d:22:d3:90:
         db:c3:8d:e5:8e:54:24:32:0a:c6:3c:32:f6:bd:b7:72:02:08:
         d8:8c:eb:b6:e0:40:09:3e:44:15:74:54:2d:9e:38:c0:71:38:
         1e:46:07:2c:72:99:20:96:1e:83:1b:d1:a1:43:52:9e:2f:cc:
         a7:e6:e6:5b:01:03:dc:5a:1b:39:f0:33:3b:4e:4a:a4:6d:98:
         10:2d:35:c8:c6:a0:03:5e:26:aa:35:84:e4:62:ee:03:58:20:
         d3:ea:62:7e:7c:02:af:dc:0c:a3:bb:2b:e9:0a:19:79:88:43:
         ea:f2:07:a6:27:10:9e:a7:d7:35:20:a5:fa:05:f9:89:29:27:
         60:77:a6:02:81:1a:fc:70:75:1d:3d:a0:a5:62:90:9a:96:c4:
         1c:7d:ea:72:95:d3:d7:34:6d:43:19:ce:52:bb:56:bd:37:0a:
         87:32:0e:0c:a2:b3:0a:ac:f2:36:a9:d9:8a:99:a1:e4:a0:05:
         1f:89:e6:f7:be:91:4c:2a:7d:27:92:3e:a1:e0:62:e0:39:e2:
         86:ac:88:8f:58:a0:3a:69:89:58:df:2a:ac:6c:6b:bb:b4:d9:
         61:c3:40:54:7f:11:a9:ea:86:81:60:74:ae:e1:df:c5:21:28:
         34:14:8b:63:7b:d3:24:8e:16:28:ae:f0:eb:36:83:0f:54:24:
         c0:40:1e:fc:4f:bf:3e:f0:80:48:c5:04:58:07:94:ec:c6:1f:
         e2:84:e0:0b:e0:5e:e5:9c:12:33:63:6a:29:da:59:a9:9a:ce:
         e0:70:d4:7a:52:4b:fa:7f:2d:84:c7:75:d0:14:7b:f9:bc:33:
         92:ba:cf:11:35:82:87:f4:10:ed:0e:76:6e:9d:87:22:1f:25:
         ba:cc:d2:64:22:c1:de:41

I have also used export DOCKER_CONTENT_TRUST=1 export DOCKER_CONTENT_TRUST_SERVER=https://ec2-54-169-113-36.ap-southeast-1.compute.amazonaws.com:4443

I hope I do not have to generate new certificates :)

brandongohwh commented 7 years ago

Ok, I finally solved the problem. I reinstated all the lines that I deleted (as indicated from the previous post) and edited a number of things.

So I realised i edited some files wrongly (or blindly) without fully understanding the document (since I was in a rush to get it up for testing), so I thought it would be nice to put up my configuration files here as a reference (for others as well):

Notary Server configuration:

{
        "server": {
                "http_addr": ":4443",
                "tls_key_file": "./notary-serverkey.pem",
                "tls_cert_file": "./notary-server.pem"
        },
        "trust_service": {
                "type": "local",
                "hostname": "ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com",
                "port": "7899",
                "tls_ca_file": "./root-ca.pem",
                "key_algorithm": "rsa",
                "tls_client_cert": "./notary-server.pem",
                "tls_client_key": "./notary-serverkey.pem"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "server@tcp(mysql:3306)/notaryserver?parseTime=True"
        }
}

Notary Signer Configuration:

{
        "server": {
                "grpc_addr": ":7899",
                "tls_cert_file": "./notary-signer.pem",
                "tls_key_file": "./notary-signerkey.pem",
                "client_ca_file": "./notary-server.pem"
        },
        "logging": {
                "level": "debug"
        },
        "storage": {
                "backend": "mysql",
                "db_url": "signer@tcp(mysql:3306)/notarysigner?parseTime=True"
        }
}

Notary Client Configuration:

{
        "remote_server": {
                "url": "https://ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com:4443",
                "root_ca": "root-ca.pem"
        }
}

To summarise the changes (from the original files when installing using git clone):

Notary Server Configuration:

  1. Changed trust_service.type to local
  2. Changed trust_service.hostname to <Public DNS hostname>
  3. Changed trust_service.Key_algorithm from ecdsa to rsa (as my generated cert was RSA with encryption)
  4. Changed all tls certificate and key files to point to the respective certificate and keys. *(1 & 2 solved my connection reset issue)

Notary Signer Configuration:

  1. Changed all tls certificate and key files to point to the respective certificate and keys.

Notary Client Configuration:

  1. Changed remote_server.url to <Public DNS hostname>
  2. Changed remote_server.root_ca to point to the respective root certificate file.

Other configurations done:

  1. cp root-ca.pem /usr/local/share/ca-certificates/root-ca.crt && update-ca-certificates (I realised that the x509 unknown certificate authority issue kept persisting even though I did step 1 multiple times, this was due to me constantly overriding the root certificates with the same name, so the newer root certificates did not update to /etc/ssl/certs and I had to manually remove any previously generated root certificate before running the update command again.)

  2. export DOCKER_CONTENT_TRUST=1

  3. export DOCKER_CONTENT_TRUST_SERVER=https://<Public DNS Hostname>:4443

*Generating all my different .pem files was done according to the instructions on http://stackoverflow.com/questions/21297139/how-do-you-sign-certificate-signing-request-with-your-certification-authority

For reference, this is my root-ca.pem:

openssl x509 -in root-ca.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 18249259504409239986 (0xfd4264d49e3be1b2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 13:08:40 2017 GMT
            Not After : Jun 16 13:08:40 2017 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:b6:65:c4:41:32:34:85:eb:24:c3:e5:94:8d:70:
                    ...
                    63:4c:33
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                CE:41:5D:16:50:B9:D4:39:E3:31:9E:4D:0F:EC:9E:A4:B6:36:79:15
            X509v3 Authority Key Identifier:
                keyid:CE:41:5D:16:50:B9:D4:39:E3:31:9E:4D:0F:EC:9E:A4:B6:36:79:15

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         b2:d0:1f:20:7f:99:dc:59:46:05:ce:ea:eb:3a:a8:16:fb:d0:
         ...
         62:e1:54:7e:5d:9a:88:ed

notary-signer.pem:

openssl x509 -in notary-signer.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 13:21:24 2017 GMT
            Not After : Feb 11 13:21:24 2020 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:be:c6:8b:9e:8f:c3:17:1a:75:f3:c6:2b:5a:d0:
                    ...
                    50:9e:e3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                4A:39:45:4A:80:29:81:F2:77:E1:92:CA:C9:07:B3:F7:88:49:09:A6
            X509v3 Authority Key Identifier:
                keyid:CE:41:5D:16:50:B9:D4:39:E3:31:9E:4D:0F:EC:9E:A4:B6:36:79:15

            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com, DNS:ip-172-31-31-110.ap-southeast-1.compute.internal, DNS:localhost, DNS:notary-server, DNS:notaryserver, DNS:notary-signer, DNS:notarysigner, IP Address:54.179.148.24, IP Address:172.31.31.110
    Signature Algorithm: sha256WithRSAEncryption
         af:37:10:ee:24:46:20:51:e7:71:45:87:32:29:68:94:91:e0:
         ...
         89:47:35:a2:d0:04:4e:01

notary-server.pem:

openssl x509 -in notary-server.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com/emailAddress=<value>
        Validity
            Not Before: May 17 13:21:07 2017 GMT
            Not After : Feb 11 13:21:07 2020 GMT
        Subject: C=SG, ST=Singapore, L=Singapore, O=<value>, OU=<value>, CN=ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:e2:74:1d:b2:3b:b5:4d:9b:47:cc:5e:4e:57:bb:
                    ...
                    85:e7:a5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                65:0C:0E:55:E8:3F:27:E1:D5:23:D9:01:99:41:38:ED:5A:0C:FB:24
            X509v3 Authority Key Identifier:
                keyid:CE:41:5D:16:50:B9:D4:39:E3:31:9E:4D:0F:EC:9E:A4:B6:36:79:15

            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:ec2-54-179-148-24.ap-southeast-1.compute.amazonaws.com, DNS:ip-172-31-31-110.ap-southeast-1.compute.internal, DNS:localhost, DNS:notary-server, DNS:notaryserver, DNS:notary-signer, DNS:notarysigner, IP Address:54.179.148.24, IP Address:172.31.31.110
    Signature Algorithm: sha256WithRSAEncryption
         55:fb:cc:5e:02:a4:45:fc:89:2d:43:6a:75:cf:eb:97:6e:1b:
         ...
         e8:27:25:f3:e0:fa:d0:85

*Parts marked with <value> are non-empty but can also be left blank (redacted for privacy).

brandongohwh commented 7 years ago

I asked a couple of my colleagues about what they could understand after reading the documentation (or a bit of hands-on) and most of them told me they found it confusing. I myself stumbled when I first started on this project 6 days ago. I'm not sure about using Hugo or Markdown, but should I open another thread on some suggestions to improving the docs?

riyazdf commented 7 years ago

@brandongohwh glad you were able to figure it out, thank you for posting your configuration

And yes please! We'd love any and all feedback to make the docs better. Please feel free to open an issue on this repo - we'd love to hear your suggestions :)

cyli commented 7 years ago

Changed trust_service.type to local

This would fix the connection issues because it's not using the signer at all. This option is mainly to be used for testing - a local trust service just uses an in-memory signer. So once you shut down the server, the keys wouldn't exist.

Changed trust_service.Key_algorithm from ecdsa to rsa (as my generated cert was RSA with encryption)

Key algorithm just specifies which key algorithm to use to generate signing keys for TUF, and should not have to do with the certificate.

brandongohwh commented 7 years ago

@cyli Thanks for the input! I changed the key algorithm back to ecdsa.

I finally isolated the problem by trying to provision another server using trust_service.type = remote and was successful at first, but restarting the Notary server brought back the 500 error, primarily because the MySQL database could not connect.

Looking at the logs and googling around, I found that it was a common Amazon AWS problem since MySQL was resolving to the internal IP of the VM (why connection refused? I really don't know), so it really was a hit or miss issue and the easiest workaround is to set trust_service.type = local. This issue affects Docker images in general. Other than that, there was no issue with the other functions like initialising collections, pushing files etc. (Might want to take note of that) :)

Error log from notary_server_1:

{"level":"info","msg":"Using mysql backend","time":"2017-05-18T06:41:36Z"}
2017/05/18 06:41:36 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 172.31.23.223:7899: getsockopt: connection refused"; Reconnecting to {ec2-54-255-182-133.ap-southeast-1.compute.amazonaws.com:7899 <nil>}
{"level":"fatal","msg":"Error starting mysql driver: dial tcp 172.31.23.223:3306: getsockopt: connection refused","time":"2017-05-18T06:41:36Z"}