jim3ma
commented
2 years ago 配置能不能贴到代码段里？直接写进来，格式乱了，另外就是有 debug 的日志么？
Open yangchuan37326 opened 2 years ago
jim3ma
commented
2 years ago 配置能不能贴到代码段里？直接写进来，格式乱了，另外就是有 debug 的日志么？
jim3ma
commented
2 years ago 从日志上来看， 第一个请求是下面这两行，tls 已经成功握手了，不然走不到 round trip directly
2022-10-10T03:11:10.030Z DEBUG proxy/proxy_sni.go:91 Generate temporal leaf TLS cert for ServerName <harbor.test.cn>
2022-10-10T03:11:10.037Z DEBUG transport/transport.go:180 round trip directly, method: GET, url: https://harbor.test.cn/v2/下面两行是报错了，remote error 报错看上去是对端校验证书的时候不正确导致的，建议看一下宿主机上的证书是否有动过，daemonset 的 pod 会自动注入证书的，你可以贴一下 daemonset 的 spec，看看是否是后置的 postStart 改动了证书导致部分请求异常了。
2022-10-10T03:11:10.037Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn>
2022-10-10T03:11:10.040Z ERROR proxy/proxy_sni.go:105 handshake failed for harbor.test.cn: remote error: tls: bad certificate
d7y.io/dragonfly/v2/client/daemon/proxy.(*Proxy).handleTLSConn
/go/src/d7y.io/dragonfly/v2/client/daemon/proxy/proxy_sni.go:105
2022/10/10 03:11:10 http: proxy error: x509: certificate signed by unknown authority后续请求，tls 成功握手
2022-10-10T03:11:10.042Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn>
2022-10-10T03:11:10.046Z DEBUG transport/transport.go:180 round trip directly, method: HEAD, url: https://harbor.test.cn/v2/rancher/rancher-agent/manifests/v2.5.0
2022-10-10T03:11:10.046Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn>
yangchuan37326
commented
2 years ago 1、我检查了宿主机上(dfdaemon容器内挂在的这两个证书)/etc/dragonfly-ca/cacert.pem与cakey.pem证书，并确保了/etc/docker/certs.d/harbor.test.cn/ca.crt内容与cacert.pem一致，并且/etc/dragonfly-ca/cacert.pem证书追加到了/etc/pki/tls/certs/ca-bundle.crt中。 2、我还删除过/etc/dragonfly-ca/cacert.pem与cakey.pem这两个证书，由initcontainer重新生成证书，且也确认如上几个文件的证书一致，仍然报证书问题下载不了镜像。 3、dfdaemon-pod yaml文件如下
apiVersion: v1
kind: Pod
metadata:
name: dragonfly-dfdaemon-srv2s
generateName: dragonfly-dfdaemon-
namespace: dragonfly-system
selfLink: /api/v1/namespaces/dragonfly-system/pods/dragonfly-dfdaemon-srv2s
labels:
app: dragonfly
component: dfdaemon
controller-revision-hash: c77475746
pod-template-generation: '1'
release: dragonfly
ownerReferences:
- apiVersion: apps/v1
kind: DaemonSet
name: dragonfly-dfdaemon
uid: 803ac632-0726-495a-a543-7b1cdae2f077
controller: true
blockOwnerDeletion: true
spec:
volumes:
- name: config
configMap:
name: dragonfly-dfdaemon
defaultMode: 420
- name: etc
hostPath:
path: /etc
type: ''
- name: d7y-ca
hostPath:
path: /etc/dragonfly-ca
type: DirectoryOrCreate
- name: data
emptyDir: {}
- name: logs
emptyDir: {}
- name: default-token-549c7
secret:
secretName: default-token-549c7
defaultMode: 420
initContainers:
- name: update-docker-config
image: docker/dragonflyoss/openssl
command:
- /bin/sh
- '-cx'
- >-
mkdir -p /tmp/dragonfly-ca
cd /tmp/dragonfly-ca
openssl genrsa -out cakey.pem 2048
cat << EOF > root.conf
[ req ]
default_bits = 2048
default_keyfile = key.pem
default_md = sha256
distinguished_name = req_distinguished_name
req_extensions = req_ext
string_mask = nombstr
x509_extensions = x509_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Beijing
localityName = Locality Name (eg, city)
localityName_default = Beijing
organizationName = Organization Name (eg, company)
organizationName_default = Dragonfly
commonName = Common Name (e.g. server FQDN or YOUR
name)
commonName_max = 64
commonName_default = Dragonfly Authority CA
[ x509_ext ]
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
keyUsage = digitalSignature, keyEncipherment,
keyCertSign, cRLSign
subjectKeyIdentifier = hash
[ req_ext ]
basicConstraints = CA:TRUE
keyUsage = digitalSignature, keyEncipherment, keyCertSign,
cRLSign
subjectKeyIdentifier = hash
EOF
openssl req -batch -new -x509 -key ./cakey.pem -out ./cacert.pem -days
65536 -config ./root.conf
openssl x509 -inform PEM -in ./cacert.pem -outform DER -out ./CA.cer
openssl x509 -in ./cacert.pem -noout -text
# update ca for golang program(docker in host), refer:
https://github.com/golang/go/blob/go1.17/src/crypto/x509/root_linux.go#L8
ca_list="/etc/ssl/certs/ca-certificates.crt
/etc/pki/tls/certs/ca-bundle.crt /etc/ssl/ca-bundle.pem
/etc/pki/tls/cacert.pem
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/ssl/cert.pem"
for ca in $ca_list; do
ca="/host$ca"
if [[ -e "$ca" ]]; then
echo "CA $ca" found
if grep "Dragonfly Authority CA" "$ca"; then
echo "Dragonfly Authority ca found"
if [[ -e /host/etc/dragonfly-ca/cakey.pem && -e /host/etc/dragonfly-ca/cacert.pem ]]; then
echo "CA cert and key ready"
break
else
echo "Warning: CA cert and key not ready"
fi
fi
echo "Try to add Dragonfly CA"
echo "# Dragonfly Authority CA" > cacert.toadd.pem
cat cacert.pem >> cacert.toadd.pem
cat cacert.toadd.pem >> "$ca"
echo "Dragonfly CA added"
cp -f ./cakey.pem ./cacert.pem /host/etc/dragonfly-ca/
break
fi
done
domains="harbor.test.cn"
if [[ -n "$domains" ]]; then
for domain in $domains; do
# inject docker cert by registry domain
dir=/host/etc/docker/certs.d/$domain
mkdir -p "$dir"
echo copy CA cert to $dir
cp -f /host/etc/dragonfly-ca/cacert.pem "$dir/ca.crt"
done
fi
resources:
limits:
cpu: '2'
memory: 4Gi
requests:
cpu: '1'
memory: 2Gi
volumeMounts:
- name: etc
mountPath: /host/etc
- name: default-token-549c7
readOnly: true
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
imagePullPolicy: IfNotPresent
containers:
- name: dfdaemon
image: docker/dragonflyoss/dfdaemon:v2.0.3
ports:
- hostPort: 65001
containerPort: 65001
protocol: TCP
resources:
limits:
cpu: '2'
memory: 4Gi
requests:
cpu: '1'
memory: 2Gi
volumeMounts:
- name: config
mountPath: /etc/dragonfly
- name: etc
mountPath: /host/etc
- name: d7y-ca
mountPath: /etc/dragonfly-ca
- name: logs
mountPath: /var/log/dragonfly/daemon
- name: default-token-549c7
readOnly: true
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
livenessProbe:
exec:
command:
- /bin/grpc_health_probe
- '-addr=0.0.0.0:65000'
initialDelaySeconds: 15
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
exec:
command:
- /bin/grpc_health_probe
- '-addr=0.0.0.0:65000'
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
lifecycle:
postStart:
exec:
command:
- /bin/sh
- '-c'
- >
# inject hosts after dfdaemon started
domains="harbor.test.cn"
# remove static dns in pod /etc/hosts, which injected by host
network
sed -i '/# Dragonfly SNI Host/d' /etc/hosts
if [[ -n "$domains" ]]; then
for domain in $domains; do
# inject static dns into /host/etc/hosts
if grep "127.0.0.1 $domain" /host/etc/hosts; then
echo "Dragonfly SNI Host $domain Found in /host/etc/hosts"
continue
else
echo "Try to add dragonfly SNI host $domain"
echo "127.0.0.1 $domain # Dragonfly SNI Host $domain" >> /host/etc/hosts
echo "Dragonfly SNI host $domain added"
fi
done
fi
preStop:
exec:
command:
- /bin/sh
- '-c'
- >
# when stop dfdaemon, clean up injected hosts info in /etc/hosts
for current node
echo "$(sed '/# Dragonfly SNI Host/d' /host/etc/hosts)" >
/host/etc/hosts
imagePullPolicy: IfNotPresent
nodeSelector:
dragonfly: enable看一下 /etc/docker/certs.d/harbor.test.cn/ca.crt 的创建和修改时间，重建 pod 后，是否有变更？
另外问一下：
yangchuan37326
commented
2 years ago 1、重建pod后/etc/docker/certs.d/harbor.test.cn/ca.crt被重新复制了一遍，所以文件的创建时间变了，但我对比了里面的内容是与/etc/dragonfly-ca/cacert.pem一致的，且md5值一致。 2、现在是短暂下载不了镜像，还是一直下载不了镜像？ -- 重建pod后就一直下载不了镜像了 3、不能下载后，重启 docker 后，可否下载镜像了？ -- 重启过docker，宿主机也重启过，仍然下载不了，报同样的问题
jim3ma
commented
2 years ago 加我们的钉钉群吧，在线帮你看看
explore900620
commented
1 year ago 加我们的钉钉群吧，在线帮你看看
钉钉群怎么加入？扫码是“居民个案管理”
jim3ma
commented
1 year ago 最近好像都阳了，这个二维码我回头看看。 你先升级到最近的版本看看
karlhjm
commented
1 year ago 1、环境信息 dragonfly：2.0.3 docker：20.10.12 k8s：1.19.7 2、问题描述 通过官方提供的helm在k8s上部署dragonfly，第一次部署成功之后一切正常工作，node节点可下载镜像。当dfdaemon-pod异常k8s重新拉起一个新的pod之后，node节点不可下载镜像，报错如下： Error response from daemon: received unexpected HTTP status: 502 Bad Gateway 同时dfdaemon日志报错：
2022-10-10T03:11:10.030Z DEBUG proxy/proxy_sni.go:91 Generate temporal leaf TLS cert for ServerName <harbor.test.cn> 2022-10-10T03:11:10.037Z DEBUG transport/transport.go:180 round trip directly, method: GET, url: https://harbor.test.cn/v2/ 2022-10-10T03:11:10.037Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn> 2022-10-10T03:11:10.040Z ERROR proxy/proxy_sni.go:105 handshake failed for harbor.test.cn: remote error: tls: bad certificate d7y.io/dragonfly/v2/client/daemon/proxy.(*Proxy).handleTLSConn /go/src/d7y.io/dragonfly/v2/client/daemon/proxy/proxy_sni.go:105 2022/10/10 03:11:10 http: proxy error: x509: certificate signed by unknown authority 2022-10-10T03:11:10.042Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn> 2022-10-10T03:11:10.046Z DEBUG transport/transport.go:180 round trip directly, method: HEAD, url: https://harbor.test.cn/v2/rancher/rancher-agent/manifests/v2.5.0 2022-10-10T03:11:10.046Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn>3、dfdaemon配置文件
console: true verbose: false pprof-port: -1 jaeger: "" service-name: dragonfly-dfget aliveTime: duration: 0s gcInterval: duration: 1m0s metrics: "" workHome: /usr/local/dragonfly cacheDir: "" logDir: "" dataDir: /var/lib/dragonfly keepStorage: false scheduler: manager: enable: true netAddrs: - type: tcp addr: dragonfly.test.net:65003 refreshInterval: 5m0s seedPeer: enable: false type: super clusterID: 1 keepAlive: interval: 5s netAddrs: - type: tcp addr: 127.0.0.1:8002 scheduleTimeout: duration: 30s disableAutoBackSource: false host: securityDomain: "" idc: bjzdt netTopology: "" location: bj01 hostname: docker24.cloud listenIP: 0.0.0.0 advertiseIP: 10.17.18.5 download: defaultPattern: p2p totalRateLimit: limit: 2.097152e+09 perPeerRateLimit: limit: 1.048576e+09 pieceDownloadTimeout: 30s downloadGRPC: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: true tlsConfig: null unixListen: socket: /tmp/dfdamon.sock peerGRPC: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: true tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65000 end: 0 namespace: "" calculateDigest: true transportOption: null getPiecesMaxRetry: 100 prefetch: false watchdogTimeout: 0s proxy: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: false tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65001 end: 0 namespace: "" basicAuth: null defaultFilter: Expires&Signature&ns maxConcurrency: 0 registryMirror: url: https://index.docker.io dynamic: true certs: null insecure: true direct: false useProxies: false whiteList: [] proxies: - regx: blobs/sha256.* useHTTPS: false direct: false redirect: "" hijackHTTPS: cert: /etc/dragonfly-ca/cacert.pem key: /etc/dragonfly-ca/cakey.pem hosts: - regx: .* insecure: false certs: null sni: - listen: 127.0.0.1 port: start: 443 end: 0 namespace: "" dumpHTTPContent: false extraRegistryMirrors: [] upload: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: false tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65002 end: 0 namespace: "" rateLimit: limit: 5.24288e+08 storage: dataPath: "" taskExpireTime: duration: 6h0m0s diskGCThreshold: 50.0GB diskGCThresholdPercent: 95 multiplex: true strategy: io.d7y.storage.v2.simple health: null reloadOption: interval: duration: 1m0s4、排查 怀疑是证书问题，但我可以确定dfdaemon-pod重建之后使用的还是原来的证书。 请问这种情况是哪些配置参数设置的有问题吗
@jim3ma @yangchuan37326 请问这个问题最后解决了吗？我用的最新版本，安装完拉取镜像就遇到和这一样的报错，连镜像都下载不了
karlhjm
commented
1 year ago 1、环境信息 dragonfly：2.0.3 docker：20.10.12 k8s：1.19.7 2、问题描述 通过官方提供的helm在k8s上部署dragonfly，第一次部署成功之后一切正常工作，node节点可下载镜像。当dfdaemon-pod异常k8s重新拉起一个新的pod之后，node节点不可下载镜像，报错如下： Error response from daemon: received unexpected HTTP status: 502 Bad Gateway 同时dfdaemon日志报错：
2022-10-10T03:11:10.030Z DEBUG proxy/proxy_sni.go:91 Generate temporal leaf TLS cert for ServerName <harbor.test.cn> 2022-10-10T03:11:10.037Z DEBUG transport/transport.go:180 round trip directly, method: GET, url: https://harbor.test.cn/v2/ 2022-10-10T03:11:10.037Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn> 2022-10-10T03:11:10.040Z ERROR proxy/proxy_sni.go:105 handshake failed for harbor.test.cn: remote error: tls: bad certificate d7y.io/dragonfly/v2/client/daemon/proxy.(*Proxy).handleTLSConn /go/src/d7y.io/dragonfly/v2/client/daemon/proxy/proxy_sni.go:105 2022/10/10 03:11:10 http: proxy error: x509: certificate signed by unknown authority 2022-10-10T03:11:10.042Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn> 2022-10-10T03:11:10.046Z DEBUG transport/transport.go:180 round trip directly, method: HEAD, url: https://harbor.test.cn/v2/rancher/rancher-agent/manifests/v2.5.0 2022-10-10T03:11:10.046Z DEBUG proxy/proxy_sni.go:88 TLS Cache hit, cacheKey = <harbor.test.cn>3、dfdaemon配置文件
console: true verbose: false pprof-port: -1 jaeger: "" service-name: dragonfly-dfget aliveTime: duration: 0s gcInterval: duration: 1m0s metrics: "" workHome: /usr/local/dragonfly cacheDir: "" logDir: "" dataDir: /var/lib/dragonfly keepStorage: false scheduler: manager: enable: true netAddrs: - type: tcp addr: dragonfly.test.net:65003 refreshInterval: 5m0s seedPeer: enable: false type: super clusterID: 1 keepAlive: interval: 5s netAddrs: - type: tcp addr: 127.0.0.1:8002 scheduleTimeout: duration: 30s disableAutoBackSource: false host: securityDomain: "" idc: bjzdt netTopology: "" location: bj01 hostname: docker24.cloud listenIP: 0.0.0.0 advertiseIP: 10.17.18.5 download: defaultPattern: p2p totalRateLimit: limit: 2.097152e+09 perPeerRateLimit: limit: 1.048576e+09 pieceDownloadTimeout: 30s downloadGRPC: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: true tlsConfig: null unixListen: socket: /tmp/dfdamon.sock peerGRPC: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: true tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65000 end: 0 namespace: "" calculateDigest: true transportOption: null getPiecesMaxRetry: 100 prefetch: false watchdogTimeout: 0s proxy: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: false tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65001 end: 0 namespace: "" basicAuth: null defaultFilter: Expires&Signature&ns maxConcurrency: 0 registryMirror: url: https://index.docker.io dynamic: true certs: null insecure: true direct: false useProxies: false whiteList: [] proxies: - regx: blobs/sha256.* useHTTPS: false direct: false redirect: "" hijackHTTPS: cert: /etc/dragonfly-ca/cacert.pem key: /etc/dragonfly-ca/cakey.pem hosts: - regx: .* insecure: false certs: null sni: - listen: 127.0.0.1 port: start: 443 end: 0 namespace: "" dumpHTTPContent: false extraRegistryMirrors: [] upload: security: insecure: true caCert: "" cert: "" key: "" tlsVerify: false tlsConfig: null tcpListen: listen: 0.0.0.0 port: start: 65002 end: 0 namespace: "" rateLimit: limit: 5.24288e+08 storage: dataPath: "" taskExpireTime: duration: 6h0m0s diskGCThreshold: 50.0GB diskGCThresholdPercent: 95 multiplex: true strategy: io.d7y.storage.v2.simple health: null reloadOption: interval: duration: 1m0s4、排查 怀疑是证书问题，但我可以确定dfdaemon-pod重建之后使用的还是原来的证书。 请问这种情况是哪些配置参数设置的有问题吗
@jim3ma @yangchuan37326 请问这个问题最后解决了吗？我用的最新版本，安装完拉取镜像就遇到和这一样的报错，连镜像都下载不了
我的情况是因为helm安装时。halm的模板在添加proxy给docker的地方有问题，没成功给docker修改proxy，手动创建proxy给docker之后可以拉取了
1、环境信息 dragonfly：2.0.3 docker：20.10.12 k8s：1.19.7 2、问题描述 通过官方提供的helm在k8s上部署dragonfly，第一次部署成功之后一切正常工作，node节点可下载镜像。当dfdaemon-pod异常k8s重新拉起一个新的pod之后，node节点不可下载镜像，报错如下： Error response from daemon: received unexpected HTTP status: 502 Bad Gateway 同时dfdaemon日志报错：
3、dfdaemon配置文件
4、排查 怀疑是证书问题，但我可以确定dfdaemon-pod重建之后使用的还是原来的证书。 请问这种情况是哪些配置参数设置的有问题吗