search

todaygood / openshift-lab

lab on openshift
0 stars 0 forks source link

openshift部署完成之后push image 到自带的registry失败 #2

Open todaygood opened 6 years ago

todaygood commented 6 years ago

Issue

[root@ose0 ~]# docker push 172.30.165.209:5000/openshift/busybox 
The push refers to a repository [172.30.165.209:5000/openshift/busybox]
8e9a7d50b12c: Retrying in 1 second 
received unexpected HTTP status: 500 Internal Server Error

Steps

  1. 在ose5上创建nfs server
配置/etc/fstab , mount -a 
[root@ose5 /]# df -h |grep data 
/dev/vdg        200G   33M  200G   1% /data5
/dev/vdc        2.0G   33M  2.0G   2% /data1
/dev/vdd         10G   33M   10G   1% /data2
/dev/vde         20G   33M   20G   1% /data3
/dev/vdf         30G   33M   30G   1% /data4

[root@ose5 /]# mount |grep data
/dev/vdg on /data5 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vdc on /data1 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vdd on /data2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vde on /data3 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vdf on /data4 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

配置/etc/exports, systemctl start nfs-server ; systemctl enable nfs-server
[root@ose5 /]# exportfs -v 
/data1          <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,root_squash,no_all_squash)
/data2          <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,root_squash,no_all_squash)
/data3          <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,root_squash,no_all_squash)
/data4          <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,root_squash,no_all_squash)
/data5          <world>(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,root_squash,no_all_squash)
  1. 创建PV , PVC 
vim pv.yml 
vim pvc.yml
oc create -f pv.yml 
oc crate -f  pvc.yml 
[root@ose0 ~]# oc get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                  STORAGECLASS   REASON    AGE
nfs-data1   2Gi        RWX            Retain           Available                                                   3h
nfs-data2   10Gi       RWX            Retain           Available                                                   3h
nfs-data3   20Gi       RWX            Retain           Available                                                   3h
nfs-data4   30Gi       RWX            Retain           Available                                                   3h
nfs-data5   300Gi      RWX            Retain           Bound       default/nfs-registry                            3h
  1. 在registry pod 上添加volume （用pvc指定） 
    
oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc \
 --claim-name=nfs-registry --overwrite

[root@ose0 ~]# oc get pods NAME READY STATUS RESTARTS AGE docker-registry-2-5zpz2 1/1 Running 0 2h registry-console-1-b6sht 1/1 Running 0 2d router-1-lnnxj 1/1 Running 0 4d router-1-z2mmc 1/1 Running 0 4d


4. 验证
```bash
[root@ose0 ~]# oc rsh docker-registry-2-5zpz2 
sh-4.2$ cd /registry/
sh-4.2$ ls

发现是空的。

  1. 创建用户，赋予权限 htpasswd /etc/origin/master/htpasswd margin oc adm policy add-role-to-user system:registry margin oc adm policy add-role-to-user admin magin -n openshift oc adm policy add-role-to-user system:image-builder margin

  2. 登录registry ,push image 找到docker registry service地址

    [root@ose0 ~]# oc get service
NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
docker-registry    ClusterIP   172.30.165.209   <none>        5000/TCP                  4d
kubernetes         ClusterIP   172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     4d
registry-console   ClusterIP   172.30.82.59     <none>        9000/TCP                  4d
router             ClusterIP   172.30.230.149   <none>        80/TCP,443/TCP,1936/TCP   4d

    是172.30.165.209:5000

    oc login -u margin 
[root@ose0 ~]# oc whoami -t 
A7KYqKCSC2IuqOVUcydOwsogh3dfMsLvMFJxNmcktXI
[root@ose0 ~]# docker login -u margin -p A7KYqKCSC2IuqOVUcydOwsogh3dfMsLvMFJxNmcktXI 172.30.165.209:5000
Login Succeeded
[root@ose0 ~]# docker push 172.30.165.209:5000/openshift/busybox 
The push refers to a repository [172.30.165.209:5000/openshift/busybox]
8e9a7d50b12c: Retrying in 1 second 
received unexpected HTTP status: 500 Internal Server Error
todaygood commented 6 years ago

Investigation

找到该pod运行在哪个node上

[root@ose0 ~]# oc get pods -o wide 
NAME                       READY     STATUS    RESTARTS   AGE       IP                NODE
docker-registry-2-5zpz2    1/1       Running   0          3h        10.130.2.21       ose7.cloud.genomics.cn

ssh到该node, 发现nfs 已经mount上了。

[root@ose7 ~]# mount |grep nfs 
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.122.104:/data5 on /var/lib/origin/openshift.local.volumes/pods/4ef87fb1-92a8-11e8-a64a-525400a1189c/volumes/kubernetes.io~nfs/nfs-data5 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.103,local_lock=none,addr=192.168.122.104)

发现nfs目录写入失败 [root@ose7 ~]# echo "abc" > /var/lib/origin/openshift.local.volumes/pods/4ef87fb1-92a8-11e8-a64a-525400a1189c/volumes/kubernetes.io~nfs/nfs-data5/abc-bash: /var/lib/origin/openshift.local.volumes/pods/4ef87fb1-92a8-11e8-a64a-525400a1189c/volumes/kubernetes.io~nfs/nfs-data5/abc: Permission denied

Solution

[root@ose5 /]# chmod 777 data* -R

验证docker push成功，

[root@ose0 ~]# docker push 172.30.165.209:5000/openshift/busybox
The push refers to a repository [172.30.165.209:5000/openshift/busybox]
8e9a7d50b12c: Pushed 
latest: digest: sha256:1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5 size: 527

可以看到registry目录是这样的： image