search

portworx / px-dev

PX-Developer is scale-out storage for containers. Run Cassandra, Jenkins, or any application in Docker, with enterprise storage functionality on commodity servers
270 stars 84 forks source link

Failed to initialize node in cluster and devices path is #70

Closed Jeffry-ye closed 5 years ago

Jeffry-ye commented 5 years ago

portworx/px-enterprise version: 1.6.1.3

1.add one node(10.26.15.103) to cluster Failed to initialize. 2.devices:/dev/sdb1 of node(10.26.85.149) is 800GB, but pxctl status detail Capacity(10.26.85.149) 2.2 TiB

/opt/pwx/bin/px-runc run -c MY_CLUSTER_ID \ -k etcd://10.26.15.103:2379 \ -s /dev/sdb1

time="2018-11-14T16:23:34Z" level=info msg="Global license watcher installed." time="2018-11-14T16:23:34Z" level=info msg="Trial configured successfully" time="2018-11-14T16:23:34Z" level=info msg="PX starting cluster manager..." time="2018-11-14T16:23:34Z" level=info msg="PX cluster manager running." time="2018-11-14T16:23:34Z" level=info msg="Starting server on port: 9001" time="2018-11-14T16:23:34Z" level=info msg="PX API server running on port 9001." time="2018-11-14T16:23:34Z" level=info msg="Adding cluster event listener: Scheduler" time="2018-11-14T16:23:34Z" level=info msg="Starting API Server with TLS Disabled." time="2018-11-14T16:23:34Z" level=info msg="Starting Watchdog server." time="2018-11-14T16:23:40Z" level=info msg="Adding cluster event listener: PX Storage Service" time="2018-11-14T16:23:40Z" level=info msg="Cluster manager starting..." time="2018-11-14T16:23:40Z" level=info msg="initializing osdconfig manager" time="2018-11-14T16:23:40Z" level=info msg="Cluster state is OK... Joining the cluster." time="2018-11-14T16:23:40Z" level=info msg="Node 27afe747-ae4b-4a1d-8131-24dc0b082911 joining cluster..." time="2018-11-14T16:23:40Z" level=info msg="Cluster ID: MY_CLUSTER_ID" time="2018-11-14T16:23:40Z" level=info msg="Node Mgmt IP: 10.26.15.103" time="2018-11-14T16:23:40Z" level=info msg="Node Data IP: 10.26.15.103" time="2018-11-14T16:23:40Z" level=error msg="This node is already initialized but could not be found in the cluster map." time="2018-11-14T16:23:40Z" level=error msg="Failed to initialize node in cluster. This node is already initialized but could not be found in the cluster map." time="2018-11-14T16:23:40Z" level=error msg="Init Failure: This node is already initialized but could not be found in the cluster map. Please remove /etc/pwx/.private.json to initialize a new node."

[root@k8s-ceph-node-15-103 ~]# cat /etc/pwx/config.json { "alertingurl": "", "clusterid": "MY_CLUSTER_ID", "dataiface": "", "kvdb": [ "etcd://10.26.15.103:2379" ], "mgtiface": "", "scheduler": "none", "storage": { "devices": [ "/dev/sdb1" ], "journal_dev": "", "max_storage_nodes_per_zone": 0, "system_metadata_dev": "" }, "version": "1.0" }

[root@k8s-ceph-node-15-103 ~]# /opt/pwx/bin/pxctl status Status: PX is initializing... License: Trial (expires in 31 days) Node ID: 27afe747-ae4b-4a1d-8131-24dc0b082911 IP: Unavailable Local Storage Pool: 0 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION No storage pool Local Storage Devices: 0 device Device Path Media Type Size Last-Scan No storage device total-0 B Cluster Summary Cluster ID: MY_CLUSTER_ID Cluster UUID: 0990a22b-b563-4cc7-a216-3574a7399da6 Scheduler: none IPIDStorageNodeUsedCapacityStatusStorageStatusVersionKernelOS Global Storage Pool Total Used : 0 B Total Capacity: 0 B

[root@k8s-ceph-node-75-102 ~]# /opt/pwx/bin/pxctl status Status: PX is operational License: Trial (expires in 31 days) Node ID: 5a8601a8-325e-4f05-a47e-42cbf8a63285 IP: 10.26.75.102 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 HIGH raid0 800 GiB 11 GiB Online default default Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/sdb1 STORAGE_MEDIUM_SSD 800 GiB 14 Nov 18 21:33 CST total - 800 GiB Cluster Summary Cluster ID: MY_CLUSTER_ID Cluster UUID: 0990a22b-b563-4cc7-a216-3574a7399da6 Scheduler: none Nodes: 2 node(s) with storage (2 online) IP ID StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.26.85.149 a29c955c-8213-4645-87fc-36cf43809350 Yes 20 GiB 2.2 TiB Online Up 1.6.1.3-61b6770 3.10.0-327.el7.x86_64 CentOS Linux 7 (Core) 10.26.75.102 5a8601a8-325e-4f05-a47e-42cbf8a63285 Yes 11 GiB 800 GiB Online Up (This node) 1.6.1.3-61b6770 3.10.0-327.36.3.el7.x86_64 CentOS Linux 7 (Core) Global Storage Pool Total Used : 31 GiB Total Capacity: 3.0 TiB

[root@k8s-ceph-node-85-149 log]# /opt/pwx/bin/pxctl status Status: PX is operational License: Trial (expires in 31 days) Node ID: a29c955c-8213-4645-87fc-36cf43809350 IP: 10.26.85.149 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 HIGH raid0 2.2 TiB 20 GiB Online default default Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/sdb STORAGE_MEDIUM_SSD 2.2 TiB 14 Nov 18 21:33 CST total-2.2 TiB Cluster Summary Cluster ID: MY_CLUSTER_ID Cluster UUID: 0990a22b-b563-4cc7-a216-3574a7399da6 Scheduler: none Nodes: 2 node(s) with storage (2 online) IP ID StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.26.85.149 a29c955c-8213-4645-87fc-36cf43809350 Yes 20 GiB 2.2 TiB Online Up (This node) 1.6.1.3-61b6770 3.10.0-327.el7.x86_64 CentOS Linux 7 (Core) 10.26.75.102 5a8601a8-325e-4f05-a47e-42cbf8a63285 Yes 11 GiB 800 GiB Online Up 1.6.1.3-61b6770 3.10.0-327.36.3.el7.x86_64 CentOS Linux 7 (Core) Global Storage Pool Total Used : 31 GiB Total Capacity: 3.0 TiB

[root@k8s-ceph-node-85-149 log]# cat /etc/pwx/config.json { "alertingurl": "", "clusterid": "MY_CLUSTER_ID", "dataiface": "", "kvdb": [ "etcd://10.26.85.149:2379" ], "mgtiface": "", "scheduler": "none", "storage": { "devices": [ "/dev/sdb1" ], "journal_dev": "", "max_storage_nodes_per_zone": 0, "rt_opts": {}, "system_metadata_dev": "" }, "version": "1.0" }

[root@k8s-ceph-node-75-102 ~]# cat /etc/pwx/config.json { "alertingurl": "", "clusterid": "MY_CLUSTER_ID", "dataiface": "", "kvdb": [ "etcd://10.26.75.102:2379" ], "mgtiface": "", "scheduler": "none", "storage": { "devices": [ "/dev/sdb1" ], "journal_dev": "", "max_storage_nodes_per_zone": 0, "system_metadata_dev": "" }, "version": "1.0" }

[root@k8s-ceph-node-85-149 log]# fdisk -l ... ...

Disk /dev/sdb: 2400.3 GB, 2400348626944 bytes, 4688180912 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 262144 bytes / 1310720 bytes Disk label type: dos Disk identifier: 0x6d6e5144

Device Boot Start End Blocks Id System /dev/sdb1 2560 1677724159 838860800 83 Linux /dev/sdb2 1677724160 3355445759 838860800 83 Linux /dev/sdb3 3355445760 4294967294 469760767+ 83 Linux

Jeffry-ye commented 5 years ago

Thanks @adityadani ,the problem has been solved.

To reinitialize a PX node, you need to wipe the node of old PX fingerprint data. You can do that by running the following command on that node $ /opt/pwx/bin/pxctl service node-wipe