search

ClusterHQ / flocker

Container data volume manager for your Dockerized application
https://clusterhq.com
Apache License 2.0
3.39k stars 290 forks source link

flocker with two node , when node1 down and use flockerctl move volume to node2,but volume status always pending #2905

Open 332054781 opened 8 years ago

332054781 commented 8 years ago

1. i use redhat7.2 so i can't use flocker-diagnostics

2.flocker-control --version

1.15.0

3.cat /etc/os-release

NAME="Red Hat Enterprise Linux Server" VERSION="7.2 (Maipo)" ID="rhel" ID_LIKE="fedora" VERSION_ID="7.2" PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.2 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.2"

4.uname -a

Linux scio01 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

5.docker version

Client: Version: 1.10.3 API version: 1.22 Go version: go1.5.3 Git commit: 20f81dd Built: Thu Mar 10 15:39:25 2016 OS/Arch: linux/amd64

Server: Version: 1.10.3 API version: 1.22 Go version: go1.5.3 Git commit: 20f81dd Built: Thu Mar 10 15:39:25 2016 OS/Arch: linux/amd64

6. docker info

docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 3 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: docker-202:2-26672685-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 1.114 GB Data Space Total: 107.4 GB Data Space Available: 3.327 GB Metadata Space Used: 2.175 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.145 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local flocker Network: bridge null host Kernel Version: 3.10.0-327.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.451 GiB Name: scio01 ID: PIP4:Y5M5:TXRT:LBQK:5LK2:QVBF:XVUY:2ZUY:NXBR:AJE5:N4SH:WPUF WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled

6.ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000 link/ether 06:25:bc:9e:7f:a8 brd ff:ff:ff:ff:ff:ff inet 192.168.9.14/22 brd 192.168.11.255 scope global dynamic eth0 valid_lft 3332sec preferred_lft 3332sec inet6 fe80::425:bcff:fe9e:7fa8/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN link/ether 02:42:28:dd:5b:73 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:28ff:fedd:5b73/64 scope link valid_lft forever preferred_lft forever

7.hostname

scio01

8.fdisk -l

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/xvda: 10.7 GB, 10737418240 bytes, 20971520 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt

Start End Size Type Name

1 2048 4095 1M BIOS boot parti 2 4096 20971486 10G Microsoft basic

Disk /dev/xvdg: 32.2 GB, 32212254720 bytes, 62914560 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/xvdf: 107.4 GB, 107374182400 bytes, 209715200 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/docker-202:2-26672685-pool: 107.4 GB, 107374182400 bytes, 209715200 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 65536 bytes / 65536 bytes

Disk /dev/scinia: 25.8 GB, 25769803776 bytes, 50331648 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/scinib: 17.2 GB, 17179869184 bytes, 33554432 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/scinid: 85.9 GB, 85899345920 bytes, 167772160 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/scinif: 8589 MB, 8589934592 bytes, 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/scinie: 85.9 GB, 85899345920 bytes, 167772160 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/scinig: 85.9 GB, 85899345920 bytes, 167772160 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

9.lsblk --all

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 10G 0 disk ├─xvda1 202:1 0 1M 0 part └─xvda2 202:2 0 10G 0 part / xvdf 202:80 0 100G 0 disk xvdg 202:96 0 30G 0 disk loop0 7:0 0 100G 0 loop └─docker-202:2-26672685-pool 252:0 0 100G 0 dm loop1 7:1 0 2G 0 loop └─docker-202:2-26672685-pool 252:0 0 100G 0 dm scinia 253:0 0 24G 0 disk /flocker/adaaae6e-879e-4500-a4f3-dceb31777b51 scinib 253:16 0 16G 0 disk /flocker/24b7806d-f64a-4446-a186-ad43e8b98396 scinid 253:48 0 80G 0 disk /flocker/1e29d3e1-0efd-4df1-925c-ea9ee6d7b2a2 scinie 253:64 0 80G 0 disk /flocker/34208f5c-a701-4cc9-a10c-c6910b32098b scinif 253:80 0 8G 0 disk /flocker/76396940-aede-4258-a171-4fa560b51a9a scinig 253:96 0 80G 0 disk /flocker/11cbe7c5-6c99-4058-afe9-bdb32e957305

10.lshw -quiet -json

{ "id" : "scio01", "class" : "system", "claimed" : true, "handle" : "DMI:0100", "description" : "Computer", "product" : "HVM domU", "vendor" : "Xen", "version" : "4.2.amazon", "serial" : "ec2403e1-153b-e919-c571-34f9c06587aa", "width" : 64, "configuration" : { "boot" : "normal", "uuid" : "EC2403E1-153B-E919-C571-34F9C06587AA" }, "capabilities" : { "smbios-2.4" : "SMBIOS version 2.4", "dmi-2.4" : "DMI version 2.4", "vsyscall32" : "32-bit processes" }, "children" : [ { "id" : "core", "class" : "bus", "claimed" : true, "description" : "Motherboard", "physid" : "0", "children" : [ { "id" : "firmware:0", "class" : "memory", "claimed" : true, "description" : "BIOS", "vendor" : "Xen", "physid" : "0", "version" : "4.2.amazon", "date" : "05/12/2016", "units" : "bytes", "size" : 98304, "capabilities" : { "pci" : "PCI bus", "edd" : "Enhanced Disk Drive extensions" } }, { "id" : "cpu:0", "class" : "processor", "claimed" : true, "handle" : "DMI:0401", "description" : "CPU", "product" : "Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz", "vendor" : "Intel Corp.", "physid" : "1", "businfo" : "cpu@0", "slot" : "CPU 1", "units" : "Hz", "size" : 2400000000, "capacity" : 2400000000, "width" : 64, "capabilities" : { "fpu" : "mathematical co-processor", "fpu_exception" : "FPU exceptions reporting", "wp" : true, "vme" : "virtual mode extensions", "de" : "debugging extensions", "pse" : "page size extensions", "tsc" : "time stamp counter", "msr" : "model-specific registers", "pae" : "4GB+ memory addressing (Physical Address Extension)", "mce" : "machine check exceptions", "cx8" : "compare and exchange 8-byte", "apic" : "on-chip advanced programmable interrupt controller (APIC)", "sep" : "fast system calls", "mtrr" : "memory type range registers", "pge" : "page global enable", "mca" : "machine check architecture", "cmov" : "conditional move instruction", "pat" : "page attribute table", "pse36" : "36-bit page size extensions", "clflush" : true, "mmx" : "multimedia extensions (MMX)", "fxsr" : "fast floating point save/restore", "sse" : "streaming SIMD extensions (SSE)", "sse2" : "streaming SIMD extensions (SSE2)", "ht" : "HyperThreading", "syscall" : "fast system calls", "nx" : "no-execute bit (NX)", "rdtscp" : true, "x86-64" : "64bits extensions (x86-64)", "constant_tsc" : true, "rep_good" : true, "nopl" : true, "xtopology" : true, "eagerfpu" : true, "pni" : true, "pclmulqdq" : true, "ssse3" : true, "fma" : true, "cx16" : true, "pcid" : true, "sse4_1" : true, "sse4_2" : true, "x2apic" : true, "movbe" : true, "popcnt" : true, "tsc_deadline_timer" : true, "aes" : true, "xsave" : true, "avx" : true, "f16c" : true, "rdrand" : true, "hypervisor" : true, "lahf_lm" : true, "abm" : true, "fsgsbase" : true, "bmi1" : true, "avx2" : true, "smep" : true, "bmi2" : true, "erms" : true, "invpcid" : true, "xsaveopt" : true } }, { "id" : "cpu:1", "class" : "processor", "claimed" : true, "handle" : "DMI:0402", "description" : "CPU", "vendor" : "Intel", "physid" : "2", "businfo" : "cpu@1", "slot" : "CPU 2", "units" : "Hz", "size" : 2400000000, "capacity" : 2400000000 }, { "id" : "memory:0", "class" : "memory", "claimed" : true, "handle" : "DMI:1000", "description" : "System Memory", "physid" : "3", "children" : [ { "id" : "bank:0", "class" : "memory", "claimed" : true, "handle" : "DMI:1100", "description" : "DIMM RAM", "physid" : "0", "slot" : "DIMM 0", "units" : "bytes", "size" : 4294967296, "width" : 64 }, { "id" : "bank:1", "class" : "memory", "claimed" : true, "handle" : "DMI:1100", "description" : "DIMM RAM", "physid" : "1", "slot" : "DIMM 0", "units" : "bytes", "size" : 4294967296, "width" : 64 } ] }, { "id" : "firmware:1", "class" : "memory", "claimed" : true, "description" : "BIOS", "vendor" : "Xen", "physid" : "4", "version" : "4.2.amazon", "date" : "05/12/2016", "units" : "bytes", "size" : 98304, "capabilities" : { "pci" : "PCI bus", "edd" : "Enhanced Disk Drive extensions" } }, { "id" : "cpu:2", "class" : "processor", "claimed" : true, "handle" : "DMI:0401", "description" : "CPU", "vendor" : "Intel", "physid" : "5", "businfo" : "cpu@2", "slot" : "CPU 1", "units" : "Hz", "size" : 2400000000, "capacity" : 2400000000 }, { "id" : "cpu:3", "class" : "processor", "claimed" : true, "handle" : "DMI:0402", "description" : "CPU", "vendor" : "Intel", "physid" : "6", "businfo" : "cpu@3", "slot" : "CPU 2", "units" : "Hz", "size" : 2400000000, "capacity" : 2400000000 }, { "id" : "memory:1", "class" : "memory", "claimed" : true, "handle" : "DMI:1000", "description" : "System Memory", "physid" : "7" }, { "id" : "memory:2", "class" : "memory", "physid" : "8" }, { "id" : "memory:3", "class" : "memory", "physid" : "9" }, { "id" : "pci", "class" : "bridge", "claimed" : true, "handle" : "PCIBUS:0000:00", "description" : "Host bridge", "product" : "440FX - 82441FX PMC [Natoma]", "vendor" : "Intel Corporation", "physid" : "100", "businfo" : "pci@0000:00:00.0", "version" : "02", "width" : 32, "clock" : 33000000, "children" : [ { "id" : "isa", "class" : "bridge", "claimed" : true, "handle" : "PCI:0000:00:01.0", "description" : "ISA bridge", "product" : "82371SB PIIX3 ISA [Natoma/Triton II]", "vendor" : "Intel Corporation", "physid" : "1", "businfo" : "pci@0000:00:01.0", "version" : "00", "width" : 32, "clock" : 33000000, "configuration" : { "latency" : "0" }, "capabilities" : { "isa" : true, "bus_master" : "bus mastering" } }, { "id" : "ide", "class" : "storage", "claimed" : true, "handle" : "PCI:0000:00:01.1", "description" : "IDE interface", "product" : "82371SB PIIX3 IDE [Natoma/Triton II]", "vendor" : "Intel Corporation", "physid" : "1.1", "businfo" : "pci@0000:00:01.1", "version" : "00", "width" : 32, "clock" : 33000000, "configuration" : { "driver" : "ata_piix", "latency" : "64" }, "capabilities" : { "ide" : true, "bus_master" : "bus mastering" } }, { "id" : "bridge", "class" : "bridge", "handle" : "PCI:0000:00:01.3", "description" : "Bridge", "product" : "82371AB/EB/MB PIIX4 ACPI", "vendor" : "Intel Corporation", "physid" : "1.3", "businfo" : "pci@0000:00:01.3", "version" : "01", "width" : 32, "clock" : 33000000, "configuration" : { "latency" : "0" }, "capabilities" : { "bridge" : true, "bus_master" : "bus mastering" } }, { "id" : "display", "class" : "display", "claimed" : true, "handle" : "PCI:0000:00:02.0", "description" : "VGA compatible controller", "product" : "GD 5446", "vendor" : "Cirrus Logic", "physid" : "2", "businfo" : "pci@0000:00:02.0", "version" : "00", "width" : 32, "clock" : 33000000, "configuration" : { "driver" : "cirrus", "latency" : "0" }, "capabilities" : { "vga_controller" : true, "bus_master" : "bus mastering", "rom" : "extension ROM" } }, { "id" : "generic", "class" : "generic", "claimed" : true, "handle" : "PCI:0000:00:03.0", "description" : "Unassigned class", "product" : "Xen Platform Device", "vendor" : "XenSource, Inc.", "physid" : "3", "businfo" : "pci@0000:00:03.0", "version" : "01", "width" : 32, "clock" : 33000000, "configuration" : { "driver" : "xen-platform-pci", "latency" : "0" }, "capabilities" : { "bus_master" : "bus mastering" } } ] } ] }, { "id" : "network", "class" : "network", "claimed" : true, "description" : "Ethernet interface", "physid" : "1", "logicalname" : "eth0", "serial" : "06:25:bc:9e:7f:a8", "configuration" : { "broadcast" : "yes", "driver" : "vif", "ip" : "192.168.9.14", "link" : "yes", "multicast" : "yes" }, "capabilities" : { "ethernet" : true, "physical" : "Physical interface" } } ] }

11. flockerctl ls

DATASET SIZE METADATA STATUS SERVER a64c1eb9-fccb-4abc-bce8-8c2d8ac0d3f9 75.00G name=t7 pending ⌛ e14b9a63 (192.168.9.15) 76396940-aede-4258-a171-4fa560b51a9a 1.00G name=t111 attached ✅ b95d8760 (192.168.9.14) adaaae6e-879e-4500-a4f3-dceb31777b51 10.00G name=t9 attached ✅ b95d8760 (192.168.9.14) 24b7806d-f64a-4446-a186-ad43e8b98396 10.00G name=t70 attached ✅ b95d8760 (192.168.9.14) 335aaf69-f702-48da-bcd4-a493efc6e584 10.00G name=t10 attached ✅ e14b9a63 (192.168.9.15) c9da4209-9f27-4dc7-a270-11f9e4d93c15 75.00G name=2df48cd7db5ce266674c4671bb14343d16b446 pending ⌛ b95d8760 (192.168.9.14) 8a4db6b31f0275847d2d5b295e 34208f5c-a701-4cc9-a10c-c6910b32098b 75.00G name=t11 attached ✅ b95d8760 (192.168.9.14) cd805d9b-2a88-4f8d-a150-4a4f45975c80 20.00G name=t8 attached ✅ e14b9a63 (192.168.9.15) 1e29d3e1-0efd-4df1-925c-ea9ee6d7b2a2 75.00G name=apples attached ✅ b95d8760 (192.168.9.14) 11cbe7c5-6c99-4058-afe9-bdb32e957305 75.00G name=fa047c904b7ffe56349b9145ec5f4a1148d818 attached ✅ b95d8760 (192.168.9.14) f2b52c4152c257b44e13b50634

12.journalctl --all --output cat -u flocker-dataset-agent --priority=err | eliot-prettyprint

Failed to get data: Cannot assign requested address

wallnerryan commented 8 years ago

can you try journalctl --all -u flocker-dataset-agent --no-pager, also, if you run a curl against the state and configuration datasets API what do you see

curl --cacert /etc/flocker/cluster.crt \
   --cert /etc/flocker/api.crt \
   --key /etc/flocker/api.key \
   --header 'Content-type: application/json' \
   https://ceph1:4523/v1/state/datasets| python -m json.tool
curl --cacert /etc/flocker/cluster.crt \
   --cert /etc/flocker/api.crt \
   --key /etc/flocker/api.key \
   --header 'Content-type: application/json' \
   https://ceph1:4523/v1/configuration/datasets| python -m json.tool