search

portworx / px-dev

PX-Developer is scale-out storage for containers. Run Cassandra, Jenkins, or any application in Docker, with enterprise storage functionality on commodity servers
270 stars 83 forks source link

non sense space usage #57

Closed segator closed 6 years ago

segator commented 6 years ago

I have a cluster with 3 nodes, all the volumes have replica 3 but if I check the used disk size for every host, it doesn't appear exactly the same size usage, should be if I understand correctly how works portworx

Cluster Summary Cluster ID: portworx Cluster UUID: ---- Nodes: 3 node(s) with storage (3 online) IP ID StorageNode Used Capacity Status 10.14.128.0 ---- Yes 70 GiB 200 GiB Online (This node) 10.10.16.0 ---- Yes 118 GiB 200 GiB Online 10.14.160.0 ---- Yes 71 GiB 200 GiB Online Global Storage Pool Total Used : 259 GiB Total Capacity : 600 GiB

jvinod commented 6 years ago

if you have just 3 nodes and all the volumes are repl 3 then you should be consuming the same space in all nodes. The reported usage is the space allocated which can vary if you had a down node at anytime because that may change the write pattern and it should converge over time. Can you post the output /opt/pxctl -j volume list

segator commented 6 years ago

`[{ "id": "1015952309233458326", "readonly": false, "locator": { "name": "jenkins", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T12:07:28Z", "spec": { "ephemeral": false, "size": "53687091200", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "9398059008", "last_scan": "2017-12-31T12:07:28Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[92 92 0 0 0]", "TimestampBlocksTotal": "92", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "1047702641627643500", "readonly": false, "locator": { "name": "tautulli", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T12:17:21Z", "spec": { "ephemeral": false, "size": "5368709120", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "135708672", "last_scan": "2017-12-31T12:17:21Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "device_path": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_INTERNAL" },{ "id": "1076809325097857229", "readonly": false, "locator": { "name": "oscam", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-26T19:31:51Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "52408320", "last_scan": "2017-12-26T19:31:51Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "bef82b07-1b9c-4a21-ae35-213781aeb61b", "device_path": "/dev/pxd/pxd1076809325097857229", "attach_path": [ "/var/lib/osd/mounts/oscam" ], "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "1095024810687939084", "readonly": false, "locator": { "name": "prometheus-data", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T19:54:42Z", "spec": { "ephemeral": false, "size": "32212254720", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "4219371520", "last_scan": "2017-12-31T19:54:42Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[85184 85184 0 0 0]", "TimestampBlocksTotal": "85184", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "1120586685815830130", "readonly": false, "locator": { "name": "sickrage", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-21T23:58:33Z", "spec": { "ephemeral": false, "size": "10737418240", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "4194103296", "last_scan": "2017-12-21T23:58:33Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "device_path": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[8 8 0 0 0]", "TimestampBlocksTotal": "8", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "218306005218364939", "readonly": false, "locator": { "name": "prometheus-conf", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T19:54:15Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": true, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50663424", "last_scan": "2017-12-31T19:54:15Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "27072630042626804", "readonly": false, "locator": { "name": "plex-config", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T12:14:14Z", "spec": { "ephemeral": false, "size": "107374182400", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "51645358080", "last_scan": "2017-12-31T12:14:14Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "360375022102302899", "readonly": false, "locator": { "name": "deluge", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-21T23:40:17Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "54833152", "last_scan": "2017-12-21T23:40:17Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "device_path": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[105 105 0 0 0]", "TimestampBlocksTotal": "105", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "374631306495674726", "readonly": false, "locator": { "name": "jackett", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-22T00:54:20Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50683904", "last_scan": "2017-12-22T00:54:20Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "device_path": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "391032962982423852", "readonly": false, "locator": { "name": "simple-etcd" }, "ctime": "2017-12-31T16:45:13Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50749440", "last_scan": "2017-12-31T16:45:13Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "417292423831969158", "readonly": false, "locator": { "name": "grafana-data", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T19:52:55Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "56561664", "last_scan": "2017-12-31T19:52:55Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[4 4 0 0 0]", "TimestampBlocksTotal": "4", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "463448000803262042", "readonly": false, "locator": { "name": "traefik-acme" }, "ctime": "2017-12-22T01:31:23Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": true, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50765824", "last_scan": "2017-12-22T01:31:23Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "48000249624527221", "readonly": false, "locator": { "name": "ldap-etc", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T12:28:56Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50991104", "last_scan": "2017-12-31T12:28:56Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "570187626857924817", "readonly": false, "locator": { "name": "plex-logs" }, "ctime": "2017-12-31T12:15:25Z", "spec": { "ephemeral": false, "size": "5368709120", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "135708672", "last_scan": "2017-12-31T12:15:25Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "device_path": "46e50c3c-ff99-4f94-b875-3a19a32172fc", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_INTERNAL" },{ "id": "664575755678556783", "readonly": false, "locator": { "name": "tvheadend", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-22T02:14:14Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "83013632", "last_scan": "2017-12-22T02:14:14Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "bef82b07-1b9c-4a21-ae35-213781aeb61b", "device_path": "/dev/pxd/pxd664575755678556783", "attach_path": [ "/var/lib/osd/mounts/tvheadend" ], "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 287} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[20 20 0 0 0]", "TimestampBlocksTotal": "20", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "696187657595542787", "readonly": false, "locator": { "name": "grafana-conf", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T19:53:51Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50692096", "last_scan": "2017-12-31T19:53:51Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "847363518408233011", "readonly": false, "locator": { "name": "traefik-cert" }, "ctime": "2017-12-22T01:31:54Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": true, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "50688000", "last_scan": "2017-12-22T01:31:54Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "bef82b07-1b9c-4a21-ae35-213781aeb61b", "device_path": "/dev/pxd/pxd847363518408233011", "attach_path": [ "10.10.16.0", "192.168.1.18" ], "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "868983035634076472", "readonly": false, "locator": { "name": "ldap-var", "volume_labels": { "snapshot": "true" } }, "ctime": "2017-12-31T12:29:08Z", "spec": { "ephemeral": false, "size": "1073741824", "format": "ext4", "block_size": "32768", "ha_level": "3", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "61865984", "last_scan": "2017-12-31T12:29:08Z", "format": "ext4", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "46e50c3c-ff99-4f94-b875-3a19a32172fc", "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[0 1 5]", "ReplicaSetCurr": "[0 1 5]", "ReplicaSetCurrMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[0 1 5]", "ReplicaSetNextMid": "46e50c3c-ff99-4f94-b875-3a19a32172fc,369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{0 0} {1 0} {5 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[0 1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_EXTERNAL" },{ "id": "99558319756929467", "readonly": false, "locator": { "name": "test" }, "ctime": "2018-01-01T18:16:00Z", "spec": { "ephemeral": false, "size": "10737418240", "format": "none", "block_size": "32768", "ha_level": "2", "cos": "low", "io_profile": "sequential", "dedupe": false, "snapshot_interval": 0, "shared": false, "replica_set": {

}, "aggregation_level": 1, "encrypted": false, "passphrase": "", "snapshot_schedule": "", "scale": 1, "sticky": true, "group_enforced": false }, "usage": "0", "last_scan": "2018-01-01T18:16:00Z", "format": "none", "status": "up", "state": "attached", "attached_on": "369cfae7-4c96-47e4-9869-e68ae8656d78", "device_path": "369cfae7-4c96-47e4-9869-e68ae8656d78", "replica_sets": [ { "nodes": [ "369cfae7-4c96-47e4-9869-e68ae8656d78", "bef82b07-1b9c-4a21-ae35-213781aeb61b" ] } ], "error": "", "runtime_state": [ { "runtime_state": { "FullResyncBlocks": "[{1 0} {5 0} {-1 0} {-1 0} {-1 0}]", "ID": "0", "ReadQuorum": "1", "ReadSet": "[1 5]", "ReplicaSetCurr": "[1 5]", "ReplicaSetCurrMid": "369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ReplicaSetNext": "[1 5]", "ReplicaSetNextMid": "369cfae7-4c96-47e4-9869-e68ae8656d78,bef82b07-1b9c-4a21-ae35-213781aeb61b", "ResyncBlocks": "[{1 0} {5 0} {-1 0} {-1 0} {-1 0}]", "RuntimeState": "clean", "TimestampBlocksPerNode": "[0 0 0 0 0]", "TimestampBlocksTotal": "0", "WriteQuorum": "2", "WriteSet": "[1 5]" } } ], "secure_device_path": "", "attached_state": "ATTACH_STATE_INTERNAL" }]`

jvinod commented 6 years ago

One of the volume is repl 2 - but that doesn't explain it either. Could you join our slack channel slack.portworx.com - we can get this resolved with more direct communication

segator commented 6 years ago

hey, yes Sorry this was a test vol, but its empty. Anyway now after some days I checked again pxctl status and now mark 70 70 71G, so only 1G of difference, it seems like a internal process or something that "purge" unreferenced blocks or something

jvinod commented 6 years ago

Sounds good, closing as this seems resolved.