Error InvalidVolume.ZoneMismatch in scalingUP

I've setup a statefull ami accordig to this guide. In my setup, I've chosen to not use a systemd service startup script, instead, I'm just using --restart=unless-stopped from docker run parameter.

I've created 3 vols in 3 different AZs: sa-east-1a, sa-east-b and sa-east-1c, each one with 30GB:

To create the px container in my statefull ami, I've did the bellow steps:

mkdir -p /etc/pwx
mkdir -p /opt/pwx/bin
mkdir -p /var/lib/osd
mkdir -p /var/cores

docker pull portworx/px-dev:latest

# The bellow command DOESN'T starts the container. It just creates it.
/usr/bin/docker create --net=host --privileged=true     \
      --restart=unless-stopped                       \
      -v /run/docker/plugins:/run/docker/plugins     \
      -v /var/lib/osd:/var/lib/osd:shared            \
      -v /dev:/dev                                   \
      -v /etc/pwx:/etc/pwx                           \
      -v /opt/pwx/bin:/export_bin:shared             \
      -v /var/run/docker.sock:/var/run/docker.sock   \
      -v /var/cores:/var/cores                       \
      -v /usr/src:/usr/src                           \
      --name=portworx \
      portworx/px-dev -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac

After that, I've created the ami and launched a new instance from it with this userdata:

#cloud-config
portworx:
  config:
    storage:
      devices:
        - vol-0842059ddbdbd9d7f
        - vol-058d9c98020c8d4c9
        - vol-002334b285798b4ac
      max-count: 10

runcmd:
 - [ /usr/bin/git, clone, "git@github.com:myenv/repo.git", /opt/docker/ ]
 - [ /bin/bash, -c, "cd /opt/docker/ ; git pull" ]
 - [ /opt/docker/scripts/userdata/userdata.sh, userdata_infra.conf ]

After instance is running, pxctl status shows this:

PX is not running on this host

Checking the AWS console, I've noticed that portworx created a lot of 30GB disks in all 3 zones!

This is the container log. It seems that it entered in a looping state:

Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:19:36 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
checking /hostusr/src/kernels/4.4.0-83-generic
checking /hostusr/src/linux-headers-4.4.0-83-generic
checking /usr/src/kernels/4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Flushing buffer-cache..
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Creating px fs...
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
Fingerprints values:
HOSTNAME={"type": 1, "value": "ip-10-0-1-73"}
DISK={ "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }
NETWORK={ "type": 7, "value": "" }
NETWORK={ "type": 8, "value": "" }

FP_FINGERPRINT="Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }]

FP_CONFIG={ "token": "", "create_profile": true, "profile_name": "ip-10-0-1-73-NoqPxviZ", "Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }] }
To use the same hardware configuration on an identical node,        please use hardware profile ip-10-0-1-73-NoqPxviZ.
example:
    curl -fsSL http://lighthouse.portworx.com/px_bootstrap.sh > px_bootstrap.sh;  chmod u+x px_bootstrap.sh;  PWX_TOKEN= PWX_CLUSTERNAME=socialbase         bootstrap.sh -p ip-10-0-1-73-NoqPxviZ
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:20:03 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
PXD version:  33cafba6c39c862340a8c30b2677849f67bd2d6a
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
2017-07-17 20:20:04,943 CRIT Supervisor running as root (no user in config file)
2017-07-17 20:20:04,947 INFO supervisord started with pid 1
2017-07-17 20:20:05,949 INFO spawned: 'relayd' with pid 75
2017-07-17 20:20:05,950 INFO spawned: 'lttng' with pid 76
2017-07-17 20:20:05,952 INFO spawned: 'exec' with pid 77
2017-07-17 20:20:05,953 INFO spawned: 'pxdaemon' with pid 78
2017-07-17 20:20:05,954 INFO spawned: 'px-ns' with pid 79
2017-07-17 20:20:05,956 INFO spawned: 'px_event_listener' with pid 80
PXPROCS: lttng not started yet...sleeping 

Mon Jul 17 20:20:06 UTC 2017 cleaning up old traces
Mon Jul 17 20:20:06 UTC 2017 done cleaning up old traces
time="2017-07-17T20:20:07Z" level=info msg="px-ns Starting.." 
2017-07-17 20:20:07,139 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,139 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px-ns entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
NS client starting fuse module
Starting NS server
PXPROCS: lttng not started yet...sleeping 

PXPROCS: lttng not started yet...sleeping 

PXPROCS: lttng not started yet...sleeping 

Spawning a session daemon
Session pxd created.
Traces will be written in net://localhost
Live timer set to 2000000 usec
Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170717-202016.
UST channel pxd_channel enabled for session pxd
All UST events are enabled in channel pxd_channel
Tracing started for session pxd
PXPROCS: Started px-storage with pid 152
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
PXPROCS: Started px with pid 163
PXPROCS: Started watchdog with pid 164
2017-07-17_20:20:21: PX-Watchdog: Starting watcher
2017-07-17_20:20:22: PX-Watchdog: Waiting for px process to start
root       163    78  0 20:20 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:20:23: PX-Watchdog: (pid 163): Begin monitoring
time="2017-07-17T20:20:28Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:20:28Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:20:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b35b6ce-8d86-41dc-9636-373f359a3af9). Trying attachSet..." 
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:20:29Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:20:29Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:20:29Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:20:29Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:20:29Z" level=info msg="New NodeID is db74ffaa-5541-4d7e-a78d-c44f709ecc97" 
time="2017-07-17T20:20:29Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:20:29Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:20:29Z" level=info msg="Node db74ffaa-5541-4d7e-a78d-c44f709ecc97 with Index (184) is Up" 
time="2017-07-17T20:20:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d5a0524e-b2a6-4d98-a0f7-f31bac8b3f78). Trying attachSet..." 
time="2017-07-17T20:20:47Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d7c87f0b-0833-4d22-a357-001ee84188fc). Trying attachSet..." 
time="2017-07-17T20:20:50Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 944f071b-d693-4b5a-9f55-8d09132194ec). Trying attachSet..." 
time="2017-07-17T20:20:53Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0a1c8c9f619baf16e" 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-00b0fb36e769745ad" 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0dc72eda0ec3c725a" 
time="2017-07-17T20:20:57Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

152
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:20:58,301 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:20:59,303 INFO spawned: 'pxdaemon' with pid 214
PXPROCS: Started px-storage with pid 218
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:00,696 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 239
PXPROCS: Started watchdog with pid 240
2017-07-17_20:21:02: PX-Watchdog: Starting watcher
2017-07-17_20:21:02: PX-Watchdog: Waiting for px process to start
root       239   214  0 20:21 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:02: PX-Watchdog: (pid 239): Begin monitoring
time="2017-07-17T20:21:02Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:21:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:21:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 112b36fa-e59e-44f9-83cc-9c76515e5bc7). Trying attachSet..." 
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:21:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:21:03Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:21:03Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:21:03Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:21:03Z" level=info msg="New NodeID is f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00" 
time="2017-07-17T20:21:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:21:03Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:21:03Z" level=info msg="Node f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00 with Index (185) is Up" 
time="2017-07-17T20:21:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d2d87553-9173-4c45-b459-6819b68ead08). Trying attachSet..." 
time="2017-07-17T20:21:17Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8633021a-561b-452c-aead-2e824e743193). Trying attachSet..." 
time="2017-07-17T20:21:21Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d53417a1-eb9f-42bc-8132-5161445eae3c). Trying attachSet..." 
time="2017-07-17T20:21:24Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
2017-07-17 20:21:26,693 INFO reaped unknown pid 200
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-00750f1049532859a" 
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-02f7cbe7bcbafa260" 
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-0d9c96794a8ddfd40" 
time="2017-07-17T20:21:27Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

218
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:21:28,702 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:21:29,705 INFO spawned: 'pxdaemon' with pid 323
PXPROCS: Started px-storage with pid 324
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:31,346 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 333
PXPROCS: Started watchdog with pid 334
2017-07-17_20:21:32: PX-Watchdog: Starting watcher
2017-07-17_20:21:32: PX-Watchdog: Waiting for px process to start
root       333   323  0 20:21 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:32: PX-Watchdog: (pid 333): Begin monitoring
time="2017-07-17T20:21:33Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:21:33Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:21:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4996b685-5ba7-4937-9c9d-aff07f2ba1a5). Trying attachSet..." 
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:21:34Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:21:34Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:21:34Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:21:34Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:21:34Z" level=info msg="New NodeID is e3f28af0-422e-4cb3-a4bf-877932e25ebb" 
time="2017-07-17T20:21:34Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:21:34Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:21:34Z" level=info msg="Node e3f28af0-422e-4cb3-a4bf-877932e25ebb with Index (186) is Up" 
time="2017-07-17T20:21:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 886f626e-2a8f-43a8-a048-6e5903792217). Trying attachSet..." 
time="2017-07-17T20:21:51Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:54Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6b193abd-1eb6-4179-af56-c2233c08b060). Trying attachSet..." 
time="2017-07-17T20:21:54Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e6841a93-07c8-483f-9439-91a51e1d6532). Trying attachSet..." 
time="2017-07-17T20:21:58Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-07bef9c07f4ca1bb7" 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0c484dcc58221cbee" 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0cae8c2f29ba14819" 
time="2017-07-17T20:22:01Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

324
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:22:02,681 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:03,683 INFO spawned: 'pxdaemon' with pid 384
PXPROCS: Started px-storage with pid 385
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:05,318 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:22:05,381 INFO reaped unknown pid 285
PXPROCS: Started px with pid 394
PXPROCS: Started watchdog with pid 395
2017-07-17_20:22:06: PX-Watchdog: Starting watcher
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid218-time1500322887
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:21:27.956992962 +0000 UTC.stack
2017-07-17_20:22:06: PX-Watchdog: Waiting for px process to start
root       394   384  0 20:22 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:06: PX-Watchdog: (pid 394): Begin monitoring
time="2017-07-17T20:22:06Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:22:06Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:22:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 04a1c231-6237-43e4-99f0-2a18b14e0409). Trying attachSet..." 
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:22:07Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:22:07Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:22:07Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:22:07Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:22:07Z" level=info msg="New NodeID is 1e53450a-d320-47f3-82e8-e3e38cfad2f6" 
time="2017-07-17T20:22:07Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:22:07Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:22:07Z" level=info msg="Node 1e53450a-d320-47f3-82e8-e3e38cfad2f6 with Index (187) is Up" 
Waiting for state transition to "attached".
time="2017-07-17T20:22:24Z" level=warning msg="Rollback attach volume /dev/xvdf, Error InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be" 
time="2017-07-17T20:22:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be). Trying attachSet..." 
time="2017-07-17T20:22:24Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:27Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 415096f6-2eb7-49f0-b255-be4807792aa2). Trying attachSet..." 
time="2017-07-17T20:22:28Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:31Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 026f0c84-b798-4a6e-884d-20cfbdc2e26e). Trying attachSet..." 
time="2017-07-17T20:22:31Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:34Z" level=info msg="Deleted volume vol-0754cdc54af687106" 
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-0e9f2af07873a2459" 
time="2017-07-17T20:22:35Z" level=warning msg="Delete failed for volume vol-03fb2827619da2903, Error VolumeInUse: Volume vol-03fb2827619da2903 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 034beb8e-1b61-459b-8c8b-efb169ff6e46" 
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-03fb2827619da2903" 
time="2017-07-17T20:22:35Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

385
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:22:35,799 INFO reaped unknown pid 376
2017-07-17 20:22:35,833 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:36,835 INFO spawned: 'pxdaemon' with pid 451
PXPROCS: Started px-storage with pid 452
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:38,470 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 461
PXPROCS: Started watchdog with pid 462
2017-07-17_20:22:39: PX-Watchdog: Starting watcher
2017-07-17_20:22:39: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid324-time1500322921
2017-07-17_20:22:40: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:01.687706644 +0000 UTC.stack
2017-07-17_20:22:40: PX-Watchdog: Waiting for px process to start
root       461   451 14 20:22 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:40: PX-Watchdog: (pid 461): Begin monitoring
time="2017-07-17T20:22:40Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:22:40Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:22:40Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 70ef01ce-5998-4e22-bb2c-6a42a851e9f4). Trying attachSet..." 
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:22:40Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:22:40Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:22:40Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:22:40Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:22:40Z" level=info msg="New NodeID is aaf41dec-5162-48e0-a638-2090a13836e3" 
time="2017-07-17T20:22:40Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:22:40Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:22:40Z" level=info msg="Node aaf41dec-5162-48e0-a638-2090a13836e3 with Index (188) is Up" 
Waiting for state transition to "attached".
time="2017-07-17T20:22:57Z" level=warning msg="Rollback attach volume /dev/xvdg, Error InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684" 
time="2017-07-17T20:22:57Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684). Trying attachSet..." 
time="2017-07-17T20:22:58Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:01Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: d6b02a83-9411-4750-8147-61c7cbd014f8). Trying attachSet..." 
time="2017-07-17T20:23:01Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:05Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: a894b483-2b4e-4fd5-beda-13f96b58ea91). Trying attachSet..." 
time="2017-07-17T20:23:05Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:08Z" level=info msg="Deleted volume vol-0c2263b7765a2b50b" 
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-05fea0bb3a91338e3" 
time="2017-07-17T20:23:09Z" level=warning msg="Delete failed for volume vol-039e4ec32142653e2, Error VolumeInUse: Volume vol-039e4ec32142653e2 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: af83fcb4-4f42-4532-b6a7-5dd3cd8e453b" 
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-039e4ec32142653e2" 
time="2017-07-17T20:23:09Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

452
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:23:09,908 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:09,931 INFO spawned: 'pxdaemon' with pid 518
2017-07-17 20:23:09,932 INFO reaped unknown pid 442
PXPROCS: Started px-storage with pid 519
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:11,565 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 528
PXPROCS: Started watchdog with pid 529
2017-07-17_20:23:12: PX-Watchdog: Starting watcher
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid385-time1500322955
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:35.164295909 +0000 UTC.stack
2017-07-17_20:23:13: PX-Watchdog: Waiting for px process to start
root       528   518  0 20:23 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:13: PX-Watchdog: (pid 528): Begin monitoring
time="2017-07-17T20:23:13Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:23:13Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:23:13Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc68d073-1e09-4809-8d86-3afafba43fc1). Trying attachSet..." 
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:23:13Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:23:13Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:23:13Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:23:13Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:23:13Z" level=info msg="New NodeID is 0d56d20b-4c84-4505-a52e-b7faa353e301" 
time="2017-07-17T20:23:13Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:23:13Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:23:13Z" level=info msg="Node 0d56d20b-4c84-4505-a52e-b7faa353e301 with Index (189) is Up" 
time="2017-07-17T20:23:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b982094-f18f-454e-8bfd-0e8c410140a9). Trying attachSet..." 
time="2017-07-17T20:23:31Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c15f63b7-e38a-4f33-a933-4b44914d3081). Trying attachSet..." 
time="2017-07-17T20:23:34Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 383087a1-63f7-40c6-903a-daaacf5ad088). Trying attachSet..." 
time="2017-07-17T20:23:38Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0e0355738c9daf989" 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-03ce75f45bd0bc5e6" 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0b4104817dc18bb25" 
time="2017-07-17T20:23:41Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

519
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:23:42,424 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:43,056 INFO spawned: 'pxdaemon' with pid 583
2017-07-17 20:23:43,057 INFO reaped unknown pid 509
PXPROCS: Started px-storage with pid 584
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:44,678 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 593
PXPROCS: Started watchdog with pid 594
2017-07-17_20:23:46: PX-Watchdog: Starting watcher
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid452-time1500322989
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:09.155729831 +0000 UTC.stack
2017-07-17_20:23:46: PX-Watchdog: Waiting for px process to start
root       593   583  0 20:23 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:46: PX-Watchdog: (pid 593): Begin monitoring
time="2017-07-17T20:23:46Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:23:46Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:23:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 205a8bb7-2ea4-42ad-8b07-c83138bcb455). Trying attachSet..." 
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:23:47Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:23:47Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:23:47Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:23:47Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:23:47Z" level=info msg="New NodeID is 43b0f079-e184-47fa-a57e-3ddcefacc686" 
time="2017-07-17T20:23:47Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:23:47Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:23:47Z" level=info msg="Node 43b0f079-e184-47fa-a57e-3ddcefacc686 with Index (190) is Up" 
time="2017-07-17T20:24:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d6383b44-e221-4625-88ae-0f4d675ce49e). Trying attachSet..." 
time="2017-07-17T20:24:01Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:04Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b7bf557a-6236-4aa3-9e8e-1aaecf7be7cc). Trying attachSet..." 
time="2017-07-17T20:24:04Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff8f6a0-5333-4023-894e-cd9541c4cc6d). Trying attachSet..." 
time="2017-07-17T20:24:07Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-0602b317886ec4938" 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-021ace0cd6005e01d" 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-073cc4cd01ab0067a" 
time="2017-07-17T20:24:11Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

584
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:24:12,374 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:13,377 INFO spawned: 'pxdaemon' with pid 647
PXPROCS: Started px-storage with pid 648
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:15,018 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:24:16,116 INFO reaped unknown pid 576
PXPROCS: Started px with pid 657
PXPROCS: Started watchdog with pid 658
2017-07-17_20:24:16: PX-Watchdog: Starting watcher
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid519-time1500323021
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:41.620964623 +0000 UTC.stack
2017-07-17_20:24:16: PX-Watchdog: Waiting for px process to start
root       657   647  0 20:24 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:16: PX-Watchdog: (pid 657): Begin monitoring
time="2017-07-17T20:24:16Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:24:16Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:24:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff2462f-adad-4d28-a4df-7ebc3abad72f). Trying attachSet..." 
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:24:17Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:24:17Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:24:17Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:24:17Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:24:17Z" level=info msg="New NodeID is 80a52d5c-9f0b-4d91-a55d-0100a0d21543" 
time="2017-07-17T20:24:17Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:24:17Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:24:17Z" level=info msg="Node 80a52d5c-9f0b-4d91-a55d-0100a0d21543 with Index (191) is Up" 
time="2017-07-17T20:24:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d18194c7-c97c-4653-b74f-c4c327ddf09a). Trying attachSet..." 
time="2017-07-17T20:24:28Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:31Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bc0b728a-09ef-4cac-b2e8-693c940c9969). Trying attachSet..." 
time="2017-07-17T20:24:31Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d926c609-e296-47d4-9b71-ef1a3badae7f). Trying attachSet..." 
time="2017-07-17T20:24:35Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-041b6e029a322b566" 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-053f068cbfcb41ee8" 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-047bb73b87e28b7a4" 
time="2017-07-17T20:24:38Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

648
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:24:39,561 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:40,563 INFO spawned: 'pxdaemon' with pid 715
PXPROCS: Started px-storage with pid 716
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:42,172 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 725
PXPROCS: Started watchdog with pid 726
2017-07-17_20:24:43: PX-Watchdog: Starting watcher
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid584-time1500323051
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:11.35037889 +0000 UTC.stack
2017-07-17_20:24:43: PX-Watchdog: Waiting for px process to start
root       725   715  0 20:24 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:43: PX-Watchdog: (pid 725): Begin monitoring
time="2017-07-17T20:24:43Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:24:43Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:24:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 05156c82-e040-4670-977b-8ac6d28b5b2f). Trying attachSet..." 
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:24:44Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:24:44Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:24:44Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:24:44Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:24:44Z" level=info msg="New NodeID is e7a373e2-8745-4a34-a0b7-5d874ab16358" 
time="2017-07-17T20:24:44Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:24:44Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:24:44Z" level=info msg="Node e7a373e2-8745-4a34-a0b7-5d874ab16358 with Index (192) is Up" 
2017-07-17 20:24:49,228 INFO reaped unknown pid 641
time="2017-07-17T20:24:55Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6cdef988-ff36-4089-91de-0a7d31d36a06). Trying attachSet..." 
time="2017-07-17T20:24:55Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:24:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6fbd4955-6ead-4297-9394-8023998e90a1). Trying attachSet..." 
time="2017-07-17T20:24:58Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:25:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6799674c-d090-4f39-b2d4-8d4f26a89424). Trying attachSet..." 
time="2017-07-17T20:25:02Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0657b54d2782a7bc6" 
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0bb70306e52984a56" 
time="2017-07-17T20:25:06Z" level=info msg="Deleted volume vol-0d703b168c35f282d" 
time="2017-07-17T20:25:06Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

716
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:25:06,799 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:07,802 INFO spawned: 'pxdaemon' with pid 779
PXPROCS: Started px-storage with pid 780
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:09,410 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 789
PXPROCS: Started watchdog with pid 790
2017-07-17_20:25:10: PX-Watchdog: Starting watcher
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid648-time1500323078
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:38.685498093 +0000 UTC.stack
2017-07-17_20:25:10: PX-Watchdog: Waiting for px process to start
root       789   779  8 20:25 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:10: PX-Watchdog: (pid 789): Begin monitoring
time="2017-07-17T20:25:11Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:25:11Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:25:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4b606935-7d8b-45ed-b606-404ad255af68). Trying attachSet..." 
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:25:11Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:25:11Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:25:11Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:25:11Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:25:11Z" level=info msg="New NodeID is 10777ebd-50b0-4fa5-be24-fe3f1853350c" 
time="2017-07-17T20:25:11Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:25:11Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:25:11Z" level=info msg="Node 10777ebd-50b0-4fa5-be24-fe3f1853350c with Index (193) is Up" 
2017-07-17 20:25:19,526 INFO reaped unknown pid 705
time="2017-07-17T20:25:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 03968f47-0d51-4711-b4bc-299d89bceadb). Trying attachSet..." 
time="2017-07-17T20:25:22Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:25Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c8964c28-6a18-44ad-9d9e-8a981987690a). Trying attachSet..." 
time="2017-07-17T20:25:26Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 33939c5f-7e61-4c5a-9efd-adaa5fb93181). Trying attachSet..." 
time="2017-07-17T20:25:29Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0bc3d3459e618fd8b" 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0ccd3d4f8966fd354" 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0f101dfd08e86c2c5" 
time="2017-07-17T20:25:32Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

780
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:25:33,524 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:34,527 INFO spawned: 'pxdaemon' with pid 844
PXPROCS: Started px-storage with pid 845
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:36,158 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 854
PXPROCS: Started watchdog with pid 855
2017-07-17_20:25:37: PX-Watchdog: Starting watcher
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid716-time1500323106
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:06.079558835 +0000 UTC.stack
2017-07-17_20:25:37: PX-Watchdog: Waiting for px process to start
root       854   844  0 20:25 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:37: PX-Watchdog: (pid 854): Begin monitoring
time="2017-07-17T20:25:37Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:25:37Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:25:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 49eb1ddd-35e3-423c-a54d-14821585fa39). Trying attachSet..." 
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:25:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:25:38Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:25:38Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:25:38Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:25:38Z" level=info msg="New NodeID is 1cb08f7e-ec86-41de-a2f9-6adc19314511" 
time="2017-07-17T20:25:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:25:38Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:25:38Z" level=info msg="Node 1cb08f7e-ec86-41de-a2f9-6adc19314511 with Index (194) is Up" 
2017-07-17 20:25:46,728 INFO reaped unknown pid 773
time="2017-07-17T20:25:52Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4130e767-414f-4566-8a7c-04ad1f787ffb). Trying attachSet..." 
time="2017-07-17T20:25:52Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:25:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc67c369-235e-4463-aab4-f8ec7cd0d290). Trying attachSet..." 
time="2017-07-17T20:25:56Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:25:59Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 7d7e1823-c775-44b8-ba7f-e9ff9df45a9f). Trying attachSet..." 
time="2017-07-17T20:25:59Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-01a5dd80d92e4d7b9" 
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-0b4b020d6fa2b965f" 
time="2017-07-17T20:26:03Z" level=info msg="Deleted volume vol-080f3e66e286102cb" 
time="2017-07-17T20:26:03Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

845
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:26:03,930 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:04,933 INFO spawned: 'pxdaemon' with pid 910
PXPROCS: Started px-storage with pid 911
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:06,583 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 920
PXPROCS: Started watchdog with pid 921
2017-07-17_20:26:07: PX-Watchdog: Starting watcher
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid780-time1500323132
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:32.845892519 +0000 UTC.stack
2017-07-17_20:26:08: PX-Watchdog: Waiting for px process to start
root       920   910  0 20:26 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:26:08: PX-Watchdog: (pid 920): Begin monitoring
time="2017-07-17T20:26:08Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:26:08Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:26:08Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8da904b6-3771-4a9a-a141-50377f617384). Trying attachSet..." 
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:26:08Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:26:08Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:26:09Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:26:09Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:26:09Z" level=info msg="New NodeID is f081adb9-2e28-4eae-819c-cac8a914ec5a" 
time="2017-07-17T20:26:09Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:26:09Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:26:09Z" level=info msg="Node f081adb9-2e28-4eae-819c-cac8a914ec5a with Index (195) is Up" 
2017-07-17 20:26:13,991 INFO reaped unknown pid 838
Waiting for state transition to "attached".
time="2017-07-17T20:26:22Z" level=warning msg="Rollback attach volume /dev/xvdh, Error InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571" 
time="2017-07-17T20:26:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571). Trying attachSet..." 
time="2017-07-17T20:26:22Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:26Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: f4c9ed99-6e10-4b8c-8816-c9bc7ce24df0). Trying attachSet..." 
time="2017-07-17T20:26:26Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:29Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: 019de502-07d1-4eb9-916a-17447ee32a1b). Trying attachSet..." 
time="2017-07-17T20:26:29Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-050a22aa7316ebf6c" 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-0eb0a3b0133732862" 
time="2017-07-17T20:26:33Z" level=warning msg="Delete failed for volume vol-01005566936b4286a, Error VolumeInUse: Volume vol-01005566936b4286a is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: c3786596-2d41-4cfe-b653-2907144b0cd9" 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-01005566936b4286a" 
time="2017-07-17T20:26:33Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

911
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:26:34,013 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:35,017 INFO spawned: 'pxdaemon' with pid 978
PXPROCS: Started px-storage with pid 979
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:36,651 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 988
PXPROCS: Started watchdog with pid 989
2017-07-17_20:26:38: PX-Watchdog: Starting watcher
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid845-time1500323163
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:03.08213779 +0000 UTC.stack
2017-07-17_20:26:38: PX-Watchdog: Waiting for px process to start
root       988   978  0 20:26 ?        00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:26:38Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:26:38Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:26:38: PX-Watchdog: (pid 988): Begin monitoring
time="2017-07-17T20:26:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 042f48e1-4a74-4282-b62c-120f0b505275). Trying attachSet..." 
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:26:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:26:38Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:26:38Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:26:38Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:26:38Z" level=info msg="New NodeID is 7fa4353d-bdb0-407e-ad50-61009a93446e" 
time="2017-07-17T20:26:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:26:38Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:26:38Z" level=info msg="Node 7fa4353d-bdb0-407e-ad50-61009a93446e with Index (196) is Up" 
2017-07-17 20:26:40,727 INFO reaped unknown pid 902
time="2017-07-17T20:26:49Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 249eedb9-03d6-47bc-90e5-bac4a438d78d). Trying attachSet..." 
time="2017-07-17T20:26:50Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6732b0b1-4591-4653-b9a7-196cd8427385). Trying attachSet..." 
time="2017-07-17T20:26:53Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 010aa708-329c-4477-91a7-309b61759a88). Trying attachSet..." 
time="2017-07-17T20:26:56Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-099d813e8fd60f787" 
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-0ebc07b08c4cde4d3" 
time="2017-07-17T20:27:00Z" level=info msg="Deleted volume vol-0ee606659f735d922" 
time="2017-07-17T20:27:00Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

979
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:01,063 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:02,066 INFO spawned: 'pxdaemon' with pid 1044
PXPROCS: Started px-storage with pid 1045
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:03,694 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1054
PXPROCS: Started watchdog with pid 1055
2017-07-17_20:27:05: PX-Watchdog: Starting watcher
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid911-time1500323193
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:33.397134213 +0000 UTC.stack
2017-07-17_20:27:05: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:05Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:27:05Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1054  1044  0 20:27 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:05: PX-Watchdog: (pid 1054): Begin monitoring
time="2017-07-17T20:27:05Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dafee0bb-631e-485b-8a8b-ba2b859a38a8). Trying attachSet..." 
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:27:06Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:27:06Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:27:06Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:27:06Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:27:06Z" level=info msg="New NodeID is 20d763d5-b44b-40fc-83f3-9976aa6090d6" 
time="2017-07-17T20:27:06Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:27:06Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:27:06Z" level=info msg="Node 20d763d5-b44b-40fc-83f3-9976aa6090d6 with Index (197) is Up" 
2017-07-17 20:27:11,081 INFO reaped unknown pid 968
time="2017-07-17T20:27:19Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 95a12bef-eb02-4fd1-b2d9-b457b5f38350). Trying attachSet..." 
time="2017-07-17T20:27:20Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:23Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2f0b7dfc-30f9-4d2a-a3cc-3155377b78b3). Trying attachSet..." 
time="2017-07-17T20:27:23Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 9c1b5212-1047-4015-a9b4-717691e7f4a7). Trying attachSet..." 
time="2017-07-17T20:27:26Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-011bc83c090da3408" 
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-007d1c99f1baf09ab" 
time="2017-07-17T20:27:30Z" level=info msg="Deleted volume vol-0ba1e6a53ba00a031" 
time="2017-07-17T20:27:30Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1045
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:30,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:31,816 INFO spawned: 'pxdaemon' with pid 1110
PXPROCS: Started px-storage with pid 1111
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:33,430 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1120
PXPROCS: Started watchdog with pid 1121
2017-07-17_20:27:34: PX-Watchdog: Starting watcher
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid979-time1500323220
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:00.237474507 +0000 UTC.stack
2017-07-17_20:27:35: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:35Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:27:35Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1120  1110 13 20:27 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:35: PX-Watchdog: (pid 1120): Begin monitoring
time="2017-07-17T20:27:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a8c4a83c-cc21-4111-a26f-488d8ecdecb5). Trying attachSet..." 
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:27:35Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:27:35Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:27:35Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:27:35Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:27:35Z" level=info msg="New NodeID is af5717a7-6264-4758-995b-e72ec7f915cc" 
time="2017-07-17T20:27:35Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:27:35Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:27:35Z" level=info msg="Node af5717a7-6264-4758-995b-e72ec7f915cc with Index (198) is Up" 
2017-07-17 20:27:41,249 INFO reaped unknown pid 1039
time="2017-07-17T20:27:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172e05e6-8748-4f0b-8011-5d1477c0441a). Trying attachSet..." 
time="2017-07-17T20:27:46Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f2f608f7-c284-44df-b4d5-e50a71f9a46a). Trying attachSet..." 
time="2017-07-17T20:27:50Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: af94b5af-4b5b-4007-a3c4-21cbe37b7f01). Trying attachSet..." 
time="2017-07-17T20:27:53Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0689dea182e8ae258" 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0d2f6ebed91295e99" 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0a29d43b2e71318d8" 
time="2017-07-17T20:27:56Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1111
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:57,701 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:58,703 INFO spawned: 'pxdaemon' with pid 1176
PXPROCS: Started px-storage with pid 1177
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:00,347 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1186
PXPROCS: Started watchdog with pid 1187
2017-07-17_20:28:01: PX-Watchdog: Starting watcher
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1045-time1500323250
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:30.154572756 +0000 UTC.stack
2017-07-17_20:28:01: PX-Watchdog: Waiting for px process to start
root      1186  1176  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:01: PX-Watchdog: (pid 1186): Begin monitoring
time="2017-07-17T20:28:01Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:01Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:28:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f0c15705-922f-494a-89cd-84a4a543122d). Trying attachSet..." 
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:02Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:02Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:02Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:02Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:02Z" level=info msg="New NodeID is 45bb1f10-c864-4d1e-85d8-2164533074ed" 
time="2017-07-17T20:28:02Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:02Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:02Z" level=info msg="Node 45bb1f10-c864-4d1e-85d8-2164533074ed with Index (199) is Up" 
2017-07-17 20:28:08,335 INFO reaped unknown pid 1104
Waiting for state transition to "attached".
time="2017-07-17T20:28:16Z" level=warning msg="Rollback attach volume /dev/xvdi, Error InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9" 
time="2017-07-17T20:28:16Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9). Trying attachSet..." 
time="2017-07-17T20:28:16Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:20Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: b07d643c-5963-4675-94c3-55a77a0ce75c). Trying attachSet..." 
time="2017-07-17T20:28:20Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:23Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: af301d66-945c-44c1-a39c-0e60c957db56). Trying attachSet..." 
time="2017-07-17T20:28:24Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-0bed0efe74e5c82e5" 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-037a2b5f2a17e29a3" 
time="2017-07-17T20:28:27Z" level=warning msg="Delete failed for volume vol-01d3841fc605f322f, Error VolumeInUse: Volume vol-01d3841fc605f322f is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: f7e4a11b-8f8e-4414-b052-9c2b48e6bce2" 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-01d3841fc605f322f" 
time="2017-07-17T20:28:27Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1177
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:28:27,492 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:28,495 INFO spawned: 'pxdaemon' with pid 1245
PXPROCS: Started px-storage with pid 1246
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:30,130 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1255
PXPROCS: Started watchdog with pid 1256
2017-07-17_20:28:31: PX-Watchdog: Starting watcher
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1111-time1500323276
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:56.973406987 +0000 UTC.stack
2017-07-17_20:28:31: PX-Watchdog: Waiting for px process to start
root      1255  1245  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:31: PX-Watchdog: (pid 1255): Begin monitoring
time="2017-07-17T20:28:31Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:31Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:28:32Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 63e35a20-78cf-4f60-af8f-9124245cd108). Trying attachSet..." 
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:32Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:32Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:32Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:32Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:32Z" level=info msg="New NodeID is 77042558-cabb-4d05-877b-d2105e95088b" 
time="2017-07-17T20:28:32Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:32Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:32Z" level=info msg="Node 77042558-cabb-4d05-877b-d2105e95088b with Index (200) is Up" 
2017-07-17 20:28:38,064 INFO reaped unknown pid 1170
time="2017-07-17T20:28:43Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 307736dd-d7fc-462b-b35c-ff431f41c9e3). Trying attachSet..." 
time="2017-07-17T20:28:43Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 176322cd-3053-4c15-bd87-e4fe4f930de0). Trying attachSet..." 
time="2017-07-17T20:28:47Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b20869bc-923c-4a0d-9812-2e56131c407f). Trying attachSet..." 
time="2017-07-17T20:28:50Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07bd9c31eee1a5369" 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-04a8c069f6593c478" 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07b74ab9dd4c1516e" 
time="2017-07-17T20:28:53Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1246
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:28:53,992 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:54,995 INFO spawned: 'pxdaemon' with pid 1311
PXPROCS: Started px-storage with pid 1312
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:56,611 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1321
PXPROCS: Started watchdog with pid 1322
2017-07-17_20:28:58: PX-Watchdog: Starting watcher
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1177-time1500323307
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:27.364753832 +0000 UTC.stack
2017-07-17_20:28:58: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:28:58Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:58Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1321  1311  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:58: PX-Watchdog: (pid 1321): Begin monitoring
time="2017-07-17T20:28:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ea863fee-bd89-4fbb-b64c-418dd0c8e652). Trying attachSet..." 
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:58Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:58Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:58Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:58Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:58Z" level=info msg="New NodeID is bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f" 
time="2017-07-17T20:28:58Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:58Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:58Z" level=info msg="Node bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f with Index (201) is Up" 
2017-07-17 20:29:04,856 INFO reaped unknown pid 1234
Waiting for state transition to "attached".
time="2017-07-17T20:29:12Z" level=warning msg="Rollback attach volume /dev/xvdj, Error InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b" 
time="2017-07-17T20:29:12Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b). Trying attachSet..." 
time="2017-07-17T20:29:12Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:15Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: 5d08c26e-cd8b-4382-a14f-63e176371ea8). Trying attachSet..." 
time="2017-07-17T20:29:16Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:19Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: edb3fe8e-78ce-41f6-9482-b4f677b8a631). Trying attachSet..." 
time="2017-07-17T20:29:19Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:22Z" level=info msg="Deleted volume vol-0e600499b93e24abe" 
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-01da48b0445fed392" 
time="2017-07-17T20:29:23Z" level=warning msg="Delete failed for volume vol-00aa97e0ca04b783e, Error VolumeInUse: Volume vol-00aa97e0ca04b783e is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 8b5a2444-114e-4cec-a746-a44d57e0d8bd" 
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-00aa97e0ca04b783e" 
time="2017-07-17T20:29:23Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1312
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:29:23,313 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:24,315 INFO spawned: 'pxdaemon' with pid 1376
PXPROCS: Started px-storage with pid 1377
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:29:25,971 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1386
PXPROCS: Started watchdog with pid 1387
2017-07-17_20:29:27: PX-Watchdog: Starting watcher
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1246-time1500323333
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:53.69988716 +0000 UTC.stack
2017-07-17_20:29:27: PX-Watchdog: Waiting for px process to start
root      1386  1376  0 20:29 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:29:27: PX-Watchdog: (pid 1386): Begin monitoring
time="2017-07-17T20:29:27Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:29:27Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:29:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8c76aa23-2d41-47f5-a37f-6184fa1eee55). Trying attachSet..." 
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:29:28Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:29:28Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:29:28Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:29:28Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:29:28Z" level=info msg="New NodeID is d0fc3fea-3470-4ed2-9b9e-bb5d41443015" 
time="2017-07-17T20:29:28Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:29:28Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:29:28Z" level=info msg="Node d0fc3fea-3470-4ed2-9b9e-bb5d41443015 with Index (202) is Up" 
2017-07-17 20:29:34,668 INFO reaped unknown pid 1304
Waiting for state transition to "attached".
time="2017-07-17T20:29:47Z" level=warning msg="Rollback attach volume /dev/xvdk, Error InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e" 
time="2017-07-17T20:29:47Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e). Trying attachSet..." 
time="2017-07-17T20:29:47Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:51Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: 8f446eb2-bca8-408f-9d7b-90554efe480f). Trying attachSet..." 
time="2017-07-17T20:29:51Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:54Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: ccaf35d4-c687-485f-bac3-725f5690b3d7). Trying attachSet..." 
time="2017-07-17T20:29:55Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-09ebb76decd2a21a5" 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-0c275e24e94030709" 
time="2017-07-17T20:29:58Z" level=warning msg="Delete failed for volume vol-00a1007e4a44e2349, Error VolumeInUse: Volume vol-00a1007e4a44e2349 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 3d4a0160-ad3e-41c6-adbd-d2d7bfa061b8" 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-00a1007e4a44e2349" 
time="2017-07-17T20:29:58Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1377
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:29:58,749 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:59,752 INFO spawned: 'pxdaemon' with pid 1443
PXPROCS: Started px-storage with pid 1444
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:01,240 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:30:01,241 INFO reaped unknown pid 1369
PXPROCS: Started px with pid 1453
PXPROCS: Started watchdog with pid 1454
2017-07-17_20:30:02: PX-Watchdog: Starting watcher
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1312-time1500323363
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:23.190405658 +0000 UTC.stack
2017-07-17_20:30:02: PX-Watchdog: Waiting for px process to start
root      1453  1443  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:02: PX-Watchdog: (pid 1453): Begin monitoring
time="2017-07-17T20:30:02Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:30:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ba5cfc3-15cb-4f96-90ba-fa2205c7b19c). Trying attachSet..." 
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:30:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:30:03Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:30:03Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:30:03Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:30:03Z" level=info msg="New NodeID is a0704554-ad8d-4e45-9233-0e5788efa64d" 
time="2017-07-17T20:30:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:30:03Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:30:03Z" level=info msg="Node a0704554-ad8d-4e45-9233-0e5788efa64d with Index (203) is Up" 
time="2017-07-17T20:30:15Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5a88123a-1ebb-4c14-a4ac-b7af49a13e14). Trying attachSet..." 
time="2017-07-17T20:30:15Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:18Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ca24dbfa-d0ed-4733-b4a4-d7ece3d88138). Trying attachSet..." 
time="2017-07-17T20:30:18Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4cc59be9-e927-4949-a7ce-2d0cbc25bfe5). Trying attachSet..." 
time="2017-07-17T20:30:22Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-002226758ec436a2b" 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0c0c0d1b5f73d207c" 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0fddd4790e207a76e" 
time="2017-07-17T20:30:25Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1444
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:30:25,593 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:26,596 INFO spawned: 'pxdaemon' with pid 1510
PXPROCS: Started px-storage with pid 1511
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:28,238 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1520
PXPROCS: Started watchdog with pid 1521
2017-07-17_20:30:29: PX-Watchdog: Starting watcher
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1377-time1500323398
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:58.533024037 +0000 UTC.stack
2017-07-17_20:30:29: PX-Watchdog: Waiting for px process to start
root      1520  1510  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:30:29Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:29Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:30:29: PX-Watchdog: (pid 1520): Begin monitoring
2017-07-17 20:30:30,468 INFO reaped unknown pid 1434
time="2017-07-17T20:30:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2d90acf5-d8ed-417f-bdcd-02f668f7e8d2). Trying attachSet..." 
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:30:30Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:30:30Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:30:30Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:30:30Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:30:30Z" level=info msg="New NodeID is 1a26f325-6f17-4e8e-bf7e-3891de932909" 
time="2017-07-17T20:30:30Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:30:30Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:30:30Z" level=info msg="Node 1a26f325-6f17-4e8e-bf7e-3891de932909 with Index (204) is Up" 
time="2017-07-17T20:30:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc8301d9-4931-4f69-8b17-97fa2468ace8). Trying attachSet..." 
time="2017-07-17T20:30:45Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:48Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 45effe32-fe2f-4d32-bfb2-f5d1c70a055e). Trying attachSet..." 
time="2017-07-17T20:30:48Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0fa50a78-b1e6-4694-a420-1248e013a0a0). Trying attachSet..." 
time="2017-07-17T20:30:51Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-03437c1d61237e06d" 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-070c06c66a4edf5e3" 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-0def01de3c342c0b2" 
time="2017-07-17T20:30:54Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1511
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:30:55,281 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:56,283 INFO spawned: 'pxdaemon' with pid 1575
PXPROCS: Started px-storage with pid 1576
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:57,914 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1585
PXPROCS: Started watchdog with pid 1586
2017-07-17_20:30:59: PX-Watchdog: Starting watcher
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1444-time1500323425
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:25.457395001 +0000 UTC.stack
2017-07-17_20:30:59: PX-Watchdog: Waiting for px process to start
root      1585  1575  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:59: PX-Watchdog: (pid 1585): Begin monitoring
time="2017-07-17T20:30:59Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:59Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:31:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0690e408-cb5e-4c52-873d-542b796400eb). Trying attachSet..." 
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:00Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:00Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:00Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:00Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:00Z" level=info msg="New NodeID is a62882c3-5db6-47dd-aa20-e5036752dd4d" 
time="2017-07-17T20:31:00Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:00Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:00Z" level=info msg="Node a62882c3-5db6-47dd-aa20-e5036752dd4d with Index (205) is Up" 
2017-07-17 20:31:05,929 INFO reaped unknown pid 1501
time="2017-07-17T20:31:10Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b40dbeb9-7e09-4119-8d50-5335145f5edf). Trying attachSet..." 
time="2017-07-17T20:31:11Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172f4a11-653c-4ce1-b2b7-79269b9fd632). Trying attachSet..." 
time="2017-07-17T20:31:14Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c5e717fb-bb32-41bb-be69-326462c6e13c). Trying attachSet..." 
time="2017-07-17T20:31:18Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0aafb49c766743134" 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0ff134bfb92c759c0" 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0b1d81a6aeab63d89" 
time="2017-07-17T20:31:21Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1576
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:31:21,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:22,815 INFO spawned: 'pxdaemon' with pid 1640
PXPROCS: Started px-storage with pid 1641
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:24,450 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1650
PXPROCS: Started watchdog with pid 1651
2017-07-17_20:31:25: PX-Watchdog: Starting watcher
2017-07-17_20:31:25: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1511-time1500323454
2017-07-17_20:31:26: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:54.91891449 +0000 UTC.stack
time="2017-07-17T20:31:26Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:31:26Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:31:26: PX-Watchdog: Waiting for px process to start
root      1650  1640 15 20:31 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:26: PX-Watchdog: (pid 1650): Begin monitoring
time="2017-07-17T20:31:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5259b7a5-046e-4165-82be-7f1424061637). Trying attachSet..." 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:26Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:26Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:26Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:26Z" level=info msg="New NodeID is 5e7942f5-cc34-41b3-a383-2002e46dc4c3" 
time="2017-07-17T20:31:26Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:26Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:26Z" level=info msg="Node 5e7942f5-cc34-41b3-a383-2002e46dc4c3 with Index (206) is Up" 
2017-07-17 20:31:32,844 INFO reaped unknown pid 1569
time="2017-07-17T20:31:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e82c913d-2f17-429c-87dd-ea7471f7fa3f). Trying attachSet..." 
time="2017-07-17T20:31:38Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:41Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2525bc39-3e6c-4bd1-8da6-8990ed2e197d). Trying attachSet..." 
time="2017-07-17T20:31:41Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a913b333-7d5e-47bd-b034-4649afc28576). Trying attachSet..." 
time="2017-07-17T20:31:44Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0f4102b45575f73e8" 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-048e6ebce8493ec84" 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0656f33ba1a241e3d" 
time="2017-07-17T20:31:48Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1641
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:31:48,861 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:49,863 INFO spawned: 'pxdaemon' with pid 1706
PXPROCS: Started px-storage with pid 1707
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:51,498 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1716
PXPROCS: Started watchdog with pid 1717
2017-07-17_20:31:52: PX-Watchdog: Starting watcher
2017-07-17_20:31:52: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1576-time1500323481
2017-07-17_20:31:53: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:31:21.498835997 +0000 UTC.stack
2017-07-17_20:31:53: PX-Watchdog: Waiting for px process to start
root      1716  1706 10 20:31 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:53: PX-Watchdog: (pid 1716): Begin monitoring
time="2017-07-17T20:31:53Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:31:53Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:31:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fce46d5f-3bf4-4681-a1cb-84576b1d5916). Trying attachSet..." 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:53Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:54Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:54Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:54Z" level=info msg="New NodeID is a2832162-c916-452b-a9ec-056af976a982" 
time="2017-07-17T20:31:54Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:54Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:54Z" level=info msg="Node a2832162-c916-452b-a9ec-056af976a982 with Index (207) is Up" 
2017-07-17 20:32:02,471 INFO reaped unknown pid 1634
time="2017-07-17T20:32:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dc7130f6-9f18-41d7-bbfe-c869ef99da05). Trying attachSet..." 
time="2017-07-17T20:32:08Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..." 
time="2017-07-17T20:32:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6708a9f6-2fc0-4cc4-9311-7b988419744a). Trying attachSet..." 
time="2017-07-17T20:32:11Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..." 
time="2017-07-17T20:32:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 86997d13-17d6-4fec-9c23-b6dea2d4248d). Trying attachSet..." 
time="2017-07-17T20:32:14Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..."

Hi @jvinod . The problem still persists if I configure portworx with the correct volume. The issue is ocurring with portworx enterprise too. Here are some details of my environment:

I've 3 docker swarm manager nodes running on each AZ from sa-east-1 region. Each of them have portworx installed (as a docker plugin) and working very well.
Each AZ has an EBS volume with 30GB size that is used as template for portworx. As you coul see bellow, they are attached to each 3 manager nodes:

This is my pxctl status, executed from a manager node:

Status: PX is operational
Node ID: 25ca5091-4a29-4f1b-ad21-b94338be313b
    IP: 10.0.2.102 
    Local Storage Pool: 1 pool
    POOL    IO_PRIORITY RAID_LEVEL  USABLE  USED    STATUS  ZONE    REGION
    0   LOW     raid0       30 GiB  14 GiB  Online  b   sa-east-1
    Local Storage Devices: 1 device
    Device  Path        Media Type      Size        Last-Scan
    0:1 /dev/xvdg   STORAGE_MEDIUM_MAGNETIC 30 GiB      17 Jul 17 16:18 UTC
    total           -           30 GiB
Cluster Summary
    Cluster ID: socialbase
    IP      ID                  Used    Capacity    Status
    10.0.2.102  25ca5091-4a29-4f1b-ad21-b94338be313b    14 GiB  30 GiB      Online (This node)
    10.0.1.101  7a943a7e-b9b9-43a2-b6c0-8014934672fa    14 GiB  30 GiB      Online
    10.0.3.103  c130fbfd-4549-4a46-a4ef-771f82d4395c    14 GiB  30 GiB      Online
Global Storage Pool
    Total Used      :  42 GiB
    Total Capacity  :  90 GiB

After built the above scenario, I decided to add docker swarm worker nodes in the cluster. So, I've tried to build an ami with portworx running as a container (instead of docker plugin), but the error showed here ocurred... After reporting this, I've tried to make the same ami with portworx docker plugin and with px-enterprise docker container. None of them worked with the same error.
My last attempt was to use px-enterprise as a docker container setting up the parameters like you pointed. But, not worked too. Here are the container logs and info, instance and ami details:

root@ip-10-0-1-9:~# docker inspect portworx
[
    {
        "Id": "efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879",
        "Created": "2017-07-18T10:50:46.063458381Z",
        "Path": "/docker-entry-point.sh",
        "Args": [
            "-c",
            "socialbase",
            "-k",
            "consul:http://swarmmanager.sb:8500",
            "-s",
            "vol-0842059ddbdbd9d7f"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-07-18T10:50:48.254479076Z",
            "FinishedAt": "2017-07-18T10:58:10.923496743Z"
        },
        "Image": "sha256:ca9551f7c0c39dadfc66e02ec46e542d442975f3c69b937ed3b27ac05861ea07",
        "ResolvConfPath": "/var/lib/docker/containers/efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879/hostname",
        "HostsPath": "/var/lib/docker/containers/efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879/hosts",
        "LogPath": "/var/lib/docker/containers/efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879/efc155a39a367e5f70d854cc0b33673c33c3b5b7c824220fb9b7ed2d44af4879-json.log",
        "Name": "/portworx",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "unconfined",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/dev:/dev",
                "/etc/pwx:/etc/pwx",
                "/opt/pwx/bin:/export_bin:shared",
                "/var/run/docker.sock:/var/run/docker.sock",
                "/var/cores:/var/cores",
                "/usr/src:/usr/src",
                "/run/docker/plugins:/run/docker/plugins",
                "/var/lib/osd:/var/lib/osd:shared"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/652c9affa013137d8a6a1337621f1598295591a912480ff581120200110b4440-init/diff:/var/lib/docker/overlay2/816f11c26bf35deb44fa8ba5ef28da9b2003cf6f0899616952b5fadd907e3cb7/diff:/var/lib/docker/overlay2/2df992b856da08483147a98ea2a1d84ccad966af43eb7eb5d268748485116ff7/diff:/var/lib/docker/overlay2/4b19d59eac40c26af70e9b83776b3378a1fbff15f3e24652b5f9dc8277b97101/diff:/var/lib/docker/overlay2/2a79aa3e856c37f90c72bc9f1391200050c476ecd45cea6cca075759cf9821f4/diff:/var/lib/docker/overlay2/a9b129c87d2b0beb769123a69cfc59ec108d5396b59a9f42ad89a79435aea17e/diff:/var/lib/docker/overlay2/10be848ebba274f65ae8a26ef236b6fe25743a4c56d56c64c25b494ab3b7cc60/diff:/var/lib/docker/overlay2/61c514a57b9c9ec1abf4aecf08837a59515a65d5f0cffc93a73754485c2bce99/diff",
                "MergedDir": "/var/lib/docker/overlay2/652c9affa013137d8a6a1337621f1598295591a912480ff581120200110b4440/merged",
                "UpperDir": "/var/lib/docker/overlay2/652c9affa013137d8a6a1337621f1598295591a912480ff581120200110b4440/diff",
                "WorkDir": "/var/lib/docker/overlay2/652c9affa013137d8a6a1337621f1598295591a912480ff581120200110b4440/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/opt/pwx/bin",
                "Destination": "/export_bin",
                "Mode": "shared",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/cores",
                "Destination": "/var/cores",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/usr/src",
                "Destination": "/usr/src",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/run/docker/plugins",
                "Destination": "/run/docker/plugins",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/osd",
                "Destination": "/var/lib/osd",
                "Mode": "shared",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/dev",
                "Destination": "/dev",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/pwx",
                "Destination": "/etc/pwx",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ip-10-0-1-9",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive",
                "container=docker",
                "LD_LIBRARY_PATH=/usr/local/lib64:/usr/lib64/:/usr/lib",
                "GOTRACEBACK=crash",
                "GOMAXPROCS=64",
                "PXMOD_SOURCE=/home/px-fuse",
                "PXMOD_VERSION=2"
            ],
            "Cmd": [
                "-c",
                "socialbase",
                "-k",
                "consul:http://swarmmanager.sb:8500",
                "-s",
                "vol-0842059ddbdbd9d7f"
            ],
            "Image": "portworx/px-enterprise",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/docker-entry-point.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "bff969e4864b30ccfed2fbd482c2bff74a9c54019e9a9f03adb87055352cab6c",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "398c38548ba4d94e5cc226c1340bce8c82400ec9ff2ec562e76b4d5cee61697a",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

root@ip-10-0-1-9:~# docker logs portworx 
Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f
Tue Jul 18 10:51:03 UTC 2017 : Running on Linux ip-10-0-1-9 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
checking /hostusr/src/kernels/4.4.0-83-generic
checking /hostusr/src/linux-headers-4.4.0-83-generic
checking /usr/src/kernels/4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Flushing buffer-cache..
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Creating px fs...
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Fingerprints values:
HOSTNAME={"type": 1, "value": "ip-10-0-1-9"}
DISK={ "type": 6, "value": "vol-0842059ddbdbd9d7f" }
NETWORK={ "type": 7, "value": "" }
NETWORK={ "type": 8, "value": "" }

FP_FINGERPRINT="Fingerprint": [{"type": 1, "value": "ip-10-0-1-9"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz" }, { "type": 5, "value": "3852912 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }]

FP_CONFIG={ "token": "", "create_profile": true, "profile_name": "ip-10-0-1-9-ChYeAcsm", "Fingerprint": [{"type": 1, "value": "ip-10-0-1-9"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz" }, { "type": 5, "value": "3852912 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }] }
To use the same hardware configuration on an identical node,        please use hardware profile ip-10-0-1-9-ChYeAcsm.
example:
    curl -fsSL http://lighthouse.portworx.com/px_bootstrap.sh > px_bootstrap.sh;  chmod u+x px_bootstrap.sh;  PWX_TOKEN= PWX_CLUSTERNAME=socialbase         bootstrap.sh -p ip-10-0-1-9-ChYeAcsm
Failed to enable rt scheduler
Checking sysfs mount...
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
2017-07-18 10:53:07,645 CRIT Supervisor running as root (no user in config file)
2017-07-18 10:53:07,648 INFO supervisord started with pid 1
2017-07-18 10:53:08,649 INFO spawned: 'relayd' with pid 1017
2017-07-18 10:53:08,651 INFO spawned: 'lttng' with pid 1018
2017-07-18 10:53:08,653 INFO spawned: 'exec' with pid 1019
2017-07-18 10:53:08,654 INFO spawned: 'pxdaemon' with pid 1020
2017-07-18 10:53:08,655 INFO spawned: 'px-ns' with pid 1021
2017-07-18 10:53:08,656 INFO spawned: 'px_event_listener' with pid 1022
PXPROCS: lttng not started yet...sleeping 

Tue Jul 18 10:53:09 UTC 2017 cleaning up old traces
Tue Jul 18 10:53:09 UTC 2017 done cleaning up old traces
2017-07-18 10:53:10,430 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:53:10,430 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:53:10,431 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:53:10,431 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:53:10,431 INFO success: px-ns entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:53:10,431 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: lttng not started yet...sleeping 

time="2017-07-18T10:53:12Z" level=info msg="px-ns Starting.." 
NS client starting fuse module
Starting NS server
PXPROCS: lttng not started yet...sleeping 

PXPROCS: lttng not started yet...sleeping 

Spawning a session daemon
Session pxd created.
Traces will be written in net://localhost
Live timer set to 2000000 usec
Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170718-105320.
UST channel pxd_channel enabled for session pxd
All UST events are enabled in channel pxd_channel
Tracing started for session pxd
PXPROCS: Started px-storage with pid 1095
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
PXPROCS: Started px with pid 1110
PXPROCS: Started watchdog with pid 1111
2017-07-18_10:53:31: PX-Watchdog: Starting watcher
2017-07-18_10:53:31: PX-Watchdog: Waiting for px process to start
root      1110  1020  0 10:53 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:53:31: PX-Watchdog: (pid 1110): Begin monitoring
time="2017-07-18T10:53:54Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:53:54Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:53:54Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 5ea17f26-cbaf-4088-bc9d-674b36594b52). Trying attachSet..." 
time="2017-07-18T10:53:55Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:53:55Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:53:55Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:53:55Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:53:55Z" level=info msg="New NodeID is f33f56bc-8c56-4aed-a6e3-582f3a1ff74e" 
time="2017-07-18T10:53:55Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:53:55Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:53:55Z" level=info msg="Node f33f56bc-8c56-4aed-a6e3-582f3a1ff74e with Index (246) is Up" 
time="2017-07-18T10:54:05Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01e92c660a0907784' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 2205893c-860f-4957-87d6-1a1464fda794). Trying attachSet..." 
time="2017-07-18T10:54:06Z" level=info msg="Failed to attach volume set  vol-01e92c660a0907784 vol-0722a31be004a9641 vol-0748ad5024791b39c with error: No sets available. Retrying..." 
time="2017-07-18T10:54:09Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01e92c660a0907784' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 6455ed30-f7f1-4cb0-bb7e-15ea1bf63daf). Trying attachSet..." 
time="2017-07-18T10:54:10Z" level=info msg="Failed to attach volume set  vol-01e92c660a0907784 vol-0722a31be004a9641 vol-0748ad5024791b39c with error: No sets available. Retrying..." 
time="2017-07-18T10:54:13Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01e92c660a0907784' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: a9633b17-f402-4c98-8538-0dcaddd03f1c). Trying attachSet..." 
time="2017-07-18T10:54:13Z" level=info msg="Failed to attach volume set  vol-01e92c660a0907784 vol-0722a31be004a9641 vol-0748ad5024791b39c with error: No sets available. Retrying..." 
time="2017-07-18T10:54:16Z" level=info msg="Deleted volume vol-01e92c660a0907784" 
time="2017-07-18T10:54:16Z" level=info msg="Deleted volume vol-0722a31be004a9641" 
time="2017-07-18T10:54:16Z" level=info msg="Deleted volume vol-0748ad5024791b39c" 
time="2017-07-18T10:54:16Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1095
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:54:18,016 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:54:19,018 INFO spawned: 'pxdaemon' with pid 1163
PXPROCS: Started px-storage with pid 1167
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:54:20,431 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1176
PXPROCS: Started watchdog with pid 1177
2017-07-18_10:54:22: PX-Watchdog: Starting watcher
2017-07-18_10:54:22: PX-Watchdog: Waiting for px process to start
root      1176  1163  0 10:54 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:54:22: PX-Watchdog: (pid 1176): Begin monitoring
time="2017-07-18T10:54:22Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:54:22Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:54:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: a0413bca-fec1-40d4-8adf-cdcb2f1bf601). Trying attachSet..." 
time="2017-07-18T10:54:22Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:54:22Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:54:22Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:54:22Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:54:22Z" level=info msg="New NodeID is 5c5ae62f-e7c7-4957-8506-c671a5b9d737" 
time="2017-07-18T10:54:22Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:54:22Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:54:22Z" level=info msg="Node 5c5ae62f-e7c7-4957-8506-c671a5b9d737 with Index (247) is Up" 
2017-07-18 10:54:34,583 INFO reaped unknown pid 1147
time="2017-07-18T10:54:36Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0045ab04b85a40940' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 1c1be04c-4e63-4b0b-9474-c5f001ba8016). Trying attachSet..." 
time="2017-07-18T10:54:36Z" level=info msg="Failed to attach volume set  vol-0a1e5ee018c71527a vol-0045ab04b85a40940 vol-0db8469b82c820b0e with error: No sets available. Retrying..." 
time="2017-07-18T10:54:40Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0045ab04b85a40940' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: e83f499b-8e63-4278-9423-444a31f9b5de). Trying attachSet..." 
time="2017-07-18T10:54:40Z" level=info msg="Failed to attach volume set  vol-0a1e5ee018c71527a vol-0045ab04b85a40940 vol-0db8469b82c820b0e with error: No sets available. Retrying..." 
time="2017-07-18T10:54:43Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0045ab04b85a40940' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 8a785d50-0149-43c3-a61a-e1407e26751b). Trying attachSet..." 
time="2017-07-18T10:54:43Z" level=info msg="Failed to attach volume set  vol-0a1e5ee018c71527a vol-0045ab04b85a40940 vol-0db8469b82c820b0e with error: No sets available. Retrying..." 
time="2017-07-18T10:54:46Z" level=info msg="Deleted volume vol-0a1e5ee018c71527a" 
time="2017-07-18T10:54:46Z" level=info msg="Deleted volume vol-0045ab04b85a40940" 
time="2017-07-18T10:54:46Z" level=info msg="Deleted volume vol-0db8469b82c820b0e" 
time="2017-07-18T10:54:46Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1167
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:54:47,744 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:54:48,746 INFO spawned: 'pxdaemon' with pid 1270
PXPROCS: Started px-storage with pid 1271
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:54:50,333 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1280
PXPROCS: Started watchdog with pid 1281
2017-07-18_10:54:51: PX-Watchdog: Starting watcher
2017-07-18_10:54:51: PX-Watchdog: Waiting for px process to start
root      1280  1270  0 10:54 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:54:51: PX-Watchdog: (pid 1280): Begin monitoring
time="2017-07-18T10:54:52Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:54:52Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:54:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 7a7c0606-7853-4c0e-8c9b-32d92b720ffc). Trying attachSet..." 
time="2017-07-18T10:54:53Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:54:53Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:54:53Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:54:53Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:54:53Z" level=info msg="New NodeID is 8e427559-aa6a-4c4d-9aee-45a3d85a0489" 
time="2017-07-18T10:54:53Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:54:53Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:54:53Z" level=info msg="Node 8e427559-aa6a-4c4d-9aee-45a3d85a0489 with Index (248) is Up" 
Waiting for state transition to "attached".
time="2017-07-18T10:55:09Z" level=warning msg="Rollback attach volume /dev/xvdf, Error InvalidVolume.ZoneMismatch: The volume 'vol-072595a4306d3de9e' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: bf3e13d6-3fda-48e1-9c22-80af40833c14" 
time="2017-07-18T10:55:09Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-072595a4306d3de9e' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: bf3e13d6-3fda-48e1-9c22-80af40833c14). Trying attachSet..." 
time="2017-07-18T10:55:10Z" level=info msg="Failed to attach volume set  vol-072595a4306d3de9e vol-0862f831687cbcde0 vol-0076642dbddf81835 with error: No sets available. Retrying..." 
time="2017-07-18T10:55:13Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-0076642dbddf81835 is already attached to an instance\n\tstatus code: 400, request id: 9df38e28-0a35-44a6-b99e-e3894fed6044). Trying attachSet..." 
time="2017-07-18T10:55:13Z" level=info msg="Failed to attach volume set  vol-072595a4306d3de9e vol-0862f831687cbcde0 vol-0076642dbddf81835 with error: No sets available. Retrying..." 
time="2017-07-18T10:55:17Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-0076642dbddf81835 is already attached to an instance\n\tstatus code: 400, request id: 27b24a7b-a945-4904-b953-7722de251354). Trying attachSet..." 
time="2017-07-18T10:55:17Z" level=info msg="Failed to attach volume set  vol-072595a4306d3de9e vol-0862f831687cbcde0 vol-0076642dbddf81835 with error: No sets available. Retrying..." 
time="2017-07-18T10:55:20Z" level=info msg="Deleted volume vol-072595a4306d3de9e" 
time="2017-07-18T10:55:20Z" level=info msg="Deleted volume vol-0862f831687cbcde0" 
time="2017-07-18T10:55:20Z" level=warning msg="Delete failed for volume vol-0076642dbddf81835, Error VolumeInUse: Volume vol-0076642dbddf81835 is currently attached to i-07b3f498e4a2a2441\n\tstatus code: 400, request id: 384b1d45-f76e-4a3b-89a7-fb252b042837" 
time="2017-07-18T10:55:20Z" level=info msg="Deleted volume vol-0076642dbddf81835" 
time="2017-07-18T10:55:20Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1271
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:55:21,528 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:55:22,530 INFO spawned: 'pxdaemon' with pid 1334
PXPROCS: Started px-storage with pid 1335
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:55:24,113 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-18 10:55:25,080 INFO reaped unknown pid 1221
PXPROCS: Started px with pid 1344
PXPROCS: Started watchdog with pid 1345
2017-07-18_10:55:25: PX-Watchdog: Starting watcher
2017-07-18_10:55:25: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1167-time1500375286
2017-07-18_10:55:25: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:54:46.872231153 +0000 UTC.stack
2017-07-18_10:55:25: PX-Watchdog: Waiting for px process to start
root      1344  1334  0 10:55 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:55:25: PX-Watchdog: (pid 1344): Begin monitoring
time="2017-07-18T10:55:25Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:55:25Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:55:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 3336988e-151c-4e32-8c83-9a58add6bcc7). Trying attachSet..." 
time="2017-07-18T10:55:26Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:55:26Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:55:26Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:55:26Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:55:26Z" level=info msg="New NodeID is d7b5b974-00aa-42a9-aee5-f87a05be7cc0" 
time="2017-07-18T10:55:26Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:55:26Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:55:26Z" level=info msg="Node d7b5b974-00aa-42a9-aee5-f87a05be7cc0 with Index (249) is Up" 
Waiting for state transition to "attached".
time="2017-07-18T10:55:49Z" level=warning msg="Rollback attach volume /dev/xvdg, Error InvalidVolume.ZoneMismatch: The volume 'vol-07eb92cc0a06b8feb' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 69ba12b8-59ee-41e1-9cb3-9a2b4332a088" 
time="2017-07-18T10:55:49Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07eb92cc0a06b8feb' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 69ba12b8-59ee-41e1-9cb3-9a2b4332a088). Trying attachSet..." 
time="2017-07-18T10:55:49Z" level=info msg="Failed to attach volume set  vol-07eb92cc0a06b8feb vol-09c30c6651676c12c vol-0462d35b85744c45d with error: No sets available. Retrying..." 
time="2017-07-18T10:55:53Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-0462d35b85744c45d is already attached to an instance\n\tstatus code: 400, request id: cc0ef9ac-3132-447b-8937-d38dc0f78b01). Trying attachSet..." 
time="2017-07-18T10:55:53Z" level=info msg="Failed to attach volume set  vol-07eb92cc0a06b8feb vol-09c30c6651676c12c vol-0462d35b85744c45d with error: No sets available. Retrying..." 
2017-07-18 10:55:54,835 INFO reaped unknown pid 1322
time="2017-07-18T10:55:56Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-0462d35b85744c45d is already attached to an instance\n\tstatus code: 400, request id: 034ac855-3eee-4ded-90eb-0a1f4168b9af). Trying attachSet..." 
time="2017-07-18T10:55:57Z" level=info msg="Failed to attach volume set  vol-07eb92cc0a06b8feb vol-09c30c6651676c12c vol-0462d35b85744c45d with error: No sets available. Retrying..." 
time="2017-07-18T10:56:00Z" level=info msg="Deleted volume vol-07eb92cc0a06b8feb" 
time="2017-07-18T10:56:00Z" level=info msg="Deleted volume vol-09c30c6651676c12c" 
time="2017-07-18T10:56:00Z" level=warning msg="Delete failed for volume vol-0462d35b85744c45d, Error VolumeInUse: Volume vol-0462d35b85744c45d is currently attached to i-07b3f498e4a2a2441\n\tstatus code: 400, request id: 288c9da7-c3d0-4579-91ff-ea9383b611e1" 
time="2017-07-18T10:56:00Z" level=info msg="Deleted volume vol-0462d35b85744c45d" 
time="2017-07-18T10:56:00Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1335
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:56:01,184 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:56:02,186 INFO spawned: 'pxdaemon' with pid 1400
PXPROCS: Started px-storage with pid 1401
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:56:03,788 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1410
PXPROCS: Started watchdog with pid 1411
2017-07-18_10:56:05: PX-Watchdog: Starting watcher
2017-07-18_10:56:05: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1271-time1500375320
2017-07-18_10:56:05: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:55:20.638325924 +0000 UTC.stack
2017-07-18_10:56:05: PX-Watchdog: Waiting for px process to start
root      1410  1400  0 10:56 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:56:05: PX-Watchdog: (pid 1410): Begin monitoring
time="2017-07-18T10:56:05Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:56:05Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:56:05Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: d97f7fcc-841d-4991-9b36-6729171b4899). Trying attachSet..." 
time="2017-07-18T10:56:06Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:56:06Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:56:06Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:56:06Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:56:06Z" level=info msg="New NodeID is a5a3e22d-6657-4bd2-a384-15bc6dc970e8" 
time="2017-07-18T10:56:06Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:56:06Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:56:06Z" level=info msg="Node a5a3e22d-6657-4bd2-a384-15bc6dc970e8 with Index (250) is Up" 
time="2017-07-18T10:56:20Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07c8b6cdb5c947ecd' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 0a96f615-0a89-43af-9ab1-0ce0b5ca8776). Trying attachSet..." 
time="2017-07-18T10:56:20Z" level=info msg="Failed to attach volume set  vol-0971737c1e9ebbadf vol-07c8b6cdb5c947ecd vol-0cb491c9f8c50d6e6 with error: No sets available. Retrying..." 
time="2017-07-18T10:56:23Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07c8b6cdb5c947ecd' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: cb4c7fb5-df33-4eb8-bc17-6321b0e49d24). Trying attachSet..." 
time="2017-07-18T10:56:23Z" level=info msg="Failed to attach volume set  vol-0971737c1e9ebbadf vol-07c8b6cdb5c947ecd vol-0cb491c9f8c50d6e6 with error: No sets available. Retrying..." 
time="2017-07-18T10:56:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07c8b6cdb5c947ecd' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: c9597efe-0c23-48d7-ad82-6a8fa759f73c). Trying attachSet..." 
time="2017-07-18T10:56:26Z" level=info msg="Failed to attach volume set  vol-0971737c1e9ebbadf vol-07c8b6cdb5c947ecd vol-0cb491c9f8c50d6e6 with error: No sets available. Retrying..." 
2017-07-18 10:56:28,673 INFO reaped unknown pid 1392
time="2017-07-18T10:56:31Z" level=info msg="Deleted volume vol-0971737c1e9ebbadf" 
time="2017-07-18T10:56:31Z" level=info msg="Deleted volume vol-07c8b6cdb5c947ecd" 
time="2017-07-18T10:56:31Z" level=info msg="Deleted volume vol-0cb491c9f8c50d6e6" 
time="2017-07-18T10:56:31Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1401
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:56:32,332 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:56:33,334 INFO spawned: 'pxdaemon' with pid 1465
PXPROCS: Started px-storage with pid 1466
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:56:34,928 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1475
PXPROCS: Started watchdog with pid 1476
2017-07-18_10:56:36: PX-Watchdog: Starting watcher
2017-07-18_10:56:36: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1335-time1500375360
2017-07-18_10:56:36: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:56:00.475475355 +0000 UTC.stack
2017-07-18_10:56:36: PX-Watchdog: Waiting for px process to start
time="2017-07-18T10:56:36Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:56:36Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1475  1465  0 10:56 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:56:36: PX-Watchdog: (pid 1475): Begin monitoring
time="2017-07-18T10:56:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 8a443c49-dcb7-45e0-8bbf-f73f0aa758a5). Trying attachSet..." 
time="2017-07-18T10:56:37Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:56:37Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:56:37Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:56:37Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:56:37Z" level=info msg="New NodeID is 01605562-5a58-40f6-97db-a3c0e5e21cd4" 
time="2017-07-18T10:56:37Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:56:37Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:56:37Z" level=info msg="Node 01605562-5a58-40f6-97db-a3c0e5e21cd4 with Index (251) is Up" 
time="2017-07-18T10:56:42Z" level=warning msg="Unable to save new node configuration: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: e45a1262-0b6a-4088-8e4c-3af58f566909" 
PXPROCS: px daemon exited with code: 6 

1466
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:56:43,031 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:56:44,034 INFO spawned: 'pxdaemon' with pid 1531
PXPROCS: Started px-storage with pid 1532
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:56:45,609 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1541
PXPROCS: Started watchdog with pid 1542
2017-07-18_10:56:47: PX-Watchdog: Starting watcher
2017-07-18_10:56:47: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1401-time1500375391
2017-07-18_10:56:47: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:56:31.564340556 +0000 UTC.stack
2017-07-18_10:56:47: PX-Watchdog: Waiting for px process to start
root      1541  1531  0 10:56 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:56:47: PX-Watchdog: (pid 1541): Begin monitoring
time="2017-07-18T10:56:47Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:56:47Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-18T10:56:53Z" level=warning msg="AttachSet: attachVols failed with error (RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: b553ba36-2462-4ca2-900f-fc9ca146ccb1). Trying attachSet..." 
time="2017-07-18T10:56:53Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:56:53Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:56:53Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:56:53Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:56:53Z" level=info msg="New NodeID is ec9aaaef-96bf-4f35-bd69-ef951b0c3c23" 
time="2017-07-18T10:56:53Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:56:53Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:56:53Z" level=info msg="Node ec9aaaef-96bf-4f35-bd69-ef951b0c3c23 with Index (252) is Up" 
time="2017-07-18T10:56:58Z" level=warning msg="Unable to save new node configuration: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: c48e8eaa-e142-428b-88cc-616e1b080c47" 
PXPROCS: px daemon exited with code: 6 

1532
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:56:58,389 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:56:59,392 INFO spawned: 'pxdaemon' with pid 1596
PXPROCS: Started px-storage with pid 1597
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:57:00,995 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1606
PXPROCS: Started watchdog with pid 1607
2017-07-18_10:57:02: PX-Watchdog: Starting watcher
2017-07-18_10:57:02: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1466-time1500375402
time="2017-07-18T10:57:02Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:57:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-18_10:57:02: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:56:42.843407744 +0000 UTC.stack
2017-07-18_10:57:02: PX-Watchdog: Waiting for px process to start
root      1606  1596  0 10:57 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:57:02: PX-Watchdog: (pid 1606): Begin monitoring
time="2017-07-18T10:57:04Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: e83754da-f751-46f9-abba-f84d6484cae1). Trying attachSet..." 
time="2017-07-18T10:57:04Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:57:04Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:57:04Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:57:04Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:57:04Z" level=info msg="New NodeID is 64f13526-0f47-4389-a169-30054dd761ad" 
time="2017-07-18T10:57:04Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:57:04Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:57:04Z" level=info msg="Node 64f13526-0f47-4389-a169-30054dd761ad with Index (253) is Up" 
time="2017-07-18T10:57:08Z" level=warning msg="Unable to save new node configuration: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: dba20d27-9911-41f1-83bd-6b6571220d78" 
PXPROCS: px daemon exited with code: 6 

1597
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:57:08,304 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:57:08,382 INFO spawned: 'pxdaemon' with pid 1660
2017-07-18 10:57:08,383 INFO reaped unknown pid 1458
PXPROCS: Started px-storage with pid 1661
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:57:09,998 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1670
PXPROCS: Started watchdog with pid 1671
2017-07-18_10:57:11: PX-Watchdog: Starting watcher
2017-07-18_10:57:11: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1532-time1500375418
time="2017-07-18T10:57:11Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:57:11Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-18_10:57:11: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:56:58.194271715 +0000 UTC.stack
2017-07-18_10:57:11: PX-Watchdog: Waiting for px process to start
root      1670  1660  0 10:57 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:57:11: PX-Watchdog: (pid 1670): Begin monitoring
time="2017-07-18T10:57:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 7a4d7db4-0fb4-4f60-aa9b-517d11dfb65b). Trying attachSet..." 
time="2017-07-18T10:57:12Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:57:12Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:57:12Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:57:12Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:57:12Z" level=info msg="New NodeID is ea37d08f-2881-4e01-bc35-a6935911d8dd" 
time="2017-07-18T10:57:12Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:57:12Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:57:12Z" level=info msg="Node ea37d08f-2881-4e01-bc35-a6935911d8dd with Index (254) is Up" 
time="2017-07-18T10:57:18Z" level=warning msg="Unable to save new node configuration: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: d00530ee-3a80-4927-b814-b936b810309b" 
PXPROCS: px daemon exited with code: 6 

1661
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:57:18,292 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:57:19,294 INFO spawned: 'pxdaemon' with pid 1725
PXPROCS: Started px-storage with pid 1726
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:57:20,801 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1738
PXPROCS: Started watchdog with pid 1739
2017-07-18_10:57:22: PX-Watchdog: Starting watcher
2017-07-18_10:57:22: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1597-time1500375428
2017-07-18_10:57:22: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:57:08.223482517 +0000 UTC.stack
2017-07-18_10:57:22: PX-Watchdog: Waiting for px process to start
time="2017-07-18T10:57:22Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:57:22Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1738  1725  0 10:57 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:57:22: PX-Watchdog: (pid 1738): Begin monitoring
time="2017-07-18T10:57:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 6bddda3d-844b-4dd2-a2ec-035aa6f13fd4). Trying attachSet..." 
time="2017-07-18T10:57:22Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:57:22Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:57:22Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:57:22Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:57:22Z" level=info msg="New NodeID is daeac1c7-0ef3-49be-bbc9-ce2eb087712e" 
time="2017-07-18T10:57:22Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:57:22Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:57:23Z" level=info msg="Node daeac1c7-0ef3-49be-bbc9-ce2eb087712e with Index (255) is Up" 
time="2017-07-18T10:57:28Z" level=warning msg="Unable to save new node configuration: RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: d9e66ea2-1259-413f-9896-444ade4dd17f" 
PXPROCS: px daemon exited with code: 6 

1726
PXPROCS: dumping core and killing px-storage... 

2017-07-18 10:57:29,018 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-18 10:57:30,020 INFO spawned: 'pxdaemon' with pid 1794
PXPROCS: Started px-storage with pid 1795
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-18 10:57:31,620 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1804
PXPROCS: Started watchdog with pid 1805
2017-07-18_10:57:33: PX-Watchdog: Starting watcher
2017-07-18_10:57:33: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1661-time1500375438
2017-07-18_10:57:33: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-18 10:57:18.215382323 +0000 UTC.stack
2017-07-18_10:57:33: PX-Watchdog: Waiting for px process to start
time="2017-07-18T10:57:33Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-18T10:57:33Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1804  1794  0 10:57 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-18_10:57:33: PX-Watchdog: (pid 1804): Begin monitoring
time="2017-07-18T10:57:36Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 5b2f3eb3-e5e8-4fe1-b707-3eadad7afaf3). Trying attachSet..." 
time="2017-07-18T10:57:36Z" level=info msg="AttachSet while Load of private config failed: No sets available" 
time="2017-07-18T10:57:36Z" level=info msg="Node is not yet initialized" 
time="2017-07-18T10:57:36Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-18T10:57:36Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-18T10:57:36Z" level=info msg="New NodeID is 90953e20-5f0b-4483-b4b4-abb9b43e0636" 
time="2017-07-18T10:57:36Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-18T10:57:36Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-18T10:57:36Z" level=info msg="Node 90953e20-5f0b-4483-b4b4-abb9b43e0636 with Index (256) is Up" 
2017-07-18 10:57:39,537 INFO reaped unknown pid 1525
time="2017-07-18T10:57:48Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-031fddb2bffbc8d8d' is not in the same availability zone as instance 'i-07b3f498e4a2a2441'\n\tstatus code: 400, request id: 36bb811b-e847-4b66-bf2f-0e0d4c2b84d8). Trying attachSet..." 
time="2017-07-18T10:57:48Z" level=info msg="Failed to attach volume set  vol-09ed82552258e7f86 vol-031fddb2bffbc8d8d vol-0c40674bc3d576274 with error: No sets available. Retrying..." 
2017-07-18 10:57:50,164 INFO reaped unknown pid 1589
time="2017-07-18T10:57:57Z" level=warning msg="AttachSet: attachVols failed with error (RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: 34d8b49e-9904-46ed-8dd5-cd1e304a22ae). Trying attachSet..." 
time="2017-07-18T10:57:57Z" level=info msg="Failed to attach volume set  vol-09ed82552258e7f86 vol-031fddb2bffbc8d8d vol-0c40674bc3d576274 with error: No sets available. Retrying..." 
2017-07-18 10:58:05,591 INFO reaped unknown pid 1655
time="2017-07-18T10:58:06Z" level=warning msg="AttachSet: attachVols failed with error (RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503, request id: 2565b4ad-a0e0-4720-b451-d552069ae84c). Trying attachSet..." 
time="2017-07-18T10:58:06Z" level=info msg="Failed to attach volume set  vol-09ed82552258e7f86 vol-031fddb2bffbc8d8d vol-0c40674bc3d576274 with error: No sets available. Retrying..." 
2017-07-18 10:58:10,654 WARN received SIGTERM indicating exit request
2017-07-18 10:58:10,654 INFO waiting for lttng, px-ns, exec, px_event_listener, relayd, pxdaemon to die
2017-07-18 10:58:10,655 INFO stopped: px_event_listener (terminated by SIGTERM)
2017-07-18 10:58:10,656 INFO stopped: px-ns (terminated by SIGTERM)
PXPROCS: Received SIGTERM. Exiting PX gracefully....
PXPROCS: trying to kill px daemon... 

time="2017-07-18T10:58:10Z" level=info msg="Received signal 'terminated', starting shutdown..." 
time="2017-07-18T10:58:10Z" level=info msg="Shutdown complete" 
PXPROCS: px daemon exited with code: 0 

1795
PXPROCS: trying to kill px-storage... 

2017-07-18 10:58:10,713 INFO stopped: pxdaemon (exit status 0)
2017-07-18 10:58:10,714 INFO stopped: exec (terminated by SIGTERM)
Waiting for data availability
Tracing stopped for session pxd
Session pxd destroyed
Exiting lttng
2017-07-18 10:58:10,728 INFO stopped: lttng (exit status 0)
2017-07-18 10:58:10,857 INFO stopped: relayd (exit status 0)

root@ip-10-0-1-9:~# curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone ; echo -e '\n' 
sa-east-1a

portworx / px-dev

Error InvalidVolume.ZoneMismatch in scalingUP #42