Volume is made Read-only

[adrianNEMO]

I use Longhorn for a while now. I noticed that form time to time some volumes are somehow made read only. Which causes my application to crash because it cant write to the volume. Are there some circumstances known where such behavior could occur? Example output from mysql server:

chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata/routers.ibd': Read-only file system
chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata/instances.ibd': Read-only file system
chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata/hosts.ibd': Read-only file system
chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata/replicasets.ibd': Read-only file system
chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata/clusters.ibd': Read-only file system
chown: changing ownership of '/var/lib/mysql/mysql_innodb_cluster_metadata': Read-only file system

The csi-plugin container writes this:

time="2018-12-11T14:05:15Z" level=info msg="NodeServer NodePublishVolume req: volume_id:\"pvc-b54aa73afd4d11e8\" target_path:\"/var/lib/kubelet/pods/b54ce02c-fd4d-11e8-80a8-0e1f082b1b80/volumes/kubernetes.io~csi/pvc-b54aa73afd4d11e8/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_attributes:<key:\"baseImage\" value:\"\" > volume_attributes:<key:\"fromBackup\" value:\"\" > volume_attributes:<key:\"numberOfReplicas\" value:\"3\" > volume_attributes:<key:\"staleReplicaTimeout\" value:\"30\" > volume_attributes:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1544421214637-8081-rancher.io/longhorn\" > "
E1211 14:05:15.305522 1 mount_linux.go:151] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-b54aa73afd4d11e8 /var/lib/kubelet/pods/b54ce02c-fd4d-11e8-80a8-0e1f082b1b80/volumes/kubernetes.io~csi/pvc-b54aa73afd4d11e8/mount
Output: mount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-b54aa73afd4d11e8,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
time="2018-12-11T14:05:15Z" level=debug msg="NodePublishVolume: done pvc-b54aa73afd4d11e8"

[yasker]

@db-adrian Can you check if the managers' log? Normally it because of the engine crashed. Last time (https://github.com/rancher/longhorn/issues/350) we found that Kubernetes may evict the Longhorn engines because of the CPU/memory utilization on the node.

[adrianNEMO]

These are the logs from the manager pods from the time period where I think it happend:

December 13th 2018, 04:40:39.873 | time="2018-12-13T03:40:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"
-- | --

  | December 13th 2018, 04:40:40.100 | time="2018-12-13T03:40:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:40.899 | time="2018-12-13T03:40:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:41.300 | time="2018-12-13T03:40:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:43.425 | E1213 03:40:43.425720       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:43.425 | E1213 03:40:43.425549       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:43.427 | E1213 03:40:43.427515       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:43.428 | E1213 03:40:43.428549       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:43.430 | E1213 03:40:43.429859       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:44.886 | E1213 03:40:44.886460       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:47.054 | time="2018-12-13T03:40:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:47.103 | time="2018-12-13T03:40:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.423 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.755 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.844 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.901 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.936 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:48.970 | time="2018-12-13T03:40:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:49.021 | time="2018-12-13T03:40:49Z" level=debug msg="Requeue longhorn-system/worker1 due to conflict"

  | December 13th 2018, 04:40:51.208 | E1213 03:40:51.207924       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:52.175 | E1213 03:40:52.175706       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:52.454 | E1213 03:40:52.454720       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:52.470 | time="2018-12-13T03:40:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:52.497 | time="2018-12-13T03:40:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:53.444 | E1213 03:40:53.444588       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:53.448 | E1213 03:40:53.448736       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:53.451 | E1213 03:40:53.451584       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:53.454 | E1213 03:40:53.454381       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:53.455 | E1213 03:40:53.454857       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:53.616 | time="2018-12-13T03:40:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:53.645 | time="2018-12-13T03:40:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:53.671 | time="2018-12-13T03:40:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:53.692 | time="2018-12-13T03:40:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:54.828 | time="2018-12-13T03:40:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:56.031 | time="2018-12-13T03:40:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:56.229 | time="2018-12-13T03:40:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:56.429 | time="2018-12-13T03:40:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:56.640 | time="2018-12-13T03:40:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:56.831 | time="2018-12-13T03:40:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:57.029 | time="2018-12-13T03:40:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:57.231 | time="2018-12-13T03:40:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:58.052 | time="2018-12-13T03:40:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:58.213 | E1213 03:40:58.213764       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:40:58.240 | time="2018-12-13T03:40:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:58.821 | time="2018-12-13T03:40:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:58.860 | time="2018-12-13T03:40:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:58.891 | time="2018-12-13T03:40:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:59.028 | time="2018-12-13T03:40:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:59.236 | time="2018-12-13T03:40:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:59.242 | time="2018-12-13T03:40:59Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:40:59.429 | time="2018-12-13T03:40:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:40:59.829 | time="2018-12-13T03:40:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:00.237 | time="2018-12-13T03:41:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:00.838 | time="2018-12-13T03:41:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:01.049 | time="2018-12-13T03:41:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:01.239 | time="2018-12-13T03:41:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:01.243 | time="2018-12-13T03:41:01Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:41:01.438 | time="2018-12-13T03:41:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:01.638 | time="2018-12-13T03:41:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:01.642 | time="2018-12-13T03:41:01Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:41:01.838 | time="2018-12-13T03:41:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:02.036 | time="2018-12-13T03:41:02Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:02.045 | time="2018-12-13T03:41:02Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:41:02.237 | time="2018-12-13T03:41:02Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:02.437 | time="2018-12-13T03:41:02Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:03.473 | E1213 03:41:03.472204       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:03.475 | E1213 03:41:03.475291       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:03.475 | E1213 03:41:03.475224       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:03.478 | E1213 03:41:03.476593       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:03.479 | E1213 03:41:03.478856       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:06.290 | time="2018-12-13T03:41:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:08.039 | time="2018-12-13T03:41:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:08.234 | time="2018-12-13T03:41:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:08.307 | time="2018-12-13T03:41:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:08.675 | time="2018-12-13T03:41:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:09.170 | time="2018-12-13T03:41:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:09.433 | time="2018-12-13T03:41:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:09.494 | time="2018-12-13T03:41:09Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:41:09.591 | time="2018-12-13T03:41:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:09.675 | time="2018-12-13T03:41:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:10.034 | time="2018-12-13T03:41:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:10.289 | time="2018-12-13T03:41:10Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:41:10.318 | time="2018-12-13T03:41:10Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:41:10Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:41:10Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:41:10.319 | time="2018-12-13T03:41:10Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:41:10Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:41:10Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:41:10.337 | time="2018-12-13T03:41:10Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:41:10.835 | time="2018-12-13T03:41:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:11.233 | time="2018-12-13T03:41:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:11.433 | time="2018-12-13T03:41:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:13.571 | E1213 03:41:13.570824       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:13.580 | E1213 03:41:13.580472       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:13.585 | E1213 03:41:13.585178       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:13.589 | E1213 03:41:13.589503       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:13.591 | E1213 03:41:13.591038       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:14.886 | E1213 03:41:14.886677       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:17.056 | time="2018-12-13T03:41:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:17.213 | time="2018-12-13T03:41:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:18.520 | time="2018-12-13T03:41:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:18.563 | time="2018-12-13T03:41:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:18.861 | time="2018-12-13T03:41:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:18.888 | time="2018-12-13T03:41:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:21.208 | E1213 03:41:21.208150       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:22.176 | E1213 03:41:22.175892       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:22.455 | E1213 03:41:22.454880       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:22.474 | time="2018-12-13T03:41:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:22.501 | time="2018-12-13T03:41:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:23.616 | time="2018-12-13T03:41:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:23.646 | time="2018-12-13T03:41:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:23.672 | time="2018-12-13T03:41:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:23.703 | time="2018-12-13T03:41:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:23.722 | E1213 03:41:23.721904       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:23.725 | E1213 03:41:23.725177       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:23.726 | E1213 03:41:23.726420       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:23.729 | E1213 03:41:23.729448       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:23.730 | E1213 03:41:23.727775       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:24.335 | time="2018-12-13T03:41:24Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:25.469 | time="2018-12-13T03:41:25Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:26.069 | time="2018-12-13T03:41:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:26.265 | time="2018-12-13T03:41:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:26.465 | time="2018-12-13T03:41:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:28.049 | time="2018-12-13T03:41:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:28.214 | E1213 03:41:28.214046       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:28.437 | time="2018-12-13T03:41:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:28.894 | time="2018-12-13T03:41:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:28.918 | time="2018-12-13T03:41:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:29.437 | time="2018-12-13T03:41:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:29.444 | time="2018-12-13T03:41:29Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:41:30.437 | time="2018-12-13T03:41:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:31.237 | time="2018-12-13T03:41:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:31.437 | time="2018-12-13T03:41:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:31.640 | time="2018-12-13T03:41:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:31.839 | time="2018-12-13T03:41:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:32.037 | time="2018-12-13T03:41:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:32.236 | time="2018-12-13T03:41:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:33.728 | E1213 03:41:33.727794       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:33.732 | E1213 03:41:33.731849       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:33.734 | E1213 03:41:33.734497       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:33.735 | E1213 03:41:33.735743       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:33.738 | E1213 03:41:33.738349       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:36.302 | time="2018-12-13T03:41:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:37.838 | time="2018-12-13T03:41:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:37.875 | time="2018-12-13T03:41:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:38.275 | time="2018-12-13T03:41:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:38.475 | time="2018-12-13T03:41:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:38.675 | time="2018-12-13T03:41:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:39.116 | time="2018-12-13T03:41:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:39.434 | time="2018-12-13T03:41:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:39.475 | time="2018-12-13T03:41:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:39.875 | time="2018-12-13T03:41:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:40.035 | time="2018-12-13T03:41:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:40.453 | E1213 03:41:40.453000       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:41:40.453 | time="2018-12-13T03:41:40Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:41:40.881 | time="2018-12-13T03:41:40Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:41:40.892 | time="2018-12-13T03:41:40Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:41:40Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:41:40Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:41:40.893 | time="2018-12-13T03:41:40Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:41:40Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:41:40Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:41:40.907 | time="2018-12-13T03:41:40Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:41:43.752 | E1213 03:41:43.751859       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:43.753 | E1213 03:41:43.753576       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:43.757 | E1213 03:41:43.756414       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:43.792 | E1213 03:41:43.760630       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:44.041 | E1213 03:41:43.760137       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:44.887 | E1213 03:41:44.887027       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:47.052 | time="2018-12-13T03:41:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:47.085 | time="2018-12-13T03:41:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:47.139 | time="2018-12-13T03:41:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:47.170 | time="2018-12-13T03:41:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:47.193 | time="2018-12-13T03:41:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:48.628 | time="2018-12-13T03:41:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:48.655 | time="2018-12-13T03:41:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:49.009 | time="2018-12-13T03:41:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:49.043 | time="2018-12-13T03:41:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:49.082 | time="2018-12-13T03:41:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:51.208 | E1213 03:41:51.208219       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:52.176 | E1213 03:41:52.176128       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:52.455 | E1213 03:41:52.455134       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:52.476 | time="2018-12-13T03:41:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:52.512 | time="2018-12-13T03:41:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.612 | time="2018-12-13T03:41:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.637 | time="2018-12-13T03:41:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.665 | time="2018-12-13T03:41:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.702 | time="2018-12-13T03:41:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.722 | time="2018-12-13T03:41:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:53.777 | E1213 03:41:53.777311       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:53.779 | E1213 03:41:53.779825       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:53.782 | E1213 03:41:53.779875       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:53.782 | E1213 03:41:53.782106       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:53.783 | E1213 03:41:53.783435       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:54.965 | time="2018-12-13T03:41:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:55.968 | time="2018-12-13T03:41:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:56.164 | time="2018-12-13T03:41:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:56.364 | time="2018-12-13T03:41:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:58.041 | time="2018-12-13T03:41:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:58.214 | E1213 03:41:58.214206       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:41:58.655 | time="2018-12-13T03:41:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:59.051 | time="2018-12-13T03:41:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:59.108 | time="2018-12-13T03:41:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:41:59.639 | time="2018-12-13T03:41:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:00.437 | time="2018-12-13T03:42:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:01.237 | time="2018-12-13T03:42:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:01.437 | time="2018-12-13T03:42:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:01.641 | time="2018-12-13T03:42:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:01.837 | time="2018-12-13T03:42:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:02.037 | time="2018-12-13T03:42:02Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:02.237 | time="2018-12-13T03:42:02Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:03.799 | E1213 03:42:03.799489       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:03.799 | E1213 03:42:03.799436       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:03.804 | E1213 03:42:03.803383       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:03.804 | E1213 03:42:03.804301       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:03.807 | E1213 03:42:03.806855       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:06.306 | time="2018-12-13T03:42:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:07.863 | time="2018-12-13T03:42:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:07.883 | time="2018-12-13T03:42:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:08.476 | time="2018-12-13T03:42:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:08.679 | time="2018-12-13T03:42:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:09.089 | time="2018-12-13T03:42:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:09.277 | time="2018-12-13T03:42:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:09.286 | time="2018-12-13T03:42:09Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:42:09.436 | time="2018-12-13T03:42:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:09.677 | time="2018-12-13T03:42:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:10.042 | time="2018-12-13T03:42:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:10.277 | time="2018-12-13T03:42:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:10.637 | time="2018-12-13T03:42:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:10.678 | time="2018-12-13T03:42:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:11.033 | time="2018-12-13T03:42:11Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:42:11.245 | time="2018-12-13T03:42:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:11.692 | time="2018-12-13T03:42:11Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:42:11.724 | time="2018-12-13T03:42:11Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:42:11Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:42:11Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:42:11.725 | time="2018-12-13T03:42:11Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:42:11Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:42:11Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:42:11.737 | time="2018-12-13T03:42:11Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:42:11.834 | time="2018-12-13T03:42:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:12.034 | time="2018-12-13T03:42:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:13.820 | E1213 03:42:13.820000       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:13.821 | E1213 03:42:13.820948       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:13.824 | E1213 03:42:13.824256       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:13.824 | E1213 03:42:13.823869       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:13.826 | E1213 03:42:13.826799       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:14.887 | E1213 03:42:14.887112       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:17.060 | time="2018-12-13T03:42:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:17.144 | time="2018-12-13T03:42:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:17.166 | time="2018-12-13T03:42:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:18.763 | time="2018-12-13T03:42:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:18.791 | time="2018-12-13T03:42:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:18.824 | time="2018-12-13T03:42:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:19.118 | time="2018-12-13T03:42:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:21.208 | E1213 03:42:21.208557       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:22.176 | E1213 03:42:22.176270       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:22.458 | E1213 03:42:22.458513       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:22.474 | time="2018-12-13T03:42:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:22.502 | time="2018-12-13T03:42:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:23.616 | time="2018-12-13T03:42:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:23.641 | time="2018-12-13T03:42:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:23.745 | time="2018-12-13T03:42:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:23.853 | E1213 03:42:23.852950       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:23.853 | E1213 03:42:23.853703       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:23.856 | E1213 03:42:23.856061       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:23.858 | E1213 03:42:23.858166       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:23.861 | E1213 03:42:23.860715       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:24.741 | time="2018-12-13T03:42:24Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:26.146 | time="2018-12-13T03:42:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:26.941 | time="2018-12-13T03:42:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:27.141 | time="2018-12-13T03:42:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:27.343 | time="2018-12-13T03:42:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:27.547 | time="2018-12-13T03:42:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:27.741 | time="2018-12-13T03:42:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:27.942 | time="2018-12-13T03:42:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:28.049 | time="2018-12-13T03:42:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:28.214 | E1213 03:42:28.214449       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:28.524 | time="2018-12-13T03:42:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:28.562 | time="2018-12-13T03:42:28Z" level=debug msg="Requeue longhorn-system/worker5 due to conflict"

  | December 13th 2018, 04:42:29.154 | time="2018-12-13T03:42:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:29.359 | time="2018-12-13T03:42:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:29.459 | time="2018-12-13T03:42:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:29.467 | time="2018-12-13T03:42:29Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:42:29.941 | time="2018-12-13T03:42:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:30.459 | time="2018-12-13T03:42:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:31.259 | time="2018-12-13T03:42:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:31.661 | time="2018-12-13T03:42:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:31.668 | time="2018-12-13T03:42:31Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:42:31.860 | time="2018-12-13T03:42:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:32.060 | time="2018-12-13T03:42:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:32.259 | time="2018-12-13T03:42:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:33.878 | E1213 03:42:33.877533       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:33.878 | E1213 03:42:33.877812       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:33.879 | E1213 03:42:33.879510       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:33.881 | E1213 03:42:33.880993       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:33.882 | E1213 03:42:33.882638       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:36.310 | time="2018-12-13T03:42:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:37.677 | time="2018-12-13T03:42:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:37.852 | time="2018-12-13T03:42:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:38.278 | time="2018-12-13T03:42:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:38.678 | time="2018-12-13T03:42:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:39.165 | time="2018-12-13T03:42:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:39.288 | time="2018-12-13T03:42:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:39.304 | time="2018-12-13T03:42:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:39.678 | time="2018-12-13T03:42:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:39.901 | time="2018-12-13T03:42:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:40.277 | time="2018-12-13T03:42:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:40.501 | time="2018-12-13T03:42:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:40.508 | time="2018-12-13T03:42:40Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:42:40.677 | time="2018-12-13T03:42:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:41.087 | time="2018-12-13T03:42:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:41.099 | time="2018-12-13T03:42:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:41.480 | time="2018-12-13T03:42:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:41.700 | time="2018-12-13T03:42:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:41.887 | E1213 03:42:41.886990       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:42:41.887 | time="2018-12-13T03:42:41Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:42:42.149 | time="2018-12-13T03:42:42Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:42:42.168 | time="2018-12-13T03:42:42Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:42:42Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:42:42Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:42:42.170 | time="2018-12-13T03:42:42Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:42:42Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:42:42Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:42:42.183 | time="2018-12-13T03:42:42Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:42:43.898 | E1213 03:42:43.897729       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:43.900 | E1213 03:42:43.900130       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:43.901 | E1213 03:42:43.901170       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:43.903 | E1213 03:42:43.902850       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:43.907 | E1213 03:42:43.907062       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:44.887 | E1213 03:42:44.887203       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:47.052 | time="2018-12-13T03:42:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:47.107 | time="2018-12-13T03:42:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:47.128 | time="2018-12-13T03:42:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:47.148 | time="2018-12-13T03:42:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:48.896 | time="2018-12-13T03:42:48Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:49.203 | time="2018-12-13T03:42:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:49.241 | time="2018-12-13T03:42:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:51.208 | E1213 03:42:51.208652       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:52.176 | E1213 03:42:52.176433       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:52.458 | E1213 03:42:52.458721       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:52.474 | time="2018-12-13T03:42:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:52.499 | time="2018-12-13T03:42:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:52.517 | time="2018-12-13T03:42:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:52.543 | time="2018-12-13T03:42:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:53.620 | time="2018-12-13T03:42:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:53.648 | time="2018-12-13T03:42:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:53.925 | E1213 03:42:53.925087       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:53.928 | E1213 03:42:53.928334       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:53.929 | E1213 03:42:53.929630       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:53.931 | E1213 03:42:53.931028       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:53.932 | E1213 03:42:53.932586       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:54.339 | time="2018-12-13T03:42:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:55.468 | time="2018-12-13T03:42:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:55.868 | time="2018-12-13T03:42:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:56.071 | time="2018-12-13T03:42:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:56.270 | time="2018-12-13T03:42:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:58.060 | time="2018-12-13T03:42:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:58.214 | E1213 03:42:58.214614       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:42:58.237 | time="2018-12-13T03:42:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:59.248 | time="2018-12-13T03:42:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:59.250 | time="2018-12-13T03:42:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:42:59.264 | time="2018-12-13T03:42:59Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:42:59.279 | time="2018-12-13T03:42:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:00.237 | time="2018-12-13T03:43:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:00.837 | time="2018-12-13T03:43:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:01.237 | time="2018-12-13T03:43:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:01.438 | time="2018-12-13T03:43:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:03.951 | E1213 03:43:03.951060       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.957 | E1213 03:43:03.955023       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.962 | E1213 03:43:03.962589       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.963 | E1213 03:43:03.962521       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.965 | E1213 03:43:03.965066       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:06.292 | time="2018-12-13T03:43:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:07.850 | time="2018-12-13T03:43:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:07.877 | time="2018-12-13T03:43:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.478 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.696 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.880 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.236 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.290 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.637 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.644 | time="2018-12-13T03:43:09Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:09.678 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.687 | time="2018-12-13T03:43:09Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:43:10.102 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.237 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.486 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.491 | time="2018-12-13T03:43:10Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:43:10.836 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.877 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.279 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.478 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.677 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.878 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.079 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.279 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.300 | time="2018-12-13T03:43:12Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:43:12.478 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.685 | time="2018-12-13T03:43:12Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:43:13.983 | E1213 03:43:13.983730       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.988 | E1213 03:43:13.988807       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.990 | E1213 03:43:13.990434       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.994 | E1213 03:43:13.994390       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:14.887 | E1213 03:43:14.887379       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:17.057 | time="2018-12-13T03:43:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:18.970 | time="2018-12-13T03:43:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:18.994 | time="2018-12-13T03:43:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:19.330 | time="2018-12-13T03:43:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:19.357 | time="2018-12-13T03:43:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:21.208 | E1213 03:43:21.208810       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:22.459 | E1213 03:43:22.458854       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:22.475 | time="2018-12-13T03:43:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:22.506 | time="2018-12-13T03:43:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.616 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.639 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.672 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.701 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.732 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:24.014 | E1213 03:43:24.014151       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.019 | E1213 03:43:24.018966       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.022 | E1213 03:43:24.022051       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.024 | E1213 03:43:24.024800       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:25.079 | time="2018-12-13T03:43:25Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:26.070 | time="2018-12-13T03:43:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:26.271 | time="2018-12-13T03:43:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:28.053 | time="2018-12-13T03:43:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:28.215 | E1213 03:43:28.214850       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:28.267 | time="2018-12-13T03:43:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.238 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.376 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.401 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:30.238 | time="2018-12-13T03:43:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:30.837 | time="2018-12-13T03:43:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.237 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.437 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.640 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.837 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.038 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.252 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.437 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:34.048 | E1213 03:43:34.048026       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.049 | E1213 03:43:34.048920       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.052 | E1213 03:43:34.052211       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.055 | E1213 03:43:34.055119       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:36.300 | time="2018-12-13T03:43:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:37.863 | time="2018-12-13T03:43:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:37.883 | time="2018-12-13T03:43:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:38.479 | time="2018-12-13T03:43:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:38.680 | time="2018-12-13T03:43:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.413 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.443 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.446 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.837 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.843 | time="2018-12-13T03:43:39Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:40.437 | time="2018-12-13T03:43:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:40.839 | time="2018-12-13T03:43:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.236 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.436 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.444 | time="2018-12-13T03:43:41Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:41.640 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.837 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:42.837 | E1213 03:43:42.837697       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:43:42.838 | time="2018-12-13T03:43:42Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:43:42.885 | time="2018-12-13T03:43:42Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:43:44.070 | E1213 03:43:44.070565       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.072 | E1213 03:43:44.072363       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.075 | E1213 03:43:44.074061       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.080 | E1213 03:43:44.079669       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.887 | E1213 03:43:44.887506       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:47.047 | time="2018-12-13T03:43:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.060 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.091 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.444 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.472 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:51.209 | E1213 03:43:51.209115       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:52.459 | E1213 03:43:52.459001       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:52.476 | time="2018-12-13T03:43:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:52.503 | time="2018-12-13T03:43:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:53.628 | time="2018-12-13T03:43:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:53.671 | time="2018-12-13T03:43:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:54.096 | E1213 03:43:54.095126       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.097 | E1213 03:43:54.097248       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.098 | E1213 03:43:54.098364       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.101 | E1213 03:43:54.100528       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.356 | time="2018-12-13T03:43:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.268 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.468 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.669 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.869 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.070 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.268 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.469 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.668 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.869 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.075 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.269 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.483 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:58.053 | time="2018-12-13T03:43:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:58.215 | E1213 03:43:58.214966       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:58.641 | time="2018-12-13T03:43:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.494 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.522 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.606 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.644 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.644 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:00.641 | time="2018-12-13T03:44:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:02.034 | time="2018-12-13T03:44:02Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:12Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:02Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:02.035 | time="2018-12-13T03:44:02Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:12Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:02Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:02.044 | time="2018-12-13T03:44:02Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:02.221 | E1213 03:43:13.984078       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.221 | E1213 03:44:02.221400       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.223 | E1213 03:44:02.223569       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.225 | E1213 03:44:02.224885       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.226 | E1213 03:44:02.226088       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.227 | E1213 03:44:02.227317       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

December 13th 2018, 04:43:00.837 | time="2018-12-13T03:43:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"
-- | --

  | December 13th 2018, 04:43:01.237 | time="2018-12-13T03:43:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:01.438 | time="2018-12-13T03:43:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:03.951 | E1213 03:43:03.951060       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.957 | E1213 03:43:03.955023       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.962 | E1213 03:43:03.962589       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.963 | E1213 03:43:03.962521       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:03.965 | E1213 03:43:03.965066       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:06.292 | time="2018-12-13T03:43:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:07.850 | time="2018-12-13T03:43:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:07.877 | time="2018-12-13T03:43:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.478 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.696 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:08.880 | time="2018-12-13T03:43:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.236 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.290 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.637 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.644 | time="2018-12-13T03:43:09Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:09.678 | time="2018-12-13T03:43:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:09.687 | time="2018-12-13T03:43:09Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:43:10.102 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.237 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.486 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.491 | time="2018-12-13T03:43:10Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:43:10.836 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:10.877 | time="2018-12-13T03:43:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.279 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.478 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.677 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:11.878 | time="2018-12-13T03:43:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.079 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.279 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.300 | time="2018-12-13T03:43:12Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:43:12.478 | time="2018-12-13T03:43:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:12.685 | time="2018-12-13T03:43:12Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:43:13.983 | E1213 03:43:13.983730       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.988 | E1213 03:43:13.988807       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.990 | E1213 03:43:13.990434       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:13.994 | E1213 03:43:13.994390       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:14.887 | E1213 03:43:14.887379       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:17.057 | time="2018-12-13T03:43:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:18.970 | time="2018-12-13T03:43:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:18.994 | time="2018-12-13T03:43:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:19.330 | time="2018-12-13T03:43:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:19.357 | time="2018-12-13T03:43:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:21.208 | E1213 03:43:21.208810       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:22.459 | E1213 03:43:22.458854       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:22.475 | time="2018-12-13T03:43:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:22.506 | time="2018-12-13T03:43:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.616 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.639 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.672 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.701 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:23.732 | time="2018-12-13T03:43:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:24.014 | E1213 03:43:24.014151       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.019 | E1213 03:43:24.018966       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.022 | E1213 03:43:24.022051       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:24.024 | E1213 03:43:24.024800       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:25.079 | time="2018-12-13T03:43:25Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:26.070 | time="2018-12-13T03:43:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:26.271 | time="2018-12-13T03:43:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:28.053 | time="2018-12-13T03:43:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:28.215 | E1213 03:43:28.214850       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:28.267 | time="2018-12-13T03:43:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.238 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.376 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:29.401 | time="2018-12-13T03:43:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:30.238 | time="2018-12-13T03:43:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:30.837 | time="2018-12-13T03:43:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.237 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.437 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.640 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:31.837 | time="2018-12-13T03:43:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.038 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.252 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:32.437 | time="2018-12-13T03:43:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:34.048 | E1213 03:43:34.048026       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.049 | E1213 03:43:34.048920       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.052 | E1213 03:43:34.052211       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:34.055 | E1213 03:43:34.055119       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:36.300 | time="2018-12-13T03:43:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:37.863 | time="2018-12-13T03:43:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:37.883 | time="2018-12-13T03:43:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:38.479 | time="2018-12-13T03:43:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:38.680 | time="2018-12-13T03:43:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.413 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.443 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.446 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.837 | time="2018-12-13T03:43:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:39.843 | time="2018-12-13T03:43:39Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:40.437 | time="2018-12-13T03:43:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:40.839 | time="2018-12-13T03:43:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.236 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.436 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.444 | time="2018-12-13T03:43:41Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:43:41.640 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:41.837 | time="2018-12-13T03:43:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:42.837 | E1213 03:43:42.837697       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:43:42.838 | time="2018-12-13T03:43:42Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:43:42.885 | time="2018-12-13T03:43:42Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:43:44.070 | E1213 03:43:44.070565       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.072 | E1213 03:43:44.072363       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.075 | E1213 03:43:44.074061       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.080 | E1213 03:43:44.079669       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:44.887 | E1213 03:43:44.887506       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:47.047 | time="2018-12-13T03:43:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.060 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.091 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.444 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:49.472 | time="2018-12-13T03:43:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:51.209 | E1213 03:43:51.209115       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:52.459 | E1213 03:43:52.459001       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:52.476 | time="2018-12-13T03:43:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:52.503 | time="2018-12-13T03:43:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:53.628 | time="2018-12-13T03:43:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:53.671 | time="2018-12-13T03:43:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:54.096 | E1213 03:43:54.095126       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.097 | E1213 03:43:54.097248       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.098 | E1213 03:43:54.098364       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.101 | E1213 03:43:54.100528       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:54.356 | time="2018-12-13T03:43:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.268 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.468 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.669 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:55.869 | time="2018-12-13T03:43:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.070 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.268 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.469 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.668 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:56.869 | time="2018-12-13T03:43:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.075 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.269 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:57.483 | time="2018-12-13T03:43:57Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:58.053 | time="2018-12-13T03:43:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:58.215 | E1213 03:43:58.214966       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:43:58.641 | time="2018-12-13T03:43:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.494 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.522 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.606 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.644 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:43:59.644 | time="2018-12-13T03:43:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:00.641 | time="2018-12-13T03:44:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:02.034 | time="2018-12-13T03:44:02Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:12Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:02Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:02.035 | time="2018-12-13T03:44:02Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:12Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:02Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:02.044 | time="2018-12-13T03:44:02Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:02.221 | E1213 03:43:13.984078       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.221 | E1213 03:44:02.221400       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.223 | E1213 03:44:02.223569       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.225 | E1213 03:44:02.224885       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.226 | E1213 03:44:02.226088       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.227 | E1213 03:44:02.227317       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:02.228 | E1213 03:44:02.228387       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:03.965 | time="2018-12-13T03:44:03Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:42Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:03Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:03.965 | time="2018-12-13T03:44:03Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:43:42Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:03Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:03.973 | time="2018-12-13T03:44:03Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:04.127 | E1213 03:44:04.127383       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:04.133 | E1213 03:44:04.133692       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:04.135 | E1213 03:44:04.135222       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:04.142 | E1213 03:44:04.135775       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:04.145 | E1213 03:44:04.132106       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:06.370 | time="2018-12-13T03:44:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:07.861 | time="2018-12-13T03:44:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:08.719 | time="2018-12-13T03:44:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:09.047 | time="2018-12-13T03:44:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:09.293 | time="2018-12-13T03:44:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:09.311 | time="2018-12-13T03:44:09Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:09.674 | time="2018-12-13T03:44:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:09.685 | time="2018-12-13T03:44:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:10.240 | time="2018-12-13T03:44:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:10.286 | time="2018-12-13T03:44:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:10.303 | time="2018-12-13T03:44:10Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:44:10.654 | time="2018-12-13T03:44:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:10.885 | time="2018-12-13T03:44:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:11.238 | time="2018-12-13T03:44:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:11.486 | time="2018-12-13T03:44:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:11.837 | time="2018-12-13T03:44:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:11.845 | time="2018-12-13T03:44:11Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:44:12.089 | time="2018-12-13T03:44:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:12.242 | time="2018-12-13T03:44:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:12.286 | time="2018-12-13T03:44:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:12.637 | time="2018-12-13T03:44:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:12.685 | time="2018-12-13T03:44:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.038 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.057 | time="2018-12-13T03:44:13Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:44:13.092 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.118 | time="2018-12-13T03:44:13Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:13.437 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.444 | time="2018-12-13T03:44:13Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:44:13.486 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.726 | time="2018-12-13T03:44:13Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:13.755 | time="2018-12-13T03:44:13Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:44:13Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:13Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:13.762 | time="2018-12-13T03:44:13Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:44:13Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:13Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:13.770 | time="2018-12-13T03:44:13Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:13.839 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:13.887 | time="2018-12-13T03:44:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:14.150 | E1213 03:44:14.150532       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:14.152 | E1213 03:44:14.151814       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:14.155 | E1213 03:44:14.155111       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:14.156 | E1213 03:44:14.156536       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:14.157 | E1213 03:44:14.156802       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:14.305 | time="2018-12-13T03:44:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:14.439 | time="2018-12-13T03:44:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:14.490 | time="2018-12-13T03:44:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:14.503 | time="2018-12-13T03:44:14Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:14.887 | E1213 03:44:14.887619       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:15.038 | time="2018-12-13T03:44:15Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:15.102 | time="2018-12-13T03:44:15Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:15.642 | time="2018-12-13T03:44:15Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:15.694 | time="2018-12-13T03:44:15Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.041 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.095 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.107 | time="2018-12-13T03:44:16Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:16.437 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.485 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.637 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.703 | time="2018-12-13T03:44:16Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:16.726 | time="2018-12-13T03:44:16Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:17.038 | time="2018-12-13T03:44:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:17.106 | time="2018-12-13T03:44:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:17.238 | time="2018-12-13T03:44:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:18.089 | time="2018-12-13T03:44:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:18.485 | time="2018-12-13T03:44:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:18.704 | time="2018-12-13T03:44:18Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:19.110 | time="2018-12-13T03:44:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:19.238 | time="2018-12-13T03:44:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:19.886 | time="2018-12-13T03:44:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:20.486 | time="2018-12-13T03:44:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:20.885 | time="2018-12-13T03:44:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:21.209 | E1213 03:44:21.209239       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:21.485 | time="2018-12-13T03:44:21Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:22.177 | E1213 03:44:22.177126       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:22.459 | E1213 03:44:22.459250       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:22.899 | time="2018-12-13T03:44:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:23.895 | time="2018-12-13T03:44:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:24.178 | E1213 03:44:24.178763       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:24.179 | E1213 03:44:24.179462       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:24.180 | E1213 03:44:24.179870       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:24.183 | E1213 03:44:24.183415       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:24.192 | E1213 03:44:24.192424       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:24.939 | time="2018-12-13T03:44:24Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:24.955 | time="2018-12-13T03:44:24Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:26.687 | time="2018-12-13T03:44:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:28.047 | time="2018-12-13T03:44:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:28.086 | time="2018-12-13T03:44:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:28.215 | E1213 03:44:28.215374       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:28.486 | time="2018-12-13T03:44:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:28.494 | time="2018-12-13T03:44:28Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:44:28.842 | time="2018-12-13T03:44:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:28.887 | time="2018-12-13T03:44:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:29.302 | time="2018-12-13T03:44:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:29.693 | time="2018-12-13T03:44:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:29.841 | time="2018-12-13T03:44:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:30.087 | time="2018-12-13T03:44:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:30.439 | time="2018-12-13T03:44:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:30.697 | time="2018-12-13T03:44:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:31.056 | time="2018-12-13T03:44:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:31.102 | time="2018-12-13T03:44:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:31.664 | time="2018-12-13T03:44:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:31.686 | time="2018-12-13T03:44:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:31.885 | time="2018-12-13T03:44:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:32.239 | time="2018-12-13T03:44:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:32.285 | time="2018-12-13T03:44:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:32.641 | time="2018-12-13T03:44:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:32.686 | time="2018-12-13T03:44:32Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.040 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.092 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.446 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.486 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.876 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:33.891 | time="2018-12-13T03:44:33Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:34.207 | E1213 03:44:34.207484       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:34.210 | E1213 03:44:34.209773       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:34.214 | E1213 03:44:34.214004       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:34.215 | E1213 03:44:34.215706       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:34.216 | E1213 03:44:34.215960       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:34.241 | time="2018-12-13T03:44:34Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:34.455 | time="2018-12-13T03:44:34Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:34.840 | time="2018-12-13T03:44:34Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:35.239 | time="2018-12-13T03:44:35Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:36.288 | time="2018-12-13T03:44:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:37.853 | time="2018-12-13T03:44:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:38.285 | time="2018-12-13T03:44:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:39.252 | time="2018-12-13T03:44:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:39.285 | time="2018-12-13T03:44:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:39.641 | time="2018-12-13T03:44:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:39.713 | time="2018-12-13T03:44:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:40.038 | time="2018-12-13T03:44:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:40.086 | time="2018-12-13T03:44:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:40.638 | time="2018-12-13T03:44:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:40.686 | time="2018-12-13T03:44:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:41.240 | time="2018-12-13T03:44:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:43.871 | E1213 03:44:43.870848       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:44:43.871 | time="2018-12-13T03:44:43Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:44:43.920 | time="2018-12-13T03:44:43Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:43.943 | time="2018-12-13T03:44:43Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:44:43Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:43Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:43.944 | time="2018-12-13T03:44:43Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:44:43Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:44:43Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:44:43.961 | time="2018-12-13T03:44:43Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:44:44.233 | E1213 03:44:44.232914       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:44.237 | E1213 03:44:44.237106       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:44.240 | E1213 03:44:44.239965       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:44.241 | E1213 03:44:44.241206       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:44.241 | E1213 03:44:44.241314       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:44.888 | E1213 03:44:44.887912       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:47.055 | time="2018-12-13T03:44:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:47.109 | time="2018-12-13T03:44:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.297 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.323 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.348 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.378 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.403 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.426 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.735 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.779 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.804 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.829 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:49.947 | time="2018-12-13T03:44:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:51.209 | E1213 03:44:51.209358       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:52.177 | E1213 03:44:52.177293       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:52.459 | E1213 03:44:52.459361       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:52.474 | time="2018-12-13T03:44:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:52.497 | time="2018-12-13T03:44:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:53.626 | time="2018-12-13T03:44:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:53.746 | time="2018-12-13T03:44:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:54.256 | E1213 03:44:54.256737       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:54.262 | E1213 03:44:54.262041       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:54.264 | E1213 03:44:54.263824       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:54.265 | E1213 03:44:54.265306       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:54.265 | E1213 03:44:54.264788       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:54.338 | time="2018-12-13T03:44:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:55.564 | time="2018-12-13T03:44:55Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:56.367 | time="2018-12-13T03:44:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:56.881 | time="2018-12-13T03:44:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:56.946 | time="2018-12-13T03:44:56Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:58.055 | time="2018-12-13T03:44:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:58.215 | E1213 03:44:58.215450       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:44:58.438 | time="2018-12-13T03:44:58Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:59.438 | time="2018-12-13T03:44:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:59.776 | time="2018-12-13T03:44:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:44:59.822 | time="2018-12-13T03:44:59Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:00.439 | time="2018-12-13T03:45:00Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:01.038 | time="2018-12-13T03:45:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:01.440 | time="2018-12-13T03:45:01Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:04.299 | E1213 03:45:04.299260       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:04.300 | E1213 03:45:04.299995       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:04.300 | E1213 03:45:04.300495       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:04.301 | E1213 03:45:04.301637       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:04.303 | E1213 03:45:04.303752       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:06.307 | time="2018-12-13T03:45:06Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:07.852 | time="2018-12-13T03:45:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:07.883 | time="2018-12-13T03:45:07Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:08.483 | time="2018-12-13T03:45:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:08.699 | time="2018-12-13T03:45:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:08.883 | time="2018-12-13T03:45:08Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:09.106 | time="2018-12-13T03:45:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:09.250 | time="2018-12-13T03:45:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:09.297 | time="2018-12-13T03:45:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:09.667 | time="2018-12-13T03:45:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:09.700 | time="2018-12-13T03:45:09Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:10.042 | time="2018-12-13T03:45:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:10.083 | time="2018-12-13T03:45:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:10.643 | time="2018-12-13T03:45:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:10.682 | time="2018-12-13T03:45:10Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:11.247 | time="2018-12-13T03:45:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:11.639 | time="2018-12-13T03:45:11Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:12.040 | time="2018-12-13T03:45:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:12.239 | time="2018-12-13T03:45:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:12.439 | time="2018-12-13T03:45:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:12.639 | time="2018-12-13T03:45:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:12.646 | time="2018-12-13T03:45:12Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:45:12.840 | time="2018-12-13T03:45:12Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:13.039 | time="2018-12-13T03:45:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:13.045 | time="2018-12-13T03:45:13Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:45:13.239 | time="2018-12-13T03:45:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:13.439 | time="2018-12-13T03:45:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:13.641 | time="2018-12-13T03:45:13Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:14.039 | time="2018-12-13T03:45:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:14.074 | time="2018-12-13T03:45:14Z" level=warning msg="Error syncing Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:45:14.239 | time="2018-12-13T03:45:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:14.326 | E1213 03:45:14.326232       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:14.327 | E1213 03:45:14.327103       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:14.328 | E1213 03:45:14.327656       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:14.328 | E1213 03:45:14.328232       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:14.346 | E1213 03:45:14.344978       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:14.504 | time="2018-12-13T03:45:14Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:45:14.523 | time="2018-12-13T03:45:14Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:45:14Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:45:14Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:45:14.524 | time="2018-12-13T03:45:14Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:45:14Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:45:14Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:45:14.539 | time="2018-12-13T03:45:14Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:45:14.639 | time="2018-12-13T03:45:14Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:14.888 | E1213 03:45:14.888224       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:15.039 | time="2018-12-13T03:45:15Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:17.049 | time="2018-12-13T03:45:17Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.589 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.642 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.856 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.903 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.921 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.945 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.975 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:19.997 | time="2018-12-13T03:45:19Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:20.028 | time="2018-12-13T03:45:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:20.066 | time="2018-12-13T03:45:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:20.097 | time="2018-12-13T03:45:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:20.174 | time="2018-12-13T03:45:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:20.375 | time="2018-12-13T03:45:20Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:21.209 | E1213 03:45:21.209710       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:21.732 | time="2018-12-13T03:45:21Z" level=debug msg="Requeue longhorn-system/worker5 due to conflict"

  | December 13th 2018, 04:45:22.177 | E1213 03:45:22.177414       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:22.459 | E1213 03:45:22.459575       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:22.475 | time="2018-12-13T03:45:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:22.972 | time="2018-12-13T03:45:22Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:23.375 | time="2018-12-13T03:45:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:23.575 | time="2018-12-13T03:45:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:23.772 | time="2018-12-13T03:45:23Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:24.359 | E1213 03:45:24.358958       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:24.359 | E1213 03:45:24.359653       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:24.359 | E1213 03:45:24.359307       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:24.359 | E1213 03:45:24.359464       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:24.360 | E1213 03:45:24.359990       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:24.773 | time="2018-12-13T03:45:24Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:26.173 | time="2018-12-13T03:45:26Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:26.180 | time="2018-12-13T03:45:26Z" level=debug msg="Requeue longhorn-system/worker2 due to conflict"

  | December 13th 2018, 04:45:27.372 | time="2018-12-13T03:45:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:27.574 | time="2018-12-13T03:45:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:27.773 | time="2018-12-13T03:45:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:27.973 | time="2018-12-13T03:45:27Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:28.054 | time="2018-12-13T03:45:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:28.215 | E1213 03:45:28.215684       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:28.641 | time="2018-12-13T03:45:28Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:29.642 | time="2018-12-13T03:45:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:29.883 | time="2018-12-13T03:45:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:29.947 | time="2018-12-13T03:45:29Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:30.439 | time="2018-12-13T03:45:30Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:31.039 | time="2018-12-13T03:45:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:31.638 | time="2018-12-13T03:45:31Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:34.370 | E1213 03:45:34.370261       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:34.374 | E1213 03:45:34.374749       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:34.375 | E1213 03:45:34.375151       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:34.375 | E1213 03:45:34.375481       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:34.377 | E1213 03:45:34.376820       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:36.309 | time="2018-12-13T03:45:36Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:37.861 | time="2018-12-13T03:45:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:37.882 | time="2018-12-13T03:45:37Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:38.484 | time="2018-12-13T03:45:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:38.884 | time="2018-12-13T03:45:38Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:39.082 | time="2018-12-13T03:45:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:39.283 | time="2018-12-13T03:45:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:39.444 | time="2018-12-13T03:45:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:39.485 | time="2018-12-13T03:45:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:39.841 | time="2018-12-13T03:45:39Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:40.241 | time="2018-12-13T03:45:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:40.283 | time="2018-12-13T03:45:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:40.839 | time="2018-12-13T03:45:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:40.919 | time="2018-12-13T03:45:40Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:41.439 | time="2018-12-13T03:45:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:41.445 | time="2018-12-13T03:45:41Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:45:41.703 | time="2018-12-13T03:45:41Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:42.040 | time="2018-12-13T03:45:42Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:42.089 | time="2018-12-13T03:45:42Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:42.483 | time="2018-12-13T03:45:42Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:42.640 | time="2018-12-13T03:45:42Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:43.041 | time="2018-12-13T03:45:43Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:43.240 | time="2018-12-13T03:45:43Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:43.245 | time="2018-12-13T03:45:43Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:45:43.440 | time="2018-12-13T03:45:43Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:43.640 | time="2018-12-13T03:45:43Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:44.039 | time="2018-12-13T03:45:44Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:44.047 | time="2018-12-13T03:45:44Z" level=debug msg="Requeue longhorn-system/worker3 due to conflict"

  | December 13th 2018, 04:45:44.399 | E1213 03:45:44.399575       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:44.400 | E1213 03:45:44.400373       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:44.400 | E1213 03:45:44.400348       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:44.401 | E1213 03:45:44.401388       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:44.406 | E1213 03:45:44.406585       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:44.439 | time="2018-12-13T03:45:44Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:44.641 | time="2018-12-13T03:45:44Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:44.652 | E1213 03:45:44.651910       1 engine_controller.go:206] fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition

  | December 13th 2018, 04:45:44.652 | time="2018-12-13T03:45:44Z" level=warning msg="Dropping Longhorn engine longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436 out of the queue: fail to sync engine for longhorn-system/pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: fail to start rebuild for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436: timed out waiting for the condition"

  | December 13th 2018, 04:45:44.859 | time="2018-12-13T03:45:44Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:44.888 | E1213 03:45:44.888363       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:45.081 | time="2018-12-13T03:45:45Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7 with IP 10.42.5.50 for pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:45:45.098 | time="2018-12-13T03:45:45Z" level=error msg="Failed rebuilding 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:45:45Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:45:45Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:45:45.098 | time="2018-12-13T03:45:45Z" level=info msg="Event(v1.ObjectReference{Kind:"Engine", Namespace:"longhorn-system", Name:"pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-e-bfd14436", UID:"3888fdd3-b86b-11e8-a13b-8e7ee66bf8b9", APIVersion:"longhorn.rancher.io/v1alpha1", ResourceVersion:"60225191", FieldPath:""}): type: 'Warning' reason: 'FailedRebuilding' Failed rebuilding replica with IP 10.42.5.50: failed to add replica address='tcp://10.42.5.50:9502' to controller 'pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9': Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.4.88:9501 add tcp://10.42.5.50:9502], output time="2018-12-13T03:45:45Z" level=info msg="Adding replica tcp://10.42.5.50:9502 in WO mode"  time="2018-12-13T03:45:45Z" level=fatal msg="Error running add replica command: Bad response: 500 500 Internal Server Error: {\"actions\":{},\"code\":\"Server Error\",\"detail\":\"\",\"links\":{\"self\":\"http://10.42.4.88:9501/v1/replicas\"},\"message\":\"Replica must be closed, Can not add in state: open\",\"status\":500,\"type\":\"error\"}\n"  , error exit status 1"

  | December 13th 2018, 04:45:45.110 | time="2018-12-13T03:45:45Z" level=error msg="Removed failed rebuilding replica 10.42.5.50 of pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9"

  | December 13th 2018, 04:45:47.057 | time="2018-12-13T03:45:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:47.094 | time="2018-12-13T03:45:47Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:49.697 | time="2018-12-13T03:45:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:49.715 | time="2018-12-13T03:45:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:49.965 | time="2018-12-13T03:45:49Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:50.027 | time="2018-12-13T03:45:50Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:50.106 | time="2018-12-13T03:45:50Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:50.148 | time="2018-12-13T03:45:50Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:50.185 | time="2018-12-13T03:45:50Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:50.208 | time="2018-12-13T03:45:50Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:51.209 | E1213 03:45:51.209716       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:52.177 | E1213 03:45:52.177683       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:52.459 | E1213 03:45:52.459713       1 node_controller.go:510] Couldn't get node master1: node.longhorn.rancher.io "master1" not found

  | December 13th 2018, 04:45:52.475 | time="2018-12-13T03:45:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:52.502 | time="2018-12-13T03:45:52Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:53.625 | time="2018-12-13T03:45:53Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:54.080 | time="2018-12-13T03:45:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

  | December 13th 2018, 04:45:54.276 | time="2018-12-13T03:45:54Z" level=debug msg="Use pid 1 namespace for initiator: Failed to find the ancestor process: dockerd"

Is there a way to automatically reattach the volume in rw? The only way I found out to get it working again is to manually delete the application pod and therefore the volume is remounted.

[courcelm]

I also experienced this issue. It works fine for the day and the next morning, the application crashed.

[yasker]

@db-adrian Manager reports cannot find node master1, which may be the cause of the problem. Is anything abnormal about master1? Also, the engine reports fail to rebuild due to replica state is open, which can be caused by unstable network.

@courcelm Can you provide the manager log? We can start working from there. There can be different symptoms.

[adrianNEMO]

master1 does not have the worker role only controlplane and etcd. That's the only thing different from the other nodes. I thought this was because on master1 is no manager. The rebuild is hanging for quite some time. It is the only volume and I don't have network issues on any other volumes or applications.

[yasker]

@db-adrian The log doesn't show volume failure...

The rebuilding failure is probably due to the first time rebuild failure, but the replica wasn't deleted so it's remain in open and not usable state. That would be a bug. We will investigate that. And it should be worked around by deleting the replica manually (in this case, pvc-3064fe2d-b86b-11e8-a13b-8e7ee66bf8b9-r-f1b5bfe7)

[courcelm]

Here is the read only error for a Redis container: 1:M 18 Dec 11:41:39.083 * Background saving started by pid 14691 14691:C 18 Dec 11:41:39.083 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Read-only file system 1:M 18 Dec 11:41:39.183 # Background saving error

Here is the tail of the longhorn manager:

time="2018-12-15T10:37:09Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:37:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595491\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595491\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595497\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595497\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595500\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595500\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595506\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595506\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595511\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595511\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595514\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595514\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595515\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595515\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595517\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595517\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595519\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595519\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595515\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595515\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595527\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595527\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595536\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595536\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595537\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595537\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595539\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595539\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595541\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595542\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595541\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595542\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595544\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595544\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595549\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595550\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595549\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595550\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595553\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595557\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595553\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595558\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595557\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595565\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595558\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595566\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595565\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595565\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595566\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595565\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:19Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:37:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595577\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595578\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595577\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:20Z" level=debug msg="Requeue longhorn-system/tp88-i due to conflict" time="2018-12-15T10:37:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595578\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:20Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:37:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595585\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595591\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595585\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:21Z" level=debug msg="Requeue longhorn-system/tp88-i due to conflict" time="2018-12-15T10:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595592\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595591\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595601\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595592\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp88-i not ready: KubeletNotReady" time="2018-12-15T10:37:22Z" level=debug msg="Requeue longhorn-system/tp88-i due to conflict" time="2018-12-15T10:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp88-i\", UID:\"db9f5f28-dd1b-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595602\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp88-i is ready" time="2018-12-15T10:37:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595601\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:23Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:37:24Z" level=debug msg="Requeue longhorn-system/tp88-i due to conflict" time="2018-12-15T10:37:26Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595636\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595636\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:27Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:37:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595711\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595711\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595736\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595736\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595756\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595756\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595762\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595762\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595763\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595763\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595767\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595767\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595767\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595767\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595774\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595774\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595775\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595775\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595783\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595783\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:50Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595792\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:50Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595792\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595794\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595794\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595796\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595796\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595804\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595804\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595804\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595804\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595826\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:37:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595826\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:37:57Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:38:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595888\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595888\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595917\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595917\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:11Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:38:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595919\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595919\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595929\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595929\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595939\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595939\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595948\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595948\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595950\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595950\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595956\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595956\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595958\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595958\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595959\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595959\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595959\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595959\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595966\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595966\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595988\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:20Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595988\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595990\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595990\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595991\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595991\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595999\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24595999\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596001\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:25Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596001\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:25Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:38:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596087\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596087\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:40Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596121\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:40Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596121\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596128\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596128\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596134\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596134\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596158\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596158\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596162\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596162\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596163\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596163\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596165\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596165\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596165\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596165\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596171\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596171\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596172\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596172\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596181\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596181\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:50Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596185\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:50Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596185\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596190\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596190\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596192\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596192\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596192\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596192\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:53Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:38:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596224\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:38:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596224\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:38:57Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:39:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596272\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596272\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596300\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596300\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596308\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596308\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596314\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596314\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596316\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596316\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596329\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596329\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596330\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596330\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596349\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596349\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596351\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596351\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596355\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596355\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596359\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596359\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596362\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596362\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:17Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-15T10:39:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596364\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596364\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-15T10:39:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24596364\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-15T10:39:18Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T11:00:03Z" level=warning msg="HTTP handling error fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T11:00:03Z" level=error msg="Error in request: fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T11:00:03Z" level=warning msg="HTTP handling error fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T11:00:03Z" level=error msg="Error in request: fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T11:00:03Z" level=warning msg="HTTP handling error fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T11:00:03Z" level=error msg="Error in request: fail to list snapshot: error listing snapshot: Failed to execute: /var/lib/rancher/longhorn/engine-binaries/rancher-longhorn-engine-v0.3.0/longhorn [--url http://10.42.9.249:9501 snapshot info], output time=\"2018-12-17T11:00:03Z\" level=fatal msg=\"Error running snapshot info command: BUG: Inconsistent snapshot info: {Name:c-hc8zjg-51a70570 Parent:c-hc8zjg-913528e4 Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:585728 Labels:map[RecurringJob:c-hc8zjg]} vs {Name:c-hc8zjg-51a70570 Parent: Children:map[c-hc8zjg-8ac36ec1:{}] Removed:false UserCreated:true Created:2018-12-17T00:00:02Z Size:54112256 Labels:map[RecurringJob:c-hc8zjg]}\" \n, error exit status 1" time="2018-12-17T13:40:24Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-17T13:40:25Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-f94cdf42\", UID:\"4bfd928e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529565\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-r-f94cdf42" time="2018-12-17T13:40:25Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-846b2bf9\", UID:\"4bfedeec-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529567\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-r-846b2bf9" time="2018-12-17T13:40:26Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-b4a86f52\", UID:\"4c01aad9-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529568\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-r-b4a86f52" time="2018-12-17T13:40:28Z" level=debug msg="Instance pvc-4998dc62020111e9-r-846b2bf9 starts running, IP 10.42.3.55" time="2018-12-17T13:40:28Z" level=debug msg="Instance pvc-4998dc62020111e9-r-f94cdf42 starts running, IP 10.42.7.104" time="2018-12-17T13:40:28Z" level=debug msg="Instance pvc-4998dc62020111e9-r-b4a86f52 starts running, IP 10.42.6.106" time="2018-12-17T13:40:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529656\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-e-01cd1ba1" time="2018-12-17T13:40:37Z" level=debug msg="Instance pvc-4998dc62020111e9-e-01cd1ba1 starts running, IP 10.42.9.116" time="2018-12-17T13:40:38Z" level=debug msg="Start monitoring pvc-4998dc62020111e9-e-01cd1ba1" time="2018-12-17T13:40:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529569\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume pvc-4998dc62020111e9 has been attached to tp93-i" time="2018-12-17T13:40:43Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-17T14:33:08Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T14:33:09Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T14:33:10Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T14:33:11Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T14:33:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T15:41:27Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T16:26:56Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T16:27:26Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T18:05:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T18:15:15Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T18:15:17Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T18:15:17Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T18:31:26Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:03:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:45:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:31Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653437\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653437\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653459\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653459\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:33Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653475\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653475\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:34Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:35Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653503\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653503\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:36Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653516\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653516\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:37Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:37Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653528\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653528\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:38Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653550\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653550\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:39Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653566\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653566\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:39Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653583\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653583\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:41Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653596\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653596\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:42Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653630\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653630\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:44Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653649\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653649\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:46Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:46Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653685\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653685\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:48Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:48Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653709\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653709\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653709\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:50Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653709\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:50Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653760\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:52Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653760\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:52Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:53Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653769\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:54Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653769\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:54Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:48:56Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653815\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:48:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653815\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:48:59Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653870\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653870\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653887\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653887\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653903\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653903\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653909\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653909\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653917\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653917\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653917\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653917\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653924\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653924\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653925\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653925\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653926\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:10Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653926\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653934\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653934\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653935\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653935\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653942\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653942\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653945\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653945\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653948\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:13Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653948\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653957\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653957\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653958\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653958\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653961\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653961\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653963\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:15Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653963\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653972\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653972\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653973\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653973\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653976\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653976\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653977\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653977\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653983\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653983\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653983\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653983\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:19Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653995\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653995\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653996\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:19Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25653996\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654010\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654010\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654016\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654016\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654032\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654032\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:23Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654037\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654037\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654037\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654037\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:24Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:26Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654052\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:28Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654052\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:28Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654092\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:33Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654092\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:49:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:49:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654127\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp93-i is ready" time="2018-12-17T19:49:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp93-i\", UID:\"fc1f6d54-dd1c-11e8-924f-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25654127\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp93-i not ready: KubeletNotReady" time="2018-12-17T19:53:02Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T19:58:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T20:52:25Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T21:27:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T22:07:05Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T22:22:25Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T22:51:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T22:53:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-17T23:19:32Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T00:11:32Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T00:31:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T00:31:06Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T01:15:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:17:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:22:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:29:36Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:35:04Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:56:33Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T02:58:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T03:00:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T03:01:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T03:51:36Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T03:51:38Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T03:54:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:00:07Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:13:34Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:23:04Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:28:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:43:04Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T04:57:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T05:49:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T06:03:03Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" E1218 06:35:55.763130 1 volume_controller.go:196] fail to sync longhorn-system/pvc-4998dc62020111e9: fail to update recurring jobs for pvc-4998dc62020111e9: Timeout: request did not complete within requested timeout 30s time="2018-12-18T06:35:55Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-4998dc62020111e9 out of the queue: fail to sync longhorn-system/pvc-4998dc62020111e9: fail to update recurring jobs for pvc-4998dc62020111e9: Timeout: request did not complete within requested timeout 30s" E1218 06:35:55.763169 1 volume_controller.go:196] fail to sync longhorn-system/pvc-8e23b9a0ffeb11e8: fail to update recurring jobs for pvc-8e23b9a0ffeb11e8: Timeout: request did not complete within requested timeout 30s time="2018-12-18T06:35:55Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-8e23b9a0ffeb11e8 out of the queue: fail to sync longhorn-system/pvc-8e23b9a0ffeb11e8: fail to update recurring jobs for pvc-8e23b9a0ffeb11e8: Timeout: request did not complete within requested timeout 30s" E1218 06:36:22.883682 1 node_controller.go:289] fail to sync node for longhorn-system/tp93-i: rpc error: code = Unavailable desc = transport is closing time="2018-12-18T06:36:22Z" level=warning msg="Dropping Longhorn node longhorn-system/tp93-i out of the queue: fail to sync node for longhorn-system/tp93-i: rpc error: code = Unavailable desc = transport is closing" time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-i\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-i is ready" time="2018-12-18T06:36:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-i\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-i not ready: NodeStatusUnknown" time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376401\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica pvc-8e23b9a0ffeb11e8-r-903cf5dc (10.42.6.105) in error" time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376699\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-8e23b9a0ffeb11e8 became degraded" time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-3702d244 due to conflict" time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc due to conflict" time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376699\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-8e23b9a0ffeb11e8 became degraded" time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8 due to conflict" time="2018-12-18T06:37:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-r-903cf5dc\", UID:\"9441fa7f-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858385\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-8e23b9a0ffeb11e8-r-903cf5dc" time="2018-12-18T06:37:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-r-3702d244\", UID:\"540bbd4f-028f-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858392\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-8e23b9a0ffeb11e8-r-3702d244" time="2018-12-18T06:37:05Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc due to conflict" time="2018-12-18T06:37:06Z" level=debug msg="Instance pvc-8e23b9a0ffeb11e8-r-3702d244 starts running, IP 10.42.3.62" time="2018-12-18T06:37:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-8e23b9a0ffeb11e8-r-3702d244 with IP 10.42.3.62 for pvc-8e23b9a0ffeb11e8" time="2018-12-18T06:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica UNKNOWN-10.42.6.105 (10.42.6.105) in error" time="2018-12-18T06:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Detected rebuilding replica pvc-8e23b9a0ffeb11e8-r-3702d244 (10.42.3.62)" time="2018-12-18T06:37:18Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" time="2018-12-18T06:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858495\", FieldPath:\"\"}): type: 'Normal' reason: 'Delete' Removed unknown replica IP 10.42.6.105 from engine" time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529751\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica pvc-4998dc62020111e9-r-b4a86f52 (10.42.6.106) in error" time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" time="2018-12-18T06:37:21Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" time="2018-12-18T06:37:21Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-b4a86f52\", UID:\"4c01aad9-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858575\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4998dc62020111e9-r-b4a86f52" time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-c6c69ca8\", UID:\"5f779b2f-028f-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858577\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-r-c6c69ca8" time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-c6c69ca8 due to conflict" time="2018-12-18T06:37:23Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" time="2018-12-18T06:37:23Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" time="2018-12-18T06:37:24Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-18T06:37:24Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" time="2018-12-18T06:37:26Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" time="2018-12-18T06:37:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Replica pvc-8e23b9a0ffeb11e8-r-3702d244 with IP 10.42.3.62 has been rebuilded for volume pvc-8e23b9a0ffeb11e8" time="2018-12-18T06:37:39Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" time="2018-12-18T06:37:43Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858591\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Detected replica pvc-8e23b9a0ffeb11e8-r-3702d244 (10.42.3.62) has been rebuilded" time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858388\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-8e23b9a0ffeb11e8 became healthy" time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858388\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-8e23b9a0ffeb11e8 became healthy" time="2018-12-18T06:37:43Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8 due to conflict" time="2018-12-18T06:37:44Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-i due to conflict" time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-i due to conflict" time="2018-12-18T06:38:56Z" level=debug msg="Instance pvc-4998dc62020111e9-r-c6c69ca8 starts running, IP 10.42.4.17" time="2018-12-18T06:38:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-4998dc62020111e9-r-c6c69ca8 with IP 10.42.4.17 for pvc-4998dc62020111e9" time="2018-12-18T06:39:00Z" level=debug msg="Requeue longhorn-system/tp86-i due to conflict" time="2018-12-18T06:39:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica UNKNOWN-10.42.6.106 (10.42.6.106) in error" time="2018-12-18T06:39:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Detected rebuilding replica pvc-4998dc62020111e9-r-c6c69ca8 (10.42.4.17)" time="2018-12-18T06:39:05Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" time="2018-12-18T06:39:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859387\", FieldPath:\"\"}): type: 'Normal' reason: 'Delete' Removed unknown replica IP 10.42.6.106 from engine" time="2018-12-18T06:39:08Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" time="2018-12-18T06:39:10Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" time="2018-12-18T06:39:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T06:39:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Replica pvc-4998dc62020111e9-r-c6c69ca8 with IP 10.42.4.17 has been rebuilded for volume pvc-4998dc62020111e9" time="2018-12-18T06:39:27Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859483\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Detected replica pvc-4998dc62020111e9-r-c6c69ca8 (10.42.4.17) has been rebuilded" time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858579\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-4998dc62020111e9 became healthy" time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858579\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-4998dc62020111e9 became healthy" time="2018-12-18T06:39:27Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" time="2018-12-18T06:47:43Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T07:07:15Z" level=info msg="Cleaning up corrupted or staled replica pvc-8e23b9a0ffeb11e8-r-903cf5dc" time="2018-12-18T07:07:15Z" level=info msg="Longhorn replica longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc has been deleted" time="2018-12-18T07:09:15Z" level=info msg="Cleaning up corrupted or staled replica pvc-4998dc62020111e9-r-b4a86f52" time="2018-12-18T07:09:16Z" level=info msg="Longhorn replica longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 has been deleted" time="2018-12-18T07:16:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:00:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:04:16Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:04:19Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:04:20Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:13:55Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:14:56Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:15:26Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:19:16Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:21:26Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:35:37Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:54:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:55:59Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:57:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T08:59:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:04:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:13:31Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:19:42Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:30:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:54:15Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:54:18Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:56:43Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T09:56:43Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:00:43Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:10:56Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:19:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:34:14Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:44:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:52:12Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:54:01Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:54:02Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T10:54:04Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T11:00:13Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict" time="2018-12-18T11:33:14Z" level=debug msg="Requeue longhorn-system/tp93-i due to conflict"

[yasker]

@courcelm I also didn't see the engine crash from your log. And I noticed that node tp93-i seems had an issue. It reports KubeletNotReady constantly. And there is one other report for tp87-i as well. Is there anything else happens during the period of Redis failure?

[courcelm]

Not sure exactly where to look at. Can you guide me?

First Redis error at 18 Dec 06:36:51.440. There are timeout errors above starting at E1218 06:35:55.763130

I found some longhorn log in kubelet log (see below), I don't know if it helps. I think this is related to my hourly snapshot of the volume.

I1218 00:00:05.733254 25178 kubelet.go:1883] SyncLoop (ADD, "api"): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)" I1218 00:00:05.743791 25178 kubelet.go:1883] SyncLoop (ADD, "api"): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" I1218 00:00:05.815077 25178 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:05.815150 25178 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:05.815369 25178 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.815415 25178 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.915787 25178 reconciler.go:252] operationExecutor.MountVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:05.915922 25178 reconciler.go:252] operationExecutor.MountVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.915983 25178 reconciler.go:252] operationExecutor.MountVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.916041 25178 operation_generator.go:568] MountVolume.SetUp succeeded for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.916088 25178 reconciler.go:252] operationExecutor.MountVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:05.916166 25178 operation_generator.go:568] MountVolume.SetUp succeeded for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:05.922257 25178 operation_generator.go:568] MountVolume.SetUp succeeded for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:05.923049 25178 operation_generator.go:568] MountVolume.SetUp succeeded for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:06.043404 25178 kuberuntime_manager.go:396] No sandbox for pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)" can be found. Need to start a new one I1218 00:00:06.060740 25178 kuberuntime_manager.go:396] No sandbox for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" can be found. Need to start a new one E1218 00:00:06.364748 25178 kubelet_volumes.go:140] Orphaned pod "cc89f584-fa4e-11e8-b692-000c298019f4" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. Calico CNI fetching podCidr from Kubernetes Calico CNI fetching podCidr from Kubernetes Calico CNI passing podCidr to host-local IPAM: 10.42.9.0/24 Calico CNI passing podCidr to host-local IPAM: 10.42.9.0/24 Calico CNI using IPs: [10.42.9.183/32] Calico CNI using IPs: [10.42.9.184/32] I1218 00:00:06.612902 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e0224f74-0257-11e9-82fa-080027249e2f", Type:"ContainerStarted", Data:"67d8d3d2cb59654bb270147a305d9a3fb446a0a167c85f40ae9f2ca88e36f46e"} I1218 00:00:06.617179 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e023bf09-0257-11e9-82fa-080027249e2f", Type:"ContainerStarted", Data:"c0985e36f74b1d1595ecfe41a6f0de4fc8792fd5c7aced29bb71226b2aa86a5b"} I1218 00:00:06.621073 25178 kubelet.go:1890] SyncLoop (UPDATE, "api"): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)" I1218 00:00:06.664887 25178 kubelet.go:1890] SyncLoop (UPDATE, "api"): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" I1218 00:00:07.639749 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e023bf09-0257-11e9-82fa-080027249e2f", Type:"ContainerStarted", Data:"09dc27a7747f33448384840bbfeaab47850406ed76d8257a793402036f785a65"} I1218 00:00:07.651537 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e0224f74-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"bcf3d577d95025294053b9f08eee0865475adefcdcf1206e65a5d208c162d542"} I1218 00:00:07.822715 25178 reconciler.go:181] operationExecutor.UnmountVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries") pod "e0224f74-0257-11e9-82fa-080027249e2f" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:07.822788 25178 reconciler.go:181] operationExecutor.UnmountVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "e0224f74-0257-11e9-82fa-080027249e2f" (UID: "e0224f74-0257-11e9-82fa-080027249e2f") I1218 00:00:07.822867 25178 operation_generator.go:698] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries" (OuterVolumeSpecName: "engine-binaries") pod "e0224f74-0257-11e9-82fa-080027249e2f" (UID: "e0224f74-0257-11e9-82fa-080027249e2f"). InnerVolumeSpecName "engine-binaries". PluginName "kubernetes.io/host-path", VolumeGidValue "" I1218 00:00:07.847215 25178 operation_generator.go:698] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h" (OuterVolumeSpecName: "longhorn-service-account-token-s922h") pod "e0224f74-0257-11e9-82fa-080027249e2f" (UID: "e0224f74-0257-11e9-82fa-080027249e2f"). InnerVolumeSpecName "longhorn-service-account-token-s922h". PluginName "kubernetes.io/secret", VolumeGidValue "" 2018-12-18 00:00:07.861 [WARNING][15632] workloadendpoint.go 72: Operation Delete is not supported on WorkloadEndpoint type Calico CNI releasing IP address Calico CNI deleting device in netns /proc/15159/ns/net I1218 00:00:07.923230 25178 reconciler.go:301] Volume detached for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e0224f74-0257-11e9-82fa-080027249e2f-engine-binaries") on node "t93-i" DevicePath "" I1218 00:00:07.923277 25178 reconciler.go:301] Volume detached for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e0224f74-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") on node "t93-i" DevicePath "" E1218 00:00:08.172424 25178 kubelet_volumes.go:140] Orphaned pod "cc89f584-fa4e-11e8-b692-000c298019f4" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. I1218 00:00:08.678573 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e023bf09-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"09dc27a7747f33448384840bbfeaab47850406ed76d8257a793402036f785a65"} I1218 00:00:08.686387 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e0224f74-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"67d8d3d2cb59654bb270147a305d9a3fb446a0a167c85f40ae9f2ca88e36f46e"} W1218 00:00:08.686521 25178 pod_container_deletor.go:75] Container "67d8d3d2cb59654bb270147a305d9a3fb446a0a167c85f40ae9f2ca88e36f46e" not found in pod's containers I1218 00:00:08.686726 25178 kuberuntime_manager.go:414] No ready sandbox for pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)" can be found. Need to start a new one 2018-12-18 00:00:08.922 [WARNING][15774] workloadendpoint.go 72: Operation Delete is not supported on WorkloadEndpoint type Calico CNI releasing IP address Calico CNI deleting device in netns /proc/15168/ns/net Calico CNI fetching podCidr from Kubernetes Calico CNI passing podCidr to host-local IPAM: 10.42.9.0/24 Calico CNI using IPs: [10.42.9.185/32] I1218 00:00:09.262645 25178 kubelet.go:1890] SyncLoop (UPDATE, "api"): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)" I1218 00:00:09.710855 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e0224f74-0257-11e9-82fa-080027249e2f", Type:"ContainerStarted", Data:"33f043c50a68d47f388018c34f4a0509b7020227020e0314a6e2c5509a7f2c09"} I1218 00:00:09.717483 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e023bf09-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"c0985e36f74b1d1595ecfe41a6f0de4fc8792fd5c7aced29bb71226b2aa86a5b"} W1218 00:00:09.717568 25178 pod_container_deletor.go:75] Container "c0985e36f74b1d1595ecfe41a6f0de4fc8792fd5c7aced29bb71226b2aa86a5b" not found in pod's containers I1218 00:00:09.717892 25178 kuberuntime_manager.go:414] No ready sandbox for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" can be found. Need to start a new one I1218 00:00:09.832541 25178 reconciler.go:181] operationExecutor.UnmountVolume started for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") pod "e023bf09-0257-11e9-82fa-080027249e2f" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:09.832626 25178 reconciler.go:181] operationExecutor.UnmountVolume started for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries") pod "e023bf09-0257-11e9-82fa-080027249e2f" (UID: "e023bf09-0257-11e9-82fa-080027249e2f") I1218 00:00:09.832723 25178 operation_generator.go:698] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries" (OuterVolumeSpecName: "engine-binaries") pod "e023bf09-0257-11e9-82fa-080027249e2f" (UID: "e023bf09-0257-11e9-82fa-080027249e2f"). InnerVolumeSpecName "engine-binaries". PluginName "kubernetes.io/host-path", VolumeGidValue "" I1218 00:00:09.854944 25178 operation_generator.go:698] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h" (OuterVolumeSpecName: "longhorn-service-account-token-s922h") pod "e023bf09-0257-11e9-82fa-080027249e2f" (UID: "e023bf09-0257-11e9-82fa-080027249e2f"). InnerVolumeSpecName "longhorn-service-account-token-s922h". PluginName "kubernetes.io/secret", VolumeGidValue "" I1218 00:00:09.933093 25178 reconciler.go:301] Volume detached for volume "longhorn-service-account-token-s922h" (UniqueName: "kubernetes.io/secret/e023bf09-0257-11e9-82fa-080027249e2f-longhorn-service-account-token-s922h") on node "t93-i" DevicePath "" I1218 00:00:09.933148 25178 reconciler.go:301] Volume detached for volume "engine-binaries" (UniqueName: "kubernetes.io/host-path/e023bf09-0257-11e9-82fa-080027249e2f-engine-binaries") on node "t93-i" DevicePath "" I1218 00:00:10.140229 25178 kubelet_pods.go:1083] Killing unwanted pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" I1218 00:00:10.145041 25178 kuberuntime_container.go:555] Killing container "docker://09dc27a7747f33448384840bbfeaab47850406ed76d8257a793402036f785a65" with 30 second grace period E1218 00:00:10.146050 25178 kuberuntime_container.go:67] Can't make a ref to pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", container pvc-8e23b9a0ffeb11e8-c-hc8zjg-c: selfLink was empty, can't make reference E1218 00:00:10.151725 25178 kubelet_volumes.go:140] Orphaned pod "cc89f584-fa4e-11e8-b692-000c298019f4" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. Calico CNI fetching podCidr from Kubernetes Calico CNI passing podCidr to host-local IPAM: 10.42.9.0/24 Calico CNI using IPs: [10.42.9.186/32] 2018-12-18 00:00:10.198 [ERROR][16070] network.go 186: Error creating veth: failed to Statfs "/proc/16021/ns/net": no such file or directory ContainerID="43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" Namespace="longhorn-system" Pod="pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" WorkloadEndpoint="t93--i-k8s-pvc--8e23b9a0ffeb11e8--c--hc8zjg--c--1545091200--znxdw-eth0" 2018-12-18 00:00:10.198 [ERROR][16070] k8s.go 280: Error setting up networking ContainerID="43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" Namespace="longhorn-system" Pod="pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw" WorkloadEndpoint="t93--i-k8s-pvc--8e23b9a0ffeb11e8--c--hc8zjg--c--1545091200--znxdw-eth0" error=failed to Statfs "/proc/16021/ns/net": no such file or directory E1218 00:00:10.208028 25178 cni.go:310] Error adding network: failed to Statfs "/proc/16021/ns/net": no such file or directory E1218 00:00:10.208071 25178 cni.go:278] Error while adding to cni network: failed to Statfs "/proc/16021/ns/net": no such file or directory 2018-12-18 00:00:10.431 [WARNING][16142] workloadendpoint.go 72: Operation Delete is not supported on WorkloadEndpoint type Calico CNI releasing IP address E1218 00:00:10.440962 25178 remote_runtime.go:96] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" network for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw": NetworkPlugin cni failed to set up pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system" network: failed to Statfs "/proc/16021/ns/net": no such file or directory E1218 00:00:10.441032 25178 kuberuntime_sandbox.go:65] CreatePodSandbox for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" network for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw": NetworkPlugin cni failed to set up pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system" network: failed to Statfs "/proc/16021/ns/net": no such file or directory E1218 00:00:10.441065 25178 kuberuntime_manager.go:657] createPodSandbox for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" network for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw": NetworkPlugin cni failed to set up pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system" network: failed to Statfs "/proc/16021/ns/net": no such file or directory E1218 00:00:10.441164 25178 pod_workers.go:186] Error syncing pod e023bf09-0257-11e9-82fa-080027249e2f ("pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)"), skipping: failed to "CreatePodSandbox" for "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)\" failed: rpc error: code = Unknown desc = failed to set up sandbox container \"43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573\" network for pod \"pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw\": NetworkPlugin cni failed to set up pod \"pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system\" network: failed to Statfs \"/proc/16021/ns/net\": no such file or directory" W1218 00:00:10.734032 25178 docker_sandbox.go:375] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "33f043c50a68d47f388018c34f4a0509b7020227020e0314a6e2c5509a7f2c09" I1218 00:00:10.739639 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-4998dc62020111e9-c-gkkxag-c-1545091200-hkfj4_longhorn-system(e0224f74-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e0224f74-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"33f043c50a68d47f388018c34f4a0509b7020227020e0314a6e2c5509a7f2c09"} W1218 00:00:10.739722 25178 pod_container_deletor.go:75] Container "33f043c50a68d47f388018c34f4a0509b7020227020e0314a6e2c5509a7f2c09" not found in pod's containers W1218 00:00:10.742734 25178 docker_sandbox.go:375] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" I1218 00:00:10.748336 25178 kubelet.go:1928] SyncLoop (PLEG): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545091200-znxdw_longhorn-system(e023bf09-0257-11e9-82fa-080027249e2f)", event: &pleg.PodLifecycleEvent{ID:"e023bf09-0257-11e9-82fa-080027249e2f", Type:"ContainerDied", Data:"43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573"} W1218 00:00:10.748435 25178 pod_container_deletor.go:75] Container "43d4183f4c1e87439e6bdcbdac75a441dcf17b8de28e6b5d67229e53f8eb7573" not found in pod's containers E1218 00:00:12.159774 25178 kubelet_volumes.go:140] Orphaned pod "cc89f584-fa4e-11e8-b692-000c298019f4" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. E1218 00:00:14.158368 25178 kubelet_volumes.go:140] Orphaned pod "cc89f584-fa4e-11e8-b692-000c298019f4" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them. I1218 00:00:15.863026 25178 kubelet.go:1899] SyncLoop (DELETE, "api"): "pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp_longhorn-system(b7fd129a-023e-11e9-82fa-080027249e2f)" I1218 00:00:15.873421 25178 kubelet.go:1893] SyncLoop (REMOVE, "api"): "pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp_longhorn-system(b7fd129a-023e-11e9-82fa-080027249e2f)" I1218 00:00:15.873490 25178 kubelet.go:2095] Failed to delete pod "pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp_longhorn-system(b7fd129a-023e-11e9-82fa-080027249e2f)", err: pod not found W1218 00:00:15.883431 25178 status_manager.go:501] Failed to update status for pod "pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp_longhorn-system(b7fd129a-023e-11e9-82fa-080027249e2f)": failed to patch status "{\"status\":{\"containerStatuses\":[{\"containerID\":\"docker://5cfd0496ac62b14da73ac8d18008165c5ee06936e6dba2371054519ea9ea293d\",\"image\":\"rancher/longhorn-manager:v0.3.2\",\"imageID\":\"docker-pullable://rancher/longhorn-manager@sha256:996a3884f1c8184cab7081c4131dbc8aa7610eac200d0b747fdd30011cedd46a\",\"lastState\":{},\"name\":\"pvc-4998dc62020111e9-c-gkkxag-c\",\"ready\":false,\"restartCount\":0,\"state\":{\"terminated\":{\"exitCode\":0,\"finishedAt\":null,\"startedAt\":null}}}]}}" for pod "longhorn-system"/"pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp": pods "pvc-4998dc62020111e9-c-gkkxag-c-1545080400-g7fbp" not found I1218 00:00:15.905872 25178 kubelet.go:1899] SyncLoop (DELETE, "api"): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545080400-5lqn8_longhorn-system(b7fda770-023e-11e9-82fa-080027249e2f)" I1218 00:00:15.919215 25178 kubelet.go:1893] SyncLoop (REMOVE, "api"): "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545080400-5lqn8_longhorn-system(b7fda770-023e-11e9-82fa-080027249e2f)" I1218 00:00:15.919311 25178 kubelet.go:2095] Failed to delete pod "pvc-8e23b9a0ffeb11e8-c-hc8zjg-c-1545080400-5lqn8_longhorn-system(b7fda770-023e-11e9-82fa-080027249e2f)", err: pod not found

[yasker]

@courcelm I was looking for the manager log like instance xxxx crashed, log: or instance xxxx crashed, but cannot get the log. That means the instance (can be engine or replica) crashed by itself, without the request to detach from Longhorn manager / Kubernetes. That's the case with @db-adrian . Though in your log, I didn't the crash message, nor the detach message. So it's really weird.

Can you double check if those are all the logs around the time the read-only issue happens? There are multiple managers normally. See https://github.com/rancher/longhorn/blob/master/docs/troubleshooting.md#manager-and-engines on how to collect all the manager logs.

[courcelm]

@yasker I extracted the logs using kubetail for longhorn manager. No mention of crash. Issue with nodes. I see a bug warning.

2018-12-18T06:34:49.568812942Z I1218 06:34:49.568676 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:34:49.568839685Z I1218 06:34:49.568676 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:34:49.664519358Z I1218 06:34:49.664332 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:34:50.182631349Z I1218 06:34:50.182440 1 reflector.go:428] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: Watch close - v1.PersistentVolumeClaim total 0 items received (B 2018-12-18T06:35:04.569163223Z I1218 06:35:04.568863 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:35:04.569203529Z I1218 06:35:04.568882 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:35:04.665049183Z I1218 06:35:04.664555 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:35:14.913578219Z I1218 06:35:14.913372 1 reflector.go:215] k8s.io/client-go/informers/factory.go:131: forcing resync (B 2018-12-18T06:35:14.91361632Z I1218 06:35:14.913372 1 reflector.go:215] k8s.io/client-go/informers/factory.go:131: forcing resync (B 2018-12-18T06:35:14.913767829Z I1218 06:35:14.913589 1 controller.go:197] Started PV processing "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913790998Z I1218 06:35:14.913608 1 csi_handler.go:353] CSIHandler: processing PV "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913800234Z I1218 06:35:14.913618 1 csi_handler.go:357] CSIHandler: processing PV "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.913810312Z I1218 06:35:14.913607 1 controller.go:167] Started VA processing "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:35:14.913819788Z I1218 06:35:14.913621 1 controller.go:197] Started PV processing "pvc-21a100aa-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913841898Z I1218 06:35:14.913636 1 csi_handler.go:85] CSIHandler: processing VA "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:35:14.913848666Z I1218 06:35:14.913641 1 controller.go:197] Started PV processing "pvc-4d214f14-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913855172Z I1218 06:35:14.913656 1 csi_handler.go:353] CSIHandler: processing PV "pvc-21a100aa-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913861551Z I1218 06:35:14.913688 1 csi_handler.go:357] CSIHandler: processing PV "pvc-21a100aa-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.913868199Z I1218 06:35:14.913658 1 csi_handler.go:107] "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" is already attached (B 2018-12-18T06:35:14.913874693Z I1218 06:35:14.913734 1 csi_handler.go:101] CSIHandler: finished processing "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:35:14.91389759Z I1218 06:35:14.913746 1 controller.go:197] Started PV processing "pvc-c1580c17-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913919257Z I1218 06:35:14.913754 1 controller.go:197] Started PV processing "pvc-c15da027-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913929592Z I1218 06:35:14.913776 1 csi_handler.go:353] CSIHandler: processing PV "pvc-c1580c17-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.913939934Z I1218 06:35:14.913793 1 controller.go:197] Started PV processing "pvc-4998dc62020111e9" (B 2018-12-18T06:35:14.913958817Z I1218 06:35:14.913809 1 csi_handler.go:353] CSIHandler: processing PV "pvc-4998dc62020111e9" (B 2018-12-18T06:35:14.91396557Z I1218 06:35:14.913820 1 csi_handler.go:357] CSIHandler: processing PV "pvc-4998dc62020111e9": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.913971634Z I1218 06:35:14.913581 1 controller.go:167] Started VA processing "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:35:14.913980661Z I1218 06:35:14.913868 1 csi_handler.go:85] CSIHandler: processing VA "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:35:14.913991598Z I1218 06:35:14.913882 1 csi_handler.go:107] "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" is already attached (B 2018-12-18T06:35:14.914002088Z I1218 06:35:14.913781 1 controller.go:197] Started PV processing "pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:35:14.914008666Z I1218 06:35:14.913925 1 csi_handler.go:353] CSIHandler: processing PV "pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:35:14.914014596Z I1218 06:35:14.913944 1 csi_handler.go:357] CSIHandler: processing PV "pvc-8e23b9a0ffeb11e8": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.914022655Z I1218 06:35:14.913797 1 csi_handler.go:357] CSIHandler: processing PV "pvc-c1580c17-ce50-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.914028961Z I1218 06:35:14.913669 1 csi_handler.go:353] CSIHandler: processing PV "pvc-4d214f14-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.914034854Z I1218 06:35:14.913994 1 csi_handler.go:357] CSIHandler: processing PV "pvc-4d214f14-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.914042241Z I1218 06:35:14.913779 1 csi_handler.go:353] CSIHandler: processing PV "pvc-c15da027-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:35:14.914053206Z I1218 06:35:14.914026 1 csi_handler.go:357] CSIHandler: processing PV "pvc-c15da027-ce50-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:35:14.914075377Z I1218 06:35:14.913907 1 csi_handler.go:101] CSIHandler: finished processing "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:35:16.268833572Z I1218 06:35:16.268614 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - v1beta1.VolumeAttachment total 0 items received (B 2018-12-18T06:35:19.569347603Z I1218 06:35:19.569068 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:35:19.569524337Z I1218 06:35:19.569177 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:35:19.665133836Z I1218 06:35:19.664854 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:35:34.569639401Z I1218 06:35:34.569440 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:35:34.569686454Z I1218 06:35:34.569529 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:35:34.665303672Z I1218 06:35:34.665070 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:35:49.569854265Z I1218 06:35:49.569645 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:35:49.569895357Z I1218 06:35:49.569711 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:35:49.665619637Z I1218 06:35:49.665377 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:35:55.763462925Z E1218 06:35:55.763130 1 volume_controller.go:196] fail to sync longhorn-system/pvc-4998dc62020111e9: fail to update recurring jobs for pvc-4998dc62020111e9: Timeout: request did not complete within requested timeout 30s (B 2018-12-18T06:35:55.763740371Z time="2018-12-18T06:35:55Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-4998dc62020111e9 out of the queue: fail to sync longhorn-system/pvc-4998dc62020111e9: fail to update recurring jobs for pvc-4998dc62020111e9: Timeout: request did not complete within requested timeout 30s" (B 2018-12-18T06:35:55.763759936Z E1218 06:35:55.763169 1 volume_controller.go:196] fail to sync longhorn-system/pvc-8e23b9a0ffeb11e8: fail to update recurring jobs for pvc-8e23b9a0ffeb11e8: Timeout: request did not complete within requested timeout 30s (B 2018-12-18T06:35:55.764852988Z time="2018-12-18T06:35:55Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-8e23b9a0ffeb11e8 out of the queue: fail to sync longhorn-system/pvc-8e23b9a0ffeb11e8: fail to update recurring jobs for pvc-8e23b9a0ffeb11e8: Timeout: request did not complete within requested timeout 30s" (B 2018-12-18T06:36:04.570017305Z I1218 06:36:04.569840 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:36:04.570081456Z I1218 06:36:04.569899 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:36:04.665744924Z I1218 06:36:04.665545 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:36:19.570120809Z I1218 06:36:19.570008 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:36:19.570157263Z I1218 06:36:19.570064 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:36:19.665946527Z I1218 06:36:19.665766 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:36:22.882735622Z E1218 06:36:22.882423 1 node_controller.go:289] fail to sync node for longhorn-system/tp91-ir: rpc error: code = Unavailable desc = transport is closing (B 2018-12-18T06:36:22.883901935Z E1218 06:36:22.883682 1 node_controller.go:289] fail to sync node for longhorn-system/tp93-ir: rpc error: code = Unavailable desc = transport is closing (B 2018-12-18T06:36:22.883947957Z time="2018-12-18T06:36:22Z" level=warning msg="Dropping Longhorn node longhorn-system/tp93-ir out of the queue: fail to sync node for longhorn-system/tp93-ir: rpc error: code = Unavailable desc = transport is closing" (B 2018-12-18T06:36:22.884769287Z time="2018-12-18T06:36:22Z" level=warning msg="Dropping Longhorn node longhorn-system/tp91-ir out of the queue: fail to sync node for longhorn-system/tp91-ir: rpc error: code = Unavailable desc = transport is closing" (B 2018-12-18T06:36:22.889052244Z E1218 06:36:22.888401 1 node_controller.go:289] fail to sync node for longhorn-system/tyersub3-ir: rpc error: code = Unavailable desc = transport is closing (B 2018-12-18T06:36:22.889977276Z time="2018-12-18T06:36:22Z" level=warning msg="Dropping Longhorn node longhorn-system/tyersub3-ir out of the queue: fail to sync node for longhorn-system/tyersub3-ir: rpc error: code = Unavailable desc = transport is closing" (B 2018-12-18T06:36:25.253927404Z E1218 06:36:25.253351 1 node_controller.go:289] fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir) (B 2018-12-18T06:36:25.255191258Z time="2018-12-18T06:36:25Z" level=warning msg="Dropping Longhorn node longhorn-system/tp92-ir out of the queue: fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir)" (B 2018-12-18T06:36:25.457632684Z E1218 06:36:25.456976 1 node_controller.go:289] fail to sync node for longhorn-system/tp91-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp91-ir) (B 2018-12-18T06:36:25.457708602Z time="2018-12-18T06:36:25Z" level=warning msg="Dropping Longhorn node longhorn-system/tp91-ir out of the queue: fail to sync node for longhorn-system/tp91-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp91-ir)" (B 2018-12-18T06:36:27.195442691Z E1218 06:36:27.194427 1 node_controller.go:289] fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca) (B 2018-12-18T06:36:27.195479491Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/ca out of the queue: fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca)" (B 2018-12-18T06:36:27.747854435Z E1218 06:36:27.747117 1 node_controller.go:289] fail to sync node for longhorn-system/tp93-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp93-ir) (B 2018-12-18T06:36:27.747940494Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp93-ir out of the queue: fail to sync node for longhorn-system/tp93-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp93-ir)" (B 2018-12-18T06:36:27.779699918Z E1218 06:36:27.779366 1 node_controller.go:289] fail to sync node for longhorn-system/tp91-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp91-ir) (B 2018-12-18T06:36:27.779764419Z E1218 06:36:27.779478 1 node_controller.go:289] fail to sync node for longhorn-system/tp87-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp87-ir) (B 2018-12-18T06:36:27.779778719Z E1218 06:36:27.779366 1 node_controller.go:289] fail to sync node for longhorn-system/tp90-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp90-ir) (B 2018-12-18T06:36:27.779787619Z E1218 06:36:27.779388 1 node_controller.go:289] fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir) (B 2018-12-18T06:36:27.779800119Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp91-ir out of the queue: fail to sync node for longhorn-system/tp91-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp91-ir)" (B 2018-12-18T06:36:27.78079653Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp87-ir out of the queue: fail to sync node for longhorn-system/tp87-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp87-ir)" (B 2018-12-18T06:36:27.781948943Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp90-ir out of the queue: fail to sync node for longhorn-system/tp90-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp90-ir)" (B 2018-12-18T06:36:27.782977554Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp92-ir out of the queue: fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir)" (B 2018-12-18T06:36:27.848785097Z E1218 06:36:27.848082 1 node_controller.go:289] fail to sync node for longhorn-system/tp88-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp88-ir) (B 2018-12-18T06:36:27.848844008Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp88-ir out of the queue: fail to sync node for longhorn-system/tp88-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp88-ir)" (B 2018-12-18T06:36:27.966160711Z E1218 06:36:27.965541 1 node_controller.go:289] fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir) (B 2018-12-18T06:36:27.966669847Z E1218 06:36:27.966308 1 node_controller.go:289] fail to sync node for longhorn-system/tp85-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp85-ir) (B 2018-12-18T06:36:27.966693942Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp92-ir out of the queue: fail to sync node for longhorn-system/tp92-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp92-ir)" (B 2018-12-18T06:36:27.966709615Z E1218 06:36:27.965626 1 node_controller.go:289] fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca) (B 2018-12-18T06:36:27.966746573Z E1218 06:36:27.965939 1 node_controller.go:289] fail to sync node for longhorn-system/tyersub3-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tyersub3-ir) (B 2018-12-18T06:36:27.966760392Z E1218 06:36:27.966038 1 node_controller.go:289] fail to sync node for longhorn-system/tp95-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp95-ir) (B 2018-12-18T06:36:27.967794351Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp85-ir out of the queue: fail to sync node for longhorn-system/tp85-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp85-ir)" (B 2018-12-18T06:36:27.969084595Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/ca out of the queue: fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca)" (B 2018-12-18T06:36:27.973292086Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tyersub3-ir out of the queue: fail to sync node for longhorn-system/tyersub3-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tyersub3-ir)" (B 2018-12-18T06:36:27.974389266Z time="2018-12-18T06:36:27Z" level=warning msg="Dropping Longhorn node longhorn-system/tp95-ir out of the queue: fail to sync node for longhorn-system/tp95-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp95-ir)" (B 2018-12-18T06:36:29.59844356Z E1218 06:36:29.598118 1 node_controller.go:289] fail to sync node for longhorn-system/tp94-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp94-ir) (B 2018-12-18T06:36:29.62465557Z time="2018-12-18T06:36:29Z" level=warning msg="Dropping Longhorn node longhorn-system/tp94-ir out of the queue: fail to sync node for longhorn-system/tp94-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp94-ir)" (B 2018-12-18T06:36:29.624720631Z E1218 06:36:29.598118 1 node_controller.go:289] fail to sync node for longhorn-system/tp87-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp87-ir) (B 2018-12-18T06:36:29.624735561Z E1218 06:36:29.598120 1 node_controller.go:289] fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca) (B 2018-12-18T06:36:29.624745774Z E1218 06:36:29.598170 1 node_controller.go:289] fail to sync node for longhorn-system/ubuntu: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ubuntu) (B 2018-12-18T06:36:29.624755735Z E1218 06:36:29.598135 1 node_controller.go:289] fail to sync node for longhorn-system/tp85-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp85-ir) (B 2018-12-18T06:36:29.625760831Z time="2018-12-18T06:36:29Z" level=warning msg="Dropping Longhorn node longhorn-system/tp87-ir out of the queue: fail to sync node for longhorn-system/tp87-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp87-ir)" (B 2018-12-18T06:36:29.626818221Z time="2018-12-18T06:36:29Z" level=warning msg="Dropping Longhorn node longhorn-system/ca out of the queue: fail to sync node for longhorn-system/ca: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ca)" (B 2018-12-18T06:36:29.627925442Z time="2018-12-18T06:36:29Z" level=warning msg="Dropping Longhorn node longhorn-system/ubuntu out of the queue: fail to sync node for longhorn-system/ubuntu: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes ubuntu)" (B 2018-12-18T06:36:29.629030271Z time="2018-12-18T06:36:29Z" level=warning msg="Dropping Longhorn node longhorn-system/tp85-ir out of the queue: fail to sync node for longhorn-system/tp85-ir: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes tp85-ir)" (B 2018-12-18T06:36:34.570408921Z I1218 06:36:34.570236 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:36:34.570441758Z I1218 06:36:34.570236 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:36:34.666146779Z I1218 06:36:34.665964 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:36:49.570687064Z I1218 06:36:49.570472 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:36:49.57073114Z I1218 06:36:49.570525 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:36:49.66648086Z I1218 06:36:49.666185 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:36:58.552817232Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25857934\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.55479559Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25857934\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.564606659Z time="2018-12-18T06:36:58Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:36:58.565306719Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25857934\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.566999451Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25857934\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.569547464Z time="2018-12-18T06:36:58Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:36:58.570101449Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:58.572529833Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:58.574706804Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.574971477Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:58.585051536Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.585647641Z time="2018-12-18T06:36:58Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:36:58.585996582Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.586921937Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:58.597512638Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.626630958Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:58.639611082Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:36:58.671348763Z time="2018-12-18T06:36:58Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:36:59.494362178Z time="2018-12-18T06:36:59Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:36:59.660587375Z time="2018-12-18T06:36:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858305\", FieldPath:\"\"}): type: 'Warning' reason: 'KubernetesNodeNotReady' Kubernetes node tp87-ir not ready: NodeStatusUnknown" (B 2018-12-18T06:37:02.497946228Z time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376401\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica pvc-8e23b9a0ffeb11e8-r-903cf5dc (10.42.6.105) in error" (B 2018-12-18T06:37:02.514648043Z time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376699\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-8e23b9a0ffeb11e8 became degraded" (B 2018-12-18T06:37:02.531308638Z time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-3702d244 due to conflict" (B 2018-12-18T06:37:02.531349078Z time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc due to conflict" (B 2018-12-18T06:37:02.54699514Z time="2018-12-18T06:37:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"24376699\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-8e23b9a0ffeb11e8 became degraded" (B 2018-12-18T06:37:02.90281765Z time="2018-12-18T06:37:02Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8 due to conflict" (B 2018-12-18T06:37:04.263447438Z time="2018-12-18T06:37:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-r-903cf5dc\", UID:\"9441fa7f-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858385\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-8e23b9a0ffeb11e8-r-903cf5dc" (B 2018-12-18T06:37:04.570893732Z I1218 06:37:04.570684 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:37:04.57093389Z I1218 06:37:04.570681 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:37:04.6667035Z I1218 06:37:04.666419 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:37:04.668878648Z time="2018-12-18T06:37:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-r-3702d244\", UID:\"540bbd4f-028f-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858392\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-8e23b9a0ffeb11e8-r-3702d244" (B 2018-12-18T06:37:05.464009376Z time="2018-12-18T06:37:05Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc due to conflict" (B 2018-12-18T06:37:06.483453608Z time="2018-12-18T06:37:06Z" level=debug msg="Instance pvc-8e23b9a0ffeb11e8-r-3702d244 starts running, IP 10.42.3.62" (B 2018-12-18T06:37:11.27993571Z time="2018-12-18T06:37:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-8e23b9a0ffeb11e8-r-3702d244 with IP 10.42.3.62 for pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:37:12.544984288Z time="2018-12-18T06:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica UNKNOWN-10.42.6.105 (10.42.6.105) in error" (B 2018-12-18T06:37:12.545031383Z time="2018-12-18T06:37:12Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Detected rebuilding replica pvc-8e23b9a0ffeb11e8-r-3702d244 (10.42.3.62)" (B 2018-12-18T06:37:18.467463168Z time="2018-12-18T06:37:18Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" (B 2018-12-18T06:37:18.479679123Z time="2018-12-18T06:37:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858495\", FieldPath:\"\"}): type: 'Normal' reason: 'Delete' Removed unknown replica IP 10.42.6.105 from engine" (B 2018-12-18T06:37:19.571206566Z I1218 06:37:19.570972 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:37:19.57126499Z I1218 06:37:19.570954 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:37:19.666902812Z I1218 06:37:19.666669 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:37:21.659911029Z time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529751\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica pvc-4998dc62020111e9-r-b4a86f52 (10.42.6.106) in error" (B 2018-12-18T06:37:21.676175809Z time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" (B 2018-12-18T06:37:21.689257758Z time="2018-12-18T06:37:21Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" (B 2018-12-18T06:37:21.689285166Z time="2018-12-18T06:37:21Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" (B 2018-12-18T06:37:21.693377034Z time="2018-12-18T06:37:21Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" (B 2018-12-18T06:37:22.064144327Z time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25529867\", FieldPath:\"\"}): type: 'Normal' reason: 'Degraded' volume pvc-4998dc62020111e9 became degraded" (B 2018-12-18T06:37:22.064526024Z time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-b4a86f52\", UID:\"4c01aad9-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858575\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-4998dc62020111e9-r-b4a86f52" (B 2018-12-18T06:37:22.262630326Z time="2018-12-18T06:37:22Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-r-c6c69ca8\", UID:\"5f779b2f-028f-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858577\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-4998dc62020111e9-r-c6c69ca8" (B 2018-12-18T06:37:22.265037311Z time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" (B 2018-12-18T06:37:22.286359527Z time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/tp91-ir due to conflict" (B 2018-12-18T06:37:22.686272114Z time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/tp91-ir due to conflict" (B 2018-12-18T06:37:22.862921252Z time="2018-12-18T06:37:22Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-c6c69ca8 due to conflict" (B 2018-12-18T06:37:23.263019033Z time="2018-12-18T06:37:23Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" (B 2018-12-18T06:37:23.861215452Z time="2018-12-18T06:37:23Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" (B 2018-12-18T06:37:24.064049925Z time="2018-12-18T06:37:24Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" (B 2018-12-18T06:37:24.26349777Z time="2018-12-18T06:37:24Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 due to conflict" (B 2018-12-18T06:37:26.860736038Z time="2018-12-18T06:37:26Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" (B 2018-12-18T06:37:34.571531841Z I1218 06:37:34.571383 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:37:34.571576377Z I1218 06:37:34.571373 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:37:34.667108969Z I1218 06:37:34.666923 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:37:38.869217227Z time="2018-12-18T06:37:38Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858475\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Replica pvc-8e23b9a0ffeb11e8-r-3702d244 with IP 10.42.3.62 has been rebuilded for volume pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:37:39.663716618Z time="2018-12-18T06:37:39Z" level=debug msg="Skip rebuilding for volume pvc-8e23b9a0ffeb11e8 because there is rebuilding in process" (B 2018-12-18T06:37:43.06934457Z time="2018-12-18T06:37:43Z" level=debug msg="Requeue longhorn-system/tp93-ir due to conflict" (B 2018-12-18T06:37:43.161122201Z time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8-e-c104f769\", UID:\"94414a78-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858591\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Detected replica pvc-8e23b9a0ffeb11e8-r-3702d244 (10.42.3.62) has been rebuilded" (B 2018-12-18T06:37:43.177069424Z time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858388\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-8e23b9a0ffeb11e8 became healthy" (B 2018-12-18T06:37:43.187637346Z time="2018-12-18T06:37:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-8e23b9a0ffeb11e8\", UID:\"943ddd37-ffeb-11e8-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858388\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-8e23b9a0ffeb11e8 became healthy" (B 2018-12-18T06:37:43.195358256Z time="2018-12-18T06:37:43Z" level=debug msg="Requeue longhorn-system/pvc-8e23b9a0ffeb11e8 due to conflict" (B 2018-12-18T06:37:44.668929675Z time="2018-12-18T06:37:44Z" level=debug msg="Requeue longhorn-system/tp93-ir due to conflict" (B 2018-12-18T06:37:49.571710374Z I1218 06:37:49.571589 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:37:49.571761299Z I1218 06:37:49.571607 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:37:49.667380916Z I1218 06:37:49.667141 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:38:04.572043444Z I1218 06:38:04.571817 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:38:04.572091716Z I1218 06:38:04.571882 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:38:04.667607131Z I1218 06:38:04.667401 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:38:19.572233422Z I1218 06:38:19.572022 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:38:19.572279996Z I1218 06:38:19.572022 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:38:19.667811962Z I1218 06:38:19.667604 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:38:34.572417016Z I1218 06:38:34.572249 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:38:34.572478129Z I1218 06:38:34.572253 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:38:34.667983522Z I1218 06:38:34.667769 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:38:49.481346895Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.485141108Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.572664502Z I1218 06:38:49.572489 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:38:49.572760798Z I1218 06:38:49.572511 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:38:49.590757571Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.590922873Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.591852717Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.601424188Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.668073956Z I1218 06:38:49.667904 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:38:49.861322515Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.862032036Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.86209077Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.933351034Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:49.935380097Z time="2018-12-18T06:38:49Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:50.339313396Z time="2018-12-18T06:38:50Z" level=debug msg="Requeue longhorn-system/tp87-ir due to conflict" (B 2018-12-18T06:38:50.773488575Z I1218 06:38:50.771803 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.237:58776->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:50.773541576Z I1218 06:38:50.771803 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.237:58776->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:50.773547951Z I1218 06:38:50.773293 1 reflector.go:428] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: Watch close - v1.PersistentVolumeClaim total 0 items received (B 2018-12-18T06:38:50.773563061Z I1218 06:38:50.773298 1 reflector.go:428] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: Watch close - v1.StorageClass total 0 items received (B 2018-12-18T06:38:50.77356882Z I1218 06:38:50.771818 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.237:58776->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:50.773577938Z I1218 06:38:50.773462 1 reflector.go:428] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: Watch close - v1.PersistentVolume total 0 items received (B 2018-12-18T06:38:50.807790342Z W1218 06:38:50.807622 1 reflector.go:341] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of v1.PersistentVolume ended with: too old resource version: 25529521 (25858023) (B 2018-12-18T06:38:50.807816168Z W1218 06:38:50.807622 1 reflector.go:341] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: watch of v1.StorageClass ended with: too old resource version: 23906596 (25858024) (B 2018-12-18T06:38:50.807822555Z W1218 06:38:50.807624 1 reflector.go:341] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: watch of v1.PersistentVolumeClaim ended with: too old resource version: 25529550 (25858047) (B 2018-12-18T06:38:51.808233306Z I1218 06:38:51.807994 1 reflector.go:240] Listing and watching v1.PersistentVolumeClaim from github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496 (B 2018-12-18T06:38:51.808294212Z I1218 06:38:51.807994 1 reflector.go:240] Listing and watching v1.PersistentVolume from github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497 (B 2018-12-18T06:38:51.80830374Z I1218 06:38:51.807994 1 reflector.go:240] Listing and watching v1.StorageClass from github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498 (B 2018-12-18T06:38:55.504042334Z I1218 06:38:55.503836 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.236:49758->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:55.504084432Z I1218 06:38:55.503836 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.236:49758->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:55.504099929Z I1218 06:38:55.503836 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.236:49758->10.43.0.1:443: read: connection reset by peer (B 2018-12-18T06:38:55.504572387Z I1218 06:38:55.504510 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - v1.Node total 211 items received (B 2018-12-18T06:38:55.504599173Z I1218 06:38:55.504520 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - v1beta1.VolumeAttachment total 0 items received (B 2018-12-18T06:38:55.504693624Z I1218 06:38:55.504588 1 reflector.go:357] k8s.io/client-go/informers/factory.go:131: Watch close - v1.PersistentVolume total 0 items received (B 2018-12-18T06:38:55.532299265Z W1218 06:38:55.531550 1 reflector.go:270] k8s.io/client-go/informers/factory.go:131: watch of v1beta1.VolumeAttachment ended with: too old resource version: 25529728 (25858072) (B 2018-12-18T06:38:55.532329236Z W1218 06:38:55.531550 1 reflector.go:270] k8s.io/client-go/informers/factory.go:131: watch of v1.PersistentVolume ended with: too old resource version: 25529521 (25857986) (B 2018-12-18T06:38:56.480417611Z time="2018-12-18T06:38:56Z" level=debug msg="Instance pvc-4998dc62020111e9-r-c6c69ca8 starts running, IP 10.42.4.17" (B 2018-12-18T06:38:56.532487982Z I1218 06:38:56.532332 1 reflector.go:169] Listing and watching v1beta1.VolumeAttachment from k8s.io/client-go/informers/factory.go:131 (B 2018-12-18T06:38:56.532707574Z I1218 06:38:56.532321 1 reflector.go:169] Listing and watching v1.PersistentVolume from k8s.io/client-go/informers/factory.go:131 (B 2018-12-18T06:38:56.540390158Z I1218 06:38:56.540255 1 controller.go:167] Started VA processing "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:38:56.54041776Z I1218 06:38:56.540290 1 csi_handler.go:85] CSIHandler: processing VA "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:38:56.540422739Z I1218 06:38:56.540299 1 csi_handler.go:107] "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" is already attached (B 2018-12-18T06:38:56.540439649Z I1218 06:38:56.540311 1 csi_handler.go:101] CSIHandler: finished processing "csi-4b92b693a8d7f4c00598d6baadd95b6e822b291d2f359423c8c29a9da7faea90" (B 2018-12-18T06:38:56.540444309Z I1218 06:38:56.540297 1 controller.go:167] Started VA processing "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:38:56.540448352Z I1218 06:38:56.540338 1 csi_handler.go:85] CSIHandler: processing VA "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:38:56.540452256Z I1218 06:38:56.540352 1 csi_handler.go:107] "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" is already attached (B 2018-12-18T06:38:56.540455996Z I1218 06:38:56.540361 1 csi_handler.go:101] CSIHandler: finished processing "csi-597355860aa4ab0567596a87a2300bb792e969cfd0e8ee62f7f106d7be05482c" (B 2018-12-18T06:38:56.542131763Z I1218 06:38:56.542080 1 controller.go:197] Started PV processing "pvc-c1580c17-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542148163Z I1218 06:38:56.542091 1 controller.go:197] Started PV processing "pvc-c15da027-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542156491Z I1218 06:38:56.542095 1 csi_handler.go:353] CSIHandler: processing PV "pvc-c1580c17-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542171464Z I1218 06:38:56.542101 1 csi_handler.go:353] CSIHandler: processing PV "pvc-c15da027-ce50-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542179304Z I1218 06:38:56.542112 1 csi_handler.go:357] CSIHandler: processing PV "pvc-c15da027-ce50-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542189586Z I1218 06:38:56.542116 1 controller.go:197] Started PV processing "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542208415Z I1218 06:38:56.542103 1 csi_handler.go:357] CSIHandler: processing PV "pvc-c1580c17-ce50-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542216403Z I1218 06:38:56.542126 1 csi_handler.go:353] CSIHandler: processing PV "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542229647Z I1218 06:38:56.542132 1 controller.go:197] Started PV processing "pvc-21a100aa-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542237027Z I1218 06:38:56.542141 1 csi_handler.go:353] CSIHandler: processing PV "pvc-21a100aa-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542243961Z I1218 06:38:56.542143 1 controller.go:197] Started PV processing "pvc-4998dc62020111e9" (B 2018-12-18T06:38:56.542249673Z I1218 06:38:56.542147 1 csi_handler.go:357] CSIHandler: processing PV "pvc-21a100aa-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542265662Z I1218 06:38:56.542149 1 csi_handler.go:353] CSIHandler: processing PV "pvc-4998dc62020111e9" (B 2018-12-18T06:38:56.542269658Z I1218 06:38:56.542133 1 csi_handler.go:357] CSIHandler: processing PV "pvc-0cc515fc-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542273489Z I1218 06:38:56.542153 1 controller.go:197] Started PV processing "pvc-4d214f14-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.54227711Z I1218 06:38:56.542165 1 controller.go:197] Started PV processing "pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:38:56.542282719Z I1218 06:38:56.542170 1 csi_handler.go:353] CSIHandler: processing PV "pvc-4d214f14-ce51-11e8-8f02-000c298019f4" (B 2018-12-18T06:38:56.542286623Z I1218 06:38:56.542187 1 csi_handler.go:357] CSIHandler: processing PV "pvc-4d214f14-ce51-11e8-8f02-000c298019f4": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542290562Z I1218 06:38:56.542154 1 csi_handler.go:357] CSIHandler: processing PV "pvc-4998dc62020111e9": no deletion timestamp, ignoring (B 2018-12-18T06:38:56.542294284Z I1218 06:38:56.542174 1 csi_handler.go:353] CSIHandler: processing PV "pvc-8e23b9a0ffeb11e8" (B 2018-12-18T06:38:56.542298047Z I1218 06:38:56.542233 1 csi_handler.go:357] CSIHandler: processing PV "pvc-8e23b9a0ffeb11e8": no deletion timestamp, ignoring (B 2018-12-18T06:38:59.241512168Z time="2018-12-18T06:38:59Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:38:59.241643566Z time="2018-12-18T06:38:59Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:38:59.242902881Z time="2018-12-18T06:38:59Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:38:59.436789998Z time="2018-12-18T06:38:59Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:38:59.460133497Z time="2018-12-18T06:38:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Start rebuilding replica pvc-4998dc62020111e9-r-c6c69ca8 with IP 10.42.4.17 for pvc-4998dc62020111e9" (B 2018-12-18T06:39:00.238123691Z time="2018-12-18T06:39:00Z" level=debug msg="Requeue longhorn-system/tp86-ir due to conflict" (B 2018-12-18T06:39:02.118443676Z time="2018-12-18T06:39:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Warning' reason: 'Faulted' Detected replica UNKNOWN-10.42.6.106 (10.42.6.106) in error" (B 2018-12-18T06:39:02.118510315Z time="2018-12-18T06:39:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilding' Detected rebuilding replica pvc-4998dc62020111e9-r-c6c69ca8 (10.42.4.17)" (B 2018-12-18T06:39:05.834356076Z time="2018-12-18T06:39:05Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" (B 2018-12-18T06:39:05.850610167Z time="2018-12-18T06:39:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859387\", FieldPath:\"\"}): type: 'Normal' reason: 'Delete' Removed unknown replica IP 10.42.6.106 from engine" (B 2018-12-18T06:39:06.81891516Z I1218 06:39:06.818669 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:39:06.818978874Z I1218 06:39:06.818674 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:39:06.819442431Z I1218 06:39:06.819292 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:39:08.433702407Z time="2018-12-18T06:39:08Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" (B 2018-12-18T06:39:10.83313624Z time="2018-12-18T06:39:10Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" (B 2018-12-18T06:39:13.240665355Z time="2018-12-18T06:39:13Z" level=debug msg="Requeue longhorn-system/tp93-ir due to conflict" (B 2018-12-18T06:39:16.610630878Z E1218 06:39:16.609597 1 node_controller.go:175] unable to handle object in controller.NodeController: cache.DeletedFinalStateUnknown (B 2018-12-18T06:39:16.613384949Z E1218 06:39:16.613271 1 node_controller.go:175] unable to handle object in controller.NodeController: cache.DeletedFinalStateUnknown (B 2018-12-18T06:39:16.617890183Z time="2018-12-18T06:39:16Z" level=warning msg="BUG: {longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:pvc-4998dc62020111e9-r-b4a86f52,GenerateName:,Namespace:longhorn-system,SelfLink:/api/v1/namespaces/longhorn-system/pods/pvc-4998dc62020111e9-r-b4a86f52,UID:4f4bf467-0201-11e9-9f7b-000c298019f4,ResourceVersion:25529642,Generation:0,CreationTimestamp:2018-12-17 13:40:26 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{longhorn-volume-replica: pvc-4998dc62020111e9,},Annotations:map[string]string{cni.projectcalico.org/podIP: 10.42.6.106/32,},OwnerReferences:[{longhorn.rancher.io/v1alpha1 longhorn.rancher.io/v1alpha1, Kind=Replica pvc-4998dc62020111e9-r-b4a86f52 4c01aad9-0201-11e9-9f7b-000c298019f4 }],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{volume {HostPathVolumeSource{Path:/var/lib/rancher/longhorn/replicas/pvc-4998dc62020111e9-a14db7a1,Type:,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {default-token-qqkwh {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-qqkwh,Items:[],DefaultMode:420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{pvc-4998dc62020111e9-r-b4a86f52 rancher/longhorn-engine:v0.3.0 [longhorn replica --listen 0.0.0.0:9502 --size 1073741824 /volume] [] [] [] [] {map[] map[]} [{volume false /volume } {default-token-qqkwh true /var/run/secrets/kubernetes.io/serviceaccount }] [] nil Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/v1,Port:9502,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:1,TimeoutSeconds:1,PeriodSeconds:1,SuccessThreshold:1,FailureThreshold:10,} nil /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:Default,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:tp87-ir,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc000f1a570} {node.kubernetes.io/unreachable Exists NoExecute 0xc000f1a5a0}],HostAliases:[],PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:26 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:26 +0000 UTC }],Message:,Reason:,HostIP:132.204.81.148,PodIP:10.42.6.106,StartTime:2018-12-17 13:40:26 +0000 UTC,ContainerStatuses:[{pvc-4998dc62020111e9-r-b4a86f52 {nil ContainerStateRunning{StartedAt:2018-12-17 13:40:27 +0000 UTC,} nil} {nil nil nil} true 0 rancher/longhorn-engine:v0.3.0 docker-pullable://rancher/longhorn-engine@sha256:4748e6601502c2f765cf1dbbaaff76f43f536b5d35622338b54f2413533f89d9 docker://5c59c661e801940cc83d265bad94ba7e736b0fd3cf82090c7443bd26e5466d05}],QOSClass:BestEffort,InitContainerStatuses:[],NominatedNodeName:,},}} cannot be convert to metav1.Object: object does not implement the Object interfaces" (B 2018-12-18T06:39:16.618286971Z time="2018-12-18T06:39:16Z" level=warning msg="BUG: {longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:pvc-8e23b9a0ffeb11e8-r-903cf5dc,GenerateName:,Namespace:longhorn-system,SelfLink:/api/v1/namespaces/longhorn-system/pods/pvc-8e23b9a0ffeb11e8-r-903cf5dc,UID:b89562d7-ffee-11e8-9f7b-000c298019f4,ResourceVersion:24376280,Generation:0,CreationTimestamp:2018-12-14 22:22:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{longhorn-volume-replica: pvc-8e23b9a0ffeb11e8,},Annotations:map[string]string{cni.projectcalico.org/podIP: 10.42.6.105/32,},OwnerReferences:[{longhorn.rancher.io/v1alpha1 longhorn.rancher.io/v1alpha1, Kind=Replica pvc-8e23b9a0ffeb11e8-r-903cf5dc 9441fa7f-ffeb-11e8-9f7b-000c298019f4 }],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{volume {HostPathVolumeSource{Path:/var/lib/rancher/longhorn/replicas/pvc-8e23b9a0ffeb11e8-d5df2733,Type:,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {default-token-qqkwh {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-qqkwh,Items:[],DefaultMode:420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{pvc-8e23b9a0ffeb11e8-r-903cf5dc rancher/longhorn-engine:v0.3.0 [longhorn replica --listen 0.0.0.0:9502 --size 1073741824 /volume] [] [] [] [] {map[] map[]} [{volume false /volume } {default-token-qqkwh true /var/run/secrets/kubernetes.io/serviceaccount }] [] nil Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/v1,Port:9502,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:1,TimeoutSeconds:1,PeriodSeconds:1,SuccessThreshold:1,FailureThreshold:10,} nil /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:Default,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:tp87-ir,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc001581c00} {node.kubernetes.io/unreachable Exists NoExecute 0xc001581c20}],HostAliases:[],PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:19 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:21 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:19 +0000 UTC }],Message:,Reason:,HostIP:132.204.81.148,PodIP:10.42.6.105,StartTime:2018-12-14 22:22:19 +0000 UTC,ContainerStatuses:[{pvc-8e23b9a0ffeb11e8-r-903cf5dc {nil ContainerStateRunning{StartedAt:2018-12-14 22:22:20 +0000 UTC,} nil} {nil nil nil} true 0 rancher/longhorn-engine:v0.3.0 docker-pullable://rancher/longhorn-engine@sha256:4748e6601502c2f765cf1dbbaaff76f43f536b5d35622338b54f2413533f89d9 docker://688aef9717ff7a0f2edee754b380854ea87f568be3741153cce1d76f4b70987c}],QOSClass:BestEffort,InitContainerStatuses:[],NominatedNodeName:,},}} cannot be convert to metav1.Object: object does not implement the Object interfaces" (B 2018-12-18T06:39:16.618314361Z time="2018-12-18T06:39:16Z" level=warning msg="BUG: {longhorn-system/pvc-4998dc62020111e9-r-b4a86f52 &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:pvc-4998dc62020111e9-r-b4a86f52,GenerateName:,Namespace:longhorn-system,SelfLink:/api/v1/namespaces/longhorn-system/pods/pvc-4998dc62020111e9-r-b4a86f52,UID:4f4bf467-0201-11e9-9f7b-000c298019f4,ResourceVersion:25529642,Generation:0,CreationTimestamp:2018-12-17 13:40:26 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{longhorn-volume-replica: pvc-4998dc62020111e9,},Annotations:map[string]string{cni.projectcalico.org/podIP: 10.42.6.106/32,},OwnerReferences:[{longhorn.rancher.io/v1alpha1 longhorn.rancher.io/v1alpha1, Kind=Replica pvc-4998dc62020111e9-r-b4a86f52 4c01aad9-0201-11e9-9f7b-000c298019f4 }],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{volume {HostPathVolumeSource{Path:/var/lib/rancher/longhorn/replicas/pvc-4998dc62020111e9-a14db7a1,Type:,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {default-token-qqkwh {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-qqkwh,Items:[],DefaultMode:420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{pvc-4998dc62020111e9-r-b4a86f52 rancher/longhorn-engine:v0.3.0 [longhorn replica --listen 0.0.0.0:9502 --size 1073741824 /volume] [] [] [] [] {map[] map[]} [{volume false /volume } {default-token-qqkwh true /var/run/secrets/kubernetes.io/serviceaccount }] [] nil Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/v1,Port:9502,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:1,TimeoutSeconds:1,PeriodSeconds:1,SuccessThreshold:1,FailureThreshold:10,} nil /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:Default,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:tp87-ir,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc000f1a570} {node.kubernetes.io/unreachable Exists NoExecute 0xc000f1a5a0}],HostAliases:[],PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:26 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:28 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:28 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-12-17 13:40:26 +0000 UTC }],Message:,Reason:,HostIP:132.204.81.148,PodIP:10.42.6.106,StartTime:2018-12-17 13:40:26 +0000 UTC,ContainerStatuses:[{pvc-4998dc62020111e9-r-b4a86f52 {nil ContainerStateRunning{StartedAt:2018-12-17 13:40:27 +0000 UTC,} nil} {nil nil nil} true 0 rancher/longhorn-engine:v0.3.0 docker-pullable://rancher/longhorn-engine@sha256:4748e6601502c2f765cf1dbbaaff76f43f536b5d35622338b54f2413533f89d9 docker://5c59c661e801940cc83d265bad94ba7e736b0fd3cf82090c7443bd26e5466d05}],QOSClass:BestEffort,InitContainerStatuses:[],NominatedNodeName:,},}} cannot be convert to metav1.Object: object does not implement the Object interfaces" (B 2018-12-18T06:39:16.618334284Z time="2018-12-18T06:39:16Z" level=warning msg="BUG: {longhorn-system/pvc-8e23b9a0ffeb11e8-r-903cf5dc &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:pvc-8e23b9a0ffeb11e8-r-903cf5dc,GenerateName:,Namespace:longhorn-system,SelfLink:/api/v1/namespaces/longhorn-system/pods/pvc-8e23b9a0ffeb11e8-r-903cf5dc,UID:b89562d7-ffee-11e8-9f7b-000c298019f4,ResourceVersion:24376280,Generation:0,CreationTimestamp:2018-12-14 22:22:19 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{longhorn-volume-replica: pvc-8e23b9a0ffeb11e8,},Annotations:map[string]string{cni.projectcalico.org/podIP: 10.42.6.105/32,},OwnerReferences:[{longhorn.rancher.io/v1alpha1 longhorn.rancher.io/v1alpha1, Kind=Replica pvc-8e23b9a0ffeb11e8-r-903cf5dc 9441fa7f-ffeb-11e8-9f7b-000c298019f4 }],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{volume {HostPathVolumeSource{Path:/var/lib/rancher/longhorn/replicas/pvc-8e23b9a0ffeb11e8-d5df2733,Type:,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {default-token-qqkwh {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-qqkwh,Items:[],DefaultMode:420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{pvc-8e23b9a0ffeb11e8-r-903cf5dc rancher/longhorn-engine:v0.3.0 [longhorn replica --listen 0.0.0.0:9502 --size 1073741824 /volume] [] [] [] [] {map[] map[]} [{volume false /volume } {default-token-qqkwh true /var/run/secrets/kubernetes.io/serviceaccount }] [] nil Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/v1,Port:9502,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:1,TimeoutSeconds:1,PeriodSeconds:1,SuccessThreshold:1,FailureThreshold:10,} nil /dev/termination-log File IfNotPresent &SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:Default,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:tp87-ir,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc001581c00} {node.kubernetes.io/unreachable Exists NoExecute 0xc001581c20}],HostAliases:[],PriorityClassName:,Priority:0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:19 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:21 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-12-14 22:22:19 +0000 UTC }],Message:,Reason:,HostIP:132.204.81.148,PodIP:10.42.6.105,StartTime:2018-12-14 22:22:19 +0000 UTC,ContainerStatuses:[{pvc-8e23b9a0ffeb11e8-r-903cf5dc {nil ContainerStateRunning{StartedAt:2018-12-14 22:22:20 +0000 UTC,} nil} {nil nil nil} true 0 rancher/longhorn-engine:v0.3.0 docker-pullable://rancher/longhorn-engine@sha256:4748e6601502c2f765cf1dbbaaff76f43f536b5d35622338b54f2413533f89d9 docker://688aef9717ff7a0f2edee754b380854ea87f568be3741153cce1d76f4b70987c}],QOSClass:BestEffort,InitContainerStatuses:[],NominatedNodeName:,},}} cannot be convert to metav1.Object: object does not implement the Object interfaces" (B 2018-12-18T06:39:17.766438791Z time="2018-12-18T06:39:17Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"tp87-ir\", UID:\"7dd737e9-db88-11e8-8f02-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859184\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node tp87-ir is ready" (B 2018-12-18T06:39:21.824108701Z I1218 06:39:21.818885 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:39:21.824147004Z I1218 06:39:21.818857 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:39:21.824163491Z I1218 06:39:21.819472 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:39:24.077408477Z time="2018-12-18T06:39:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859304\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Replica pvc-4998dc62020111e9-r-c6c69ca8 with IP 10.42.4.17 has been rebuilded for volume pvc-4998dc62020111e9" (B 2018-12-18T06:39:27.03328598Z time="2018-12-18T06:39:27Z" level=debug msg="Skip rebuilding for volume pvc-4998dc62020111e9 because there is rebuilding in process" (B 2018-12-18T06:39:27.37954157Z time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9-e-01cd1ba1\", UID:\"4bfb2c3e-0201-11e9-9f7b-000c298019f4\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25859483\", FieldPath:\"\"}): type: 'Normal' reason: 'Rebuilded' Detected replica pvc-4998dc62020111e9-r-c6c69ca8 (10.42.4.17) has been rebuilded" (B 2018-12-18T06:39:27.397440976Z time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858579\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-4998dc62020111e9 became healthy" (B 2018-12-18T06:39:27.406881184Z time="2018-12-18T06:39:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-4998dc62020111e9\", UID:\"4bf40025-0201-11e9-a88e-0050569082b6\", APIVersion:\"longhorn.rancher.io/v1alpha1\", ResourceVersion:\"25858579\", FieldPath:\"\"}): type: 'Normal' reason: 'Healthy' volume pvc-4998dc62020111e9 became healthy" (B 2018-12-18T06:39:27.41468329Z time="2018-12-18T06:39:27Z" level=debug msg="Requeue longhorn-system/pvc-4998dc62020111e9 due to conflict" (B 2018-12-18T06:39:36.819285554Z I1218 06:39:36.819149 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B 2018-12-18T06:39:36.819336985Z I1218 06:39:36.819149 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:496: forcing resync (B 2018-12-18T06:39:36.819964255Z I1218 06:39:36.819850 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: forcing resync (B 2018-12-18T06:39:51.819529372Z I1218 06:39:51.819364 1 reflector.go:286] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: forcing resync (B

[yasker]

@courcelm I think the network between your nodes has some issues:

2018-12-18T06:38:50.773563061Z I1218 06:38:50.773298 1 reflector.go:428] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:498: Watch close - *v1.StorageClass total 0 items received �(B�[m
2018-12-18T06:38:50.77356882Z I1218 06:38:50.771818 1 streamwatcher.go:107] Unable to decode an event from the watch stream: read tcp 10.42.10.237:58776->10.43.0.1:443: read: connection reset by peer �(B�[m
...
2018-12-18T06:38:50.807790342Z W1218 06:38:50.807622 1 reflector.go:341] github.com/kubernetes-csi/external-provisioner/vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:497: watch of *v1.PersistentVolume ended with: too old resource version: 25529521 (25858023) �(B�[m

(btw, can you put the log in a file or attachment later? it would be easy to look into)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.