[BUG] longhorn fails to attach volume on minikube

[Vilsol]

Describe the bug When running longhorn in minikube, it fails to attach a PVC to a pod.

To Reproduce

minikube config set driver docker
minikube start
minikube ssh "sudo apt update && sudo apt install -y open-iscsi"
git clone https://github.com/longhorn/longhorn.git
kubectl create namespace longhorn-system || true
helm upgrade --install --namespace longhorn-system longhorn ./longhorn/chart/ --set persistence.defaultClassReplicaCount=1 --set defaultSettings.defaultReplicaCount=1

# Wait for all pods to go up

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-claim
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: test
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: test-volume
  volumes:
    - name: test-volume
      persistentVolumeClaim:
        claimName: test-claim
EOF

Expected behavior PVC to be attached to the pod

Log

time="2020-08-18T12:06:44Z" level=info msg="Start overwriting built-in settings with customized values"
time="2020-08-18T12:06:44Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
time="2020-08-18T12:06:44Z" level=info msg="Start upgrading"
time="2020-08-18T12:06:44Z" level=warning msg="Cannot verify current CRD version, assume it's not v1alpha1: unable to verify if version matches v1alpha1: settings.longhorn.rancher.io \"default-engine-image\" is forbidden: User \"system:serviceaccount:longhorn-system:longhorn-service-account\" cannot get resource \"settings\" in API group \"longhorn.rancher.io\" in the namespace \"longhorn-system\""
time="2020-08-18T12:06:44Z" level=info msg="Initialized CRD API Version to longhorn.io/v1beta1"
time="2020-08-18T12:06:44Z" level=info msg="Finish upgrading"
time="2020-08-18T12:06:44Z" level=info msg="Upgrade leader lost: minikube"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn replica controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn engine controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn websocket controller"
time="2020-08-18T12:06:44Z" level=info msg="Starting Longhorn instance manager controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn Setting controller"
time="2020-08-18T12:06:44Z" level=info msg="Start kubernetes controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn node controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn Kubernetes node controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn Engine Image controller"
time="2020-08-18T12:06:44Z" level=info msg="Start Longhorn volume controller"
time="2020-08-18T12:06:47Z" level=info msg="Latest Longhorn version is v1.0.2"
time="2020-08-18T12:06:47Z" level=debug msg="Cannot update latest Longhorn version: settings.longhorn.io \"latest-longhorn-version\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update"
time="2020-08-18T12:06:48Z" level=debug msg="Updated setting default-engine-image to longhornio/longhorn-engine:v1.0.2"
time="2020-08-18T12:06:48Z" level=debug msg="Updated setting default-instance-manager-image to longhornio/longhorn-instance-manager:v1_20200514"
time="2020-08-18T12:06:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"minikube\", UID:\"77dd064e-baa2-4f34-a754-b1cfa8dba633\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"511\", FieldPath:\"\"}): type: 'Warning' reason: 'Ready' Node minikube is down: the manager pod longhorn-manager-xbs4p is not running"
time="2020-08-18T12:06:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"minikube\", UID:\"77dd064e-baa2-4f34-a754-b1cfa8dba633\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"511\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' "
time="2020-08-18T12:06:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"minikube\", UID:\"77dd064e-baa2-4f34-a754-b1cfa8dba633\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"511\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Disk default-disk-23105261ac064cfd(/var/lib/longhorn/) on node minikube is ready"
time="2020-08-18T12:06:49Z" level=debug msg="Prepare to create default instance manager instance-manager-e-55361ddd, node: minikube, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: engine"
time="2020-08-18T12:06:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"minikube\", UID:\"77dd064e-baa2-4f34-a754-b1cfa8dba633\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"511\", FieldPath:\"\"}): type: 'Normal' reason: 'Schedulable' Disk default-disk-23105261ac064cfd(/var/lib/longhorn/) on node minikube is schedulable"
time="2020-08-18T12:06:49Z" level=debug msg="Prepare to create default instance manager instance-manager-r-f891ce56, node: minikube, default instance manager image: longhornio/longhorn-instance-manager:v1_20200514, type: replica"
time="2020-08-18T12:06:49Z" level=debug msg="Created engine image ei-ee18f965 (longhornio/longhorn-engine:v1.0.2)"
time="2020-08-18T12:06:49Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:06:49Z" level=debug msg="Instance Manager Controller minikube picked up instance-manager-e-55361ddd"
time="2020-08-18T12:06:49Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-55361ddd with state stopped"
time="2020-08-18T12:06:50Z" level=debug msg="Engine Image Controller minikube picked up ei-ee18f965 (longhornio/longhorn-engine:v1.0.2)"
time="2020-08-18T12:06:50Z" level=info msg="Created daemon set engine-image-ei-ee18f965 for engine image ei-ee18f965 (longhornio/longhorn-engine:v1.0.2)"
time="2020-08-18T12:06:50Z" level=info msg="Created instance manager pod instance-manager-e-55361ddd for instance manager instance-manager-e-55361ddd"
time="2020-08-18T12:06:50Z" level=debug msg="Instance Manager Controller minikube picked up instance-manager-r-f891ce56"
time="2020-08-18T12:06:50Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-f891ce56 with state stopped"
time="2020-08-18T12:06:51Z" level=debug msg="Requeue longhorn-system/instance-manager-e-55361ddd due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-e-55361ddd\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-18T12:06:51Z" level=error msg="Instance Manager instance-manager-e-55361ddd is state error but the related pod is pending."
time="2020-08-18T12:06:51Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-55361ddd with state error"
time="2020-08-18T12:06:51Z" level=info msg="Created instance manager pod instance-manager-r-f891ce56 for instance manager instance-manager-r-f891ce56"
time="2020-08-18T12:06:52Z" level=debug msg="Requeue longhorn-system/instance-manager-r-f891ce56 due to conflict: Operation cannot be fulfilled on instancemanagers.longhorn.io \"instance-manager-r-f891ce56\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-18T12:06:52Z" level=error msg="Instance Manager instance-manager-r-f891ce56 is state error but the related pod is pending."
time="2020-08-18T12:06:52Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-f891ce56 with state error"
time="2020-08-18T12:06:52Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-55361ddd with state error"
time="2020-08-18T12:06:53Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-f891ce56 with state error"
time="2020-08-18T12:06:54Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-e-55361ddd with state error"
time="2020-08-18T12:06:54Z" level=info msg="Created instance manager pod instance-manager-e-55361ddd for instance manager instance-manager-e-55361ddd"
time="2020-08-18T12:06:55Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:06:56Z" level=warning msg="Starts to clean up then recreates pod for instance manager instance-manager-r-f891ce56 with state error"
time="2020-08-18T12:06:56Z" level=info msg="Created instance manager pod instance-manager-r-f891ce56 for instance manager instance-manager-r-f891ce56"
time="2020-08-18T12:07:01Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:07:07Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:07:13Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:07:19Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v1.0.2 to be ready"
time="2020-08-18T12:07:23Z" level=info msg="Event(v1.ObjectReference{Kind:\"EngineImage\", Namespace:\"longhorn-system\", Name:\"ei-ee18f965\", UID:\"f936de58-7f32-448c-819e-ea4655347ec2\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"543\", FieldPath:\"\"}): type: 'Normal' reason: 'ready' Engine image ei-ee18f965 (longhornio/longhorn-engine:v1.0.2) become ready"
time="2020-08-18T12:07:25Z" level=debug msg="Engine image longhornio/longhorn-engine:v1.0.2 is ready"
time="2020-08-18T12:07:25Z" level=info msg="Listening on 172.18.0.3:9500"
time="2020-08-18T12:07:26Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"longhorn-system\", Name:\"minikube\", UID:\"77dd064e-baa2-4f34-a754-b1cfa8dba633\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"665\", FieldPath:\"\"}): type: 'Normal' reason: 'Ready' Node minikube is ready"
172.18.0.5 - - [18/Aug/2020:12:07:26 +0000] "GET /v1 HTTP/1.1" 200 3133 "" "curl/7.58.0"
172.18.0.5 - - [18/Aug/2020:12:07:27 +0000] "GET /v1 HTTP/1.1" 200 3133 "" "Go-http-client/1.1"
time="2020-08-18T12:07:38Z" level=debug msg="Start monitoring instance manager instance-manager-e-55361ddd"
time="2020-08-18T12:07:40Z" level=debug msg="Start monitoring instance manager instance-manager-r-f891ce56"
172.18.0.18 - - [18/Aug/2020:12:08:16 +0000] "GET /v1 HTTP/1.1" 200 3133 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:08:59 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 404 0 "" "Go-http-client/1.1"
time="2020-08-18T12:08:59Z" level=debug msg="Volume controller minikube picked up pvc-5566bdfe-8158-422b-baaf-0c453c368a27"
time="2020-08-18T12:08:59Z" level=debug msg="Replica controller minikube picked up pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:08:59Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d state, old state , new state stopped"
time="2020-08-18T12:08:59Z" level=debug msg="Created volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27: {Size:1073741824 Frontend:blockdev FromBackup: NumberOfReplicas:1 StaleReplicaTimeout:30 NodeID: EngineImage:longhornio/longhorn-engine:v1.0.2 RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
172.18.0.18 - - [18/Aug/2020:12:08:59 +0000] "POST /v1/volumes HTTP/1.1" 200 971 "" "Go-http-client/1.1"
time="2020-08-18T12:08:59Z" level=debug msg="Schedule replica pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d to node minikube, disk default-disk-23105261ac064cfd, datapath /var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98"
time="2020-08-18T12:08:59Z" level=debug msg="Engine controller minikube picked up pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:08:59Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state , new state stopped"
time="2020-08-18T12:08:59Z" level=debug msg="Requeue longhorn-system/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 due to error <nil> or Operation cannot be fulfilled on engines.longhorn.io \"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\": the object has been modified; please apply your changes to the latest version and try again"
time="2020-08-18T12:08:59Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27\", UID:\"363a234c-8510-4976-ab21-c39d72026a26\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1026\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 has been detached"
172.18.0.18 - - [18/Aug/2020:12:09:01 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 3021 "" "Go-http-client/1.1"
time="2020-08-18T12:09:01Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27\", UID:\"363a234c-8510-4976-ab21-c39d72026a26\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1038\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Persistent Volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 started to use/reuse Longhorn volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27"
172.18.0.18 - - [18/Aug/2020:12:09:02 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 3161 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:02 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
time="2020-08-18T12:09:02Z" level=debug msg="Attaching volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 to minikube with disableFrontend set false"
172.18.0.18 - - [18/Aug/2020:12:09:02 +0000] "POST /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27?action=attach HTTP/1.1" 200 2458 "" "Go-http-client/1.1"
time="2020-08-18T12:09:02Z" level=debug msg="Prepare to create instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:02Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1060\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:03Z" level=debug msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d starts running, IP 172.18.0.8"
time="2020-08-18T12:09:03Z" level=debug msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d starts running, Port 10000"
time="2020-08-18T12:09:03Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d state, old state stopped, new state running"
time="2020-08-18T12:09:03Z" level=debug msg="Prepare to create instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1067\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:03Z" level=debug msg="Instance process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 had been created, need to wait for instance manager update"
time="2020-08-18T12:09:04Z" level=debug msg="Instance process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 had been created, need to wait for instance manager update"
time="2020-08-18T12:09:04Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state stopped, new state starting"
172.18.0.18 - - [18/Aug/2020:12:09:04 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:06 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:08 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:10 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:12 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:14 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2462 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:16 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:17 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:17 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:18 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:18 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:18 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:18 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:18 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:20 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:22 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:22 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:22 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:22 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:22 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:24 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:26 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:28 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
time="2020-08-18T12:09:30Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:09:30Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state starting, new state error"
time="2020-08-18T12:09:30Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Starting with replicas [\\\"tcp://172.18.0.8:10000\\\"]\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Connecting to remote: 172.18.0.8:10000\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Opening: 172.18.0.8:10000\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Adding backend: tcp://172.18.0.8:10000\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Start monitoring tcp://172.18.0.8:10000\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Get backend tcp://172.18.0.8:10000 revision counter 0\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"device pvc-5566bdfe-8158-422b-baaf-0c453c368a27: SCSI device /dev/longhorn/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 shutdown\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: go-iscsi-helper: tgtd is already running"
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"go-iscsi-helper: found available target id 1\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"New data socket connnection established\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:29Z\" level=error msg=\"Failed to startup frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found\\n, error exit status 21\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: 2020/08/18 12:09:29 failed to start up frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found"
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: , error exit status 21"
time="2020-08-18T12:09:30Z" level=warning msg="Engine of volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 dead unexpectedly, reattach the volume"
time="2020-08-18T12:09:30Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27\", UID:\"363a234c-8510-4976-ab21-c39d72026a26\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1061\", FieldPath:\"\"}): type: 'Warning' reason: 'DetachedUnexpectly' Engine of volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 dead unexpectedly, reattach the volume"
time="2020-08-18T12:09:30Z" level=warning msg="Try to get requested log for pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d on node minikube"
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Creating volume /host/var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98, size 1073741824/512\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on data server 0.0.0.0:10001\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on sync agent server 0.0.0.0:10002\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on gRPC Replica server 0.0.0.0:10000\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on sync 0.0.0.0:10002\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:03Z\" level=info msg=\"New connection from: 172.18.0.7:41888\""
time="2020-08-18T12:09:30Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Opening volume /host/var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98, size 1073741824/512\""
time="2020-08-18T12:09:31Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:31Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:09:31Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:09:31Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1123\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:31Z" level=error msg="failed to receive log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:31Z" level=warning msg="cannot get crash log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 on Instance Manager instance-manager-e-55361ddd at minikube, error rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:31Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state error, new state stopped"
time="2020-08-18T12:09:32Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1133\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:32Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:32Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1136\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
172.18.0.18 - - [18/Aug/2020:12:09:32 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2200 "" "Go-http-client/1.1"
time="2020-08-18T12:09:33Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d state, old state running, new state stopped"
time="2020-08-18T12:09:33Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27\", UID:\"363a234c-8510-4976-ab21-c39d72026a26\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1114\", FieldPath:\"\"}): type: 'Normal' reason: 'Detached' volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 has been detached"
time="2020-08-18T12:09:34Z" level=debug msg="Prepare to create instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:09:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1150\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
172.18.0.18 - - [18/Aug/2020:12:09:34 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 3161 "" "Go-http-client/1.1"
time="2020-08-18T12:09:35Z" level=debug msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d starts running, IP 172.18.0.8"
time="2020-08-18T12:09:35Z" level=debug msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d starts running, Port 10000"
time="2020-08-18T12:09:35Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d state, old state stopped, new state running"
time="2020-08-18T12:09:36Z" level=debug msg="Prepare to create instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:09:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1158\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
172.18.0.18 - - [18/Aug/2020:12:09:36 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2437 "" "Go-http-client/1.1"
time="2020-08-18T12:09:37Z" level=debug msg="Instance process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 had been created, need to wait for instance manager update"
time="2020-08-18T12:09:37Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state stopped, new state starting"
172.18.0.18 - - [18/Aug/2020:12:09:38 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:38 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:38 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:38 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:38 +0000] "GET /v1/nodes/minikube HTTP/1.1" 200 1606 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:40 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:42 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:44 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:46 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:48 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:50 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:52 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:54 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2463 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:56 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:09:58 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:10:00 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
172.18.0.18 - - [18/Aug/2020:12:10:02 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2464 "" "Go-http-client/1.1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:10:03Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state starting, new state error"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Starting with replicas [\\\"tcp://172.18.0.8:10000\\\"]\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Connecting to remote: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Opening: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Adding backend: tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Start monitoring tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Get backend tcp://172.18.0.8:10000 revision counter 0\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"device pvc-5566bdfe-8158-422b-baaf-0c453c368a27: SCSI device /dev/longhorn/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 shutdown\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: go-iscsi-helper: tgtd is already running"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"go-iscsi-helper: found available target id 1\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"New data socket connnection established\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:29Z\" level=error msg=\"Failed to startup frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found\\n, error exit status 21\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: 2020/08/18 12:09:29 failed to start up frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: , error exit status 21"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Starting with replicas [\\\"tcp://172.18.0.8:10000\\\"]\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Connecting to remote: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Opening: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Adding backend: tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Start monitoring tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Get backend tcp://172.18.0.8:10000 revision counter 0\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Shutdown SCSI target iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"device pvc-5566bdfe-8158-422b-baaf-0c453c368a27: SCSI device /dev/longhorn/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 shutdown\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: go-iscsi-helper: tgtd is already running"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:37Z\" level=info msg=\"go-iscsi-helper: found available target id 1\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:37Z\" level=info msg=\"New data socket connnection established\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:39Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:39Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:44Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:44Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:49Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:49Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:54Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:54Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:59Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:59Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:10:02Z\" level=error msg=\"Failed to startup frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found\\n, error exit status 21\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: 2020/08/18 12:10:02 failed to start up frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: , error exit status 21"
time="2020-08-18T12:10:03Z" level=warning msg="Engine of volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 dead unexpectedly, reattach the volume"
time="2020-08-18T12:10:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27\", UID:\"363a234c-8510-4976-ab21-c39d72026a26\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1153\", FieldPath:\"\"}): type: 'Warning' reason: 'DetachedUnexpectly' Engine of volume pvc-5566bdfe-8158-422b-baaf-0c453c368a27 dead unexpectedly, reattach the volume"
time="2020-08-18T12:10:03Z" level=warning msg="Try to get requested log for pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d on node minikube"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Creating volume /host/var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98, size 1073741824/512\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on data server 0.0.0.0:10001\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on sync agent server 0.0.0.0:10002\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on gRPC Replica server 0.0.0.0:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:02Z\" level=info msg=\"Listening on sync 0.0.0.0:10002\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:03Z\" level=info msg=\"New connection from: 172.18.0.7:41888\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Opening volume /host/var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98, size 1073741824/512\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:32Z\" level=warning msg=\"Received signal interrupt to shutdown\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:32Z\" level=warning msg=\"Starting to execute registered shutdown func github.com/longhorn/longhorn-engine/app/cmd.startReplica.func4\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:34Z\" level=info msg=\"Listening on data server 0.0.0.0:10001\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:34Z\" level=info msg=\"Listening on sync agent server 0.0.0.0:10002\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:34Z\" level=info msg=\"Listening on gRPC Replica server 0.0.0.0:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:34Z\" level=info msg=\"Listening on sync 0.0.0.0:10002\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:36Z\" level=info msg=\"New connection from: 172.18.0.7:42716\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Opening volume /host/var/lib/longhorn/replicas/pvc-5566bdfe-8158-422b-baaf-0c453c368a27-11f6df98, size 1073741824/512\""
time="2020-08-18T12:10:03Z" level=warning msg="Try to get requested log for pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 on node minikube"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Starting with replicas [\\\"tcp://172.18.0.8:10000\\\"]\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Connecting to remote: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Opening: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Adding backend: tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Start monitoring tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"Get backend tcp://172.18.0.8:10000 revision counter 0\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:03Z\" level=info msg=\"device pvc-5566bdfe-8158-422b-baaf-0c453c368a27: SCSI device /dev/longhorn/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 shutdown\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: go-iscsi-helper: tgtd is already running"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"go-iscsi-helper: found available target id 1\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:04Z\" level=info msg=\"New data socket connnection established\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\niscsiadm: Could not scan /sys/class/iscsi_transport.\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:06Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:11Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:16Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:21Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:26Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:29Z\" level=error msg=\"Failed to startup frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found\\n, error exit status 21\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: 2020/08/18 12:09:29 failed to start up frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: , error exit status 21"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Starting with replicas [\\\"tcp://172.18.0.8:10000\\\"]\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Connecting to remote: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Opening: 172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Adding backend: tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Start monitoring tcp://172.18.0.8:10000\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Get backend tcp://172.18.0.8:10000 revision counter 0\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"Shutdown SCSI target iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:36Z\" level=info msg=\"device pvc-5566bdfe-8158-422b-baaf-0c453c368a27: SCSI device /dev/longhorn/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 shutdown\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: go-iscsi-helper: tgtd is already running"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:37Z\" level=info msg=\"go-iscsi-helper: found available target id 1\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:37Z\" level=info msg=\"New data socket connnection established\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:39Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:39Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:44Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:44Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:49Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:49Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:54Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:54Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:59Z\" level=warning msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:09:59Z\" level=warning msg=\"Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: time=\"2020-08-18T12:10:02Z\" level=error msg=\"Failed to startup frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found\\n, error exit status 21\""
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: 2020/08/18 12:10:02 failed to start up frontend: Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m node -T iqn.2019-10.io.longhorn:pvc-5566bdfe-8158-422b-baaf-0c453c368a27 -p 172.18.0.7 --login], output , stderr, iscsiadm: No records found"
time="2020-08-18T12:10:03Z" level=warning msg="pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: , error exit status 21"
time="2020-08-18T12:10:03Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:10:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1196\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=error msg="failed to receive log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="cannot get crash log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 on Instance Manager instance-manager-e-55361ddd at minikube, error rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:10:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1204\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=error msg="failed to receive log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="cannot get crash log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 on Instance Manager instance-manager-e-55361ddd at minikube, error rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 is state error, error message: exit status 1"
time="2020-08-18T12:10:03Z" level=warning msg="Instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 crashed on Instance Manager instance-manager-e-55361ddd at minikube, try to get log"
time="2020-08-18T12:10:03Z" level=info msg="Event(v1.ObjectReference{Kind:\"Engine\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1\", UID:\"0db606ab-3c9a-446b-9c81-7e5a1b831a95\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1207\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=error msg="failed to receive log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1: rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:03Z" level=warning msg="cannot get crash log for instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 on Instance Manager instance-manager-e-55361ddd at minikube, error rpc error: code = NotFound desc = cannot find process pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1"
time="2020-08-18T12:10:04Z" level=debug msg="Instance handler updated instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-e-8971fbd1 state, old state error, new state stopped"
time="2020-08-18T12:10:04Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:10:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1214\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
172.18.0.18 - - [18/Aug/2020:12:10:04 +0000] "GET /v1/volumes/pvc-5566bdfe-8158-422b-baaf-0c453c368a27 HTTP/1.1" 200 2200 "" "Go-http-client/1.1"
time="2020-08-18T12:10:04Z" level=debug msg="Prepare to delete instance pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"
time="2020-08-18T12:10:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d\", UID:\"da0e3c78-36c7-4e1e-a3e6-65c2655c1603\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"1216\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops pvc-5566bdfe-8158-422b-baaf-0c453c368a27-r-f57ce09d"

Environment:

• Longhorn version: 1.0.2
• Kubernetes version: 1.18.3
• Node OS type and version: minikube v1.12.1 on Ubuntu 19.10 using docker driver.

[shuo-wu]

Here is the key error log:

msg=\"FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/118/ns/mnt --net=/host/proc/118/ns/net iscsiadm -m discovery -t sendtargets -p 172.18.0.7], output , stderr, Failed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\nFailed to connect to bus: No data available\\niscsiadm: can not connect to iSCSI daemon (111)!\\niscsiadm: Cannot perform discovery. Initiatorname required.\\niscsiadm: Could not perform SendTargets discovery: could not connect to iscsid\\n, error exit status 20\""

I am not sure if this is caused by the iscsi version issue, or there is something wrong with the iscsiadm setup. Notice that the OS version you are using is Ubuntu 19.10 but Longhorn is using Ubuntu 18.04

[yasker]

iscsiadm: Cannot perform discovery. Initiatorname required.

This seems due to the iscsid/open-iscsi wasn't configured correctly on the node. The initiator name wasn't set.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.