Closed oomichi closed 5 years ago
2313 // testContainerOutputMatcher runs the given pod in the given namespace and waits
2314 // for all of the containers in the podSpec to move into the 'Success' status, and tests
2315 // the specified container log against the given expected output using the given matcher.
2316 func (f *Framework) testContainerOutputMatcher(scenarioName string,
2317 pod *v1.Pod,
2318 containerIndex int,
2319 expectedOutput []string,
2320 matcher func(string, ...interface{}) gomegatypes.GomegaMatcher) {
2321 By(fmt.Sprintf("Creating a pod to test %v", scenarioName))
★このメッセージは出力され、その後この関数内部で失敗している
2322 if containerIndex < 0 || containerIndex >= len(pod.Spec.Containers) {
2323 Failf("Invalid container index: %d", containerIndex)
2324 }
2325 ExpectNoError(f.MatchContainerOutput(pod, pod.Spec.Containers[containerIndex].Name, expectedOutput, matcher))
2326 }
上記コードは下記から呼ばれる。
653 func initVolumeContent(f *framework.Framework, pod *v1.Pod, volumeFilepath, subpathFilepath string) {
654 setWriteCommand(volumeFilepath, &pod.Spec.Containers[1])
655 setReadCommand(subpathFilepath, &pod.Spec.Containers[0])
656
657 By(fmt.Sprintf("Creating pod to write volume content %s", pod.Name))
658 f.TestContainerOutput("subpath", pod, 0, []string{
659 "content of file \"" + subpathFilepath + "\": mount-tester new file",
660 })
更に上記コードは問題のテストから呼ばれる
307 It("should support existing directories when readOnly specified in the volumeSource", func() {
308 roVol := vol.getReadOnlyVolumeSpec()
309 if roVol == nil {
310 framework.Skipf("Volume type %v doesn't support readOnly source", curVolType)
311 }
312
313 // Initialize content in the volume while it's writable
314 initVolumeContent(f, pod, filePathInVolume, filePathInSubpath)
問題のログ
Sep 19 21:17:59.929: INFO: Waiting up to 5m0s for pod "pod-subpath-test-nfspvc-f47f" in namespace "e2e-tests-subpath-zrdp6" to be "success or failure"
Sep 19 21:17:59.945: INFO: Pod "pod-subpath-test-nfspvc-f47f": Phase="Pending", Reason="", readiness=false. Elapsed: 15.530343ms
Sep 19 21:18:01.961: INFO: Pod "pod-subpath-test-nfspvc-f47f": Phase="Pending", Reason="", readiness=false. Elapsed: 2.031621855s
...
Sep 19 21:22:57.838: INFO: Pod "pod-subpath-test-nfspvc-f47f": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.909506018s
Sep 19 21:22:59.851: INFO: Pod "pod-subpath-test-nfspvc-f47f": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.922088849s
Sep 19 21:23:01.868: INFO: Failed to get logs from node "k8s-node01" pod "pod-subpath-test-nfspvc-f47f" container "test-container-subpath-nfspvc-f47f": the server rejected our request for an unknown reason (get pods pod-subpath-test-nfspvc-f47f)
Sep 19 21:23:01.877: INFO: Failed to get logs from node "k8s-node01" pod "pod-subpath-test-nfspvc-f47f" container "test-container-volume-nfspvc-f47f": the server rejected our request for an unknown reason (get pods pod-subpath-test-nfspvc-f47f)
作成した Pod が Pending のまま テスト中の Pod の状態を確認
$ kubectl get pods -n e2e-tests-subpath-shrdw
NAME READY STATUS RESTARTS AGE
nfs-server 1/1 Running 0 22s
pod-subpath-test-nfspvc-4tf5 0/2 Init:0/1 0 14s
$
$ kubectl describe pod pod-subpath-test-nfspvc-4tf5 -n e2e-tests-subpath-shrdw
Name: pod-subpath-test-nfspvc-4tf5
Namespace: e2e-tests-subpath-shrdw
Priority: 0
PriorityClassName: <none>
Node: k8s-node01/192.168.1.109
Start Time: Wed, 19 Sep 2018 22:04:30 +0000
Labels: <none>
Annotations: <none>
Status: Pending
IP:
Init Containers:
init-volume-nfspvc-4tf5:
Container ID:
Image: busybox
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/probe-volume from liveness-probe-volume (rw)
/test-volume from test-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7xrsn (ro)
Containers:
test-container-subpath-nfspvc-4tf5:
Container ID:
Image: gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0
Image ID:
Port: <none>
Host Port: <none>
Args:
--file_content_in_loop=/test-volume/test-file
--retry_time=10
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/probe-volume from liveness-probe-volume (rw)
/test-volume from test-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7xrsn (ro)
test-container-volume-nfspvc-4tf5:
Container ID:
Image: gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0
Image ID:
Port: <none>
Host Port: <none>
Args:
--new_file_0644=/test-volume/e2e-tests-subpath-shrdw/test-file
--file_mode=/test-volume/e2e-tests-subpath-shrdw/test-file
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/probe-volume from liveness-probe-volume (rw)
/test-volume from test-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7xrsn (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
test-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvc-c87lv
ReadOnly: false
liveness-probe-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-7xrsn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7xrsn
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32s default-scheduler Successfully assigned e2e-tests-subpath-shrdw/pod-subpath-test-nfspvc-4tf5 to k8s-node01
Warning FailedMount 31s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-ra21a1a843da047d0ae3eb67273fd617b.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
Warning FailedMount 31s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-rd477193a3ff64722b7b7b2d737f4edad.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
Warning FailedMount 30s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-r576540b5b75f4439b8b73eeff64ec514.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
Warning FailedMount 28s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-r0ccddff96f5f4f7b92d765f81be1f850.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
Warning FailedMount 24s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-r1ef60482f9d543b1826e6bf69cee6a42.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
Warning FailedMount 16s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-r4b9803d54e0b41e5a45ef1a74b05544d.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
$
Pod の Spec が環境にマッチしていない? Pod の Spec
363 return &v1.Pod{
364 ObjectMeta: metav1.ObjectMeta{
365 Name: fmt.Sprintf("pod-subpath-test-%s", suffix),
★Pod名がテスト環境と一致
366 Namespace: f.Namespace.Name,
367 },
368 Spec: v1.PodSpec{
369 InitContainers: []v1.Container{
370 {
371 Name: fmt.Sprintf("init-volume-%s", suffix),
372 Image: "busybox",
373 VolumeMounts: []v1.VolumeMount{
374 {
375 Name: volumeName,
376 MountPath: volumePath,
377 },
378 {
379 Name: probeVolumeName,
380 MountPath: probeVolumePath,
381 },
382 },
383 SecurityContext: &v1.SecurityContext{
384 Privileged: &privileged,
385 },
386 },
387 },
388 Containers: []v1.Container{
389 {
390 Name: fmt.Sprintf("test-container-subpath-%s", suffix),
391 Image: mountImage,
392 VolumeMounts: []v1.VolumeMount{
393 {
394 Name: volumeName,
395 MountPath: volumePath,
396 SubPath: subpath,
397 },
398 {
399 Name: probeVolumeName,
400 MountPath: probeVolumePath,
401 },
402 },
403 SecurityContext: &v1.SecurityContext{
404 Privileged: &privileged,
405 },
406 },
407 {
408 Name: fmt.Sprintf("test-container-volume-%s", suffix),
409 Image: mountImage,
410 VolumeMounts: []v1.VolumeMount{
411 {
412 Name: volumeName,
413 MountPath: volumePath,
414 },
415 {
416 Name: probeVolumeName,
417 MountPath: probeVolumePath,
418 },
419 },
420 SecurityContext: &v1.SecurityContext{
421 Privileged: &privileged,
422 },
423 },
424 },
425 RestartPolicy: v1.RestartPolicyNever,
426 TerminationGracePeriodSeconds: &gracePeriod,
427 Volumes: []v1.Volume{
428 {
429 Name: volumeName,
430 VolumeSource: *source,
431 },
432 {
433 Name: probeVolumeName,
434 VolumeSource: v1.VolumeSource{
435 EmptyDir: &v1.EmptyDirVolumeSource{},
436 },
437 },
438 },
439 SecurityContext: &v1.PodSecurityContext{
440 SELinuxOptions: &v1.SELinuxOptions{
441 Level: "s0:c0,c1",
442 },
443 },
444 },
InitContainerのマウント処理で失敗している模様
Warning FailedMount 31s kubelet, k8s-node01 MountVolume.SetUp failed for volume "nfs-7kxbp" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp --scope -- mount -t nfs 10.244.1.119:/exports /var/lib/kubelet/pods/fbc26075-bc57-11e8-a146-fa163e420595/volumes/kubernetes.io~nfs/nfs-7kxbp
Output: Running scope as unit run-ra21a1a843da047d0ae3eb67273fd617b.scope.
mount: wrong fs type, bad option, bad superblock on 10.244.1.119:/exports,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
https://askubuntu.com/questions/525243/why-do-i-get-wrong-fs-type-bad-option-bad-superblock-error を参考に k8s-master、k8s-node01 に nfs-common をインストールしてみた。 通るようになった。
まとめ