kubernetes/pull/75616 失敗調査

$ go run hack/e2e.go -- --provider=skeleton --test --test_args="--ginkgo.focus=should\sremove\sfrom\sactive\slist\sjobs\sthat\shave\sbeen\sdeleted" --check-version-skew=false
..
Mar 28 22:00:16.342: INFO:
Latency metrics for node k8s-master
Mar 28 22:00:16.344: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "cronjob-1384" for this suite.
Mar 28 22:00:22.428: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Mar 28 22:00:22.662: INFO: namespace cronjob-1384 deletion completed in 6.289438672s

~ Failure [39.299 seconds]
[sig-apps] CronJob
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/framework.go:22
  should remove from active list jobs that have been deleted [It]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/cronjob.go:192

  Failed to ensure missing job event has occurred for forbid cronjob in namespace cronjob-1384
  Unexpected error:
      <*runtime.notRegisteredErr | 0xc0020ac840>: {
          schemeName: "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29",
          gvk: {Group: "", Version: "", Kind: ""},
          target: nil,
          t: {
              size: 0x550,
              ptrdata: 0x550,
              hash: 3973304303,
              tflag: 7,
              align: 8,
              fieldAlign: 8,
              kind: 25,
              alg: {hash: nil, equal: nil},
              gcdata: 85,
              str: 488165,
              ptrToThis: 15753728,
          },
      }
      no kind is registered for the type v1beta1.CronJob in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29"
  occurred

  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/cronjob.go:223
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMar 28 22:00:22.711: INFO: Running AfterSuite actions on all nodes
Mar 28 22:00:22.711: INFO: Running AfterSuite actions on node 1

------------------------------------------------------------

テストシナリオの把握

削除されたジョブは Active list からも削除されることを確認するテスト
1. CronJob の作成
2. 作成したCron Jobがアクティブになるまで待つ
3. 作成したCron Jobの状態がActive 担っていることを確認する
4. Cron Jobを削除する
5. 削除したCron Job に対する Get が NotFound になることを確認する
6. 削除したCron Job がActiveリスト上にもないことを確認する
7. MissingJob のイベントが発生するまで待つ　★ココでエラーになった原因はエラーが発生すべきではないのに発生したため

191         // deleted jobs should be removed from the active list
192         It("should remove from active list jobs that have been deleted", func() {
193                 By("Creating a ForbidConcurrent cronjob")
194                 cronJob := newTestCronJob("forbid", "*/1 * * * ?", batchv1beta1.ForbidConcurrent,
195                         sleepCommand, nil)
196                 cronJob, err := createCronJob(f.ClientSet, f.Namespace.Name, cronJob)
197                 Expect(err).NotTo(HaveOccurred(), "Failed to create CronJob in namespace %s", f.Namespace.Name)
198
199                 By("Ensuring a job is scheduled")
200                 err = waitForActiveJobs(f.ClientSet, f.Namespace.Name, cronJob.Name, 1)
201                 Expect(err).NotTo(HaveOccurred(), "Failed to ensure a %s cronjob is scheduled in namespace %s", cronJob.Name, f.Namespace.Name)
202
203                 By("Ensuring exactly one is scheduled")
204                 cronJob, err = getCronJob(f.ClientSet, f.Namespace.Name, cronJob.Name)
205                 Expect(err).NotTo(HaveOccurred(), "Failed to ensure exactly one %s cronjob is scheduled in namespace %s", cronJob.Name, f.Namespace.Name)
206                 Expect(cronJob.Status.Active).Should(HaveLen(1))
207
208                 By("Deleting the job")
209                 job := cronJob.Status.Active[0]
210                 framework.ExpectNoError(framework.DeleteResourceAndWaitForGC(f.ClientSet, batchinternal.Kind("Job"), f.Namespace.Name, job.Name))
211
212                 By("Ensuring job was deleted")
213                 _, err = framework.GetJob(f.ClientSet, f.Namespace.Name, job.Name)
214                 Expect(err).To(HaveOccurred())
215                 Expect(errors.IsNotFound(err)).To(BeTrue())
216
217                 By("Ensuring the job is not in the cronjob active list")
218                 err = waitForJobNotActive(f.ClientSet, f.Namespace.Name, cronJob.Name, job.Name)
219                 Expect(err).NotTo(HaveOccurred(), "Failed to ensure the %s cronjob is not in active list in namespace %s", cronJob.Name, f.Namespace.Name    )
220
221                 By("Ensuring MissingJob event has occurred")
222                 err = waitForEventWithReason(f.ClientSet, f.Namespace.Name, cronJob.Name, []string{"MissingJob"})
223                 Expect(err).NotTo(HaveOccurred(), "Failed to ensure missing job event has occurred for %s cronjob in namespace %s", cronJob.Name, f.Names    pace.Name)  ★ココでエラー

events, err := c.CoreV1().Events(ns).Search(legacyscheme.Scheme, sj) の呼び出しが、

      <*runtime.notRegisteredErr | 0xc001c75bc0>: {
          schemeName: "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29",
          gvk: {Group: "", Version: "", Kind: ""},
          target: nil,
          t: {
              size: 0x550,
              ptrdata: 0x550,
              hash: 3973304303,
              tflag: 7,
              align: 8,
              fieldAlign: 8,
              kind: 25,
              alg: {hash: nil, equal: nil},
              gcdata: 85,
              str: 488165,
              ptrToThis: 15753728,
          },
      }
      no kind is registered for the type v1beta1.CronJob in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29"

でエラーになった模様。

k8s.io/kubernetes/pkg/api/legacyscheme legacyscheme.Scheme

// Scheme is the default instance of runtime.Scheme to which types in the Kubernetes API are already registered.
// NOTE: If you are copying this file to start a new api group, STOP! Copy the
// extensions group instead. This Scheme is special and should appear ONLY in
// the api group, unless you really know what you're doing.
// TODO(lavalamp): make the above error impossible.
var Scheme = runtime.NewScheme()

どうも legacyscheme.Scheme が怪しいのではという話になっている。 PR適応前の値

PR適応後の値

Mar 28 23:33:47.303: INFO: legacyscheme.Scheme &{map[{ __internal Binding}:0x4470a60 { __internal ComponentStatus}:0x4470b20 { __internal ComponentStatusList}:0x4470be0 { __internal ConfigMap}:0x4565820 { __internal ConfigMapList}:0x4470d60 { __internal Endpoints}:0x4471120 { __internal EndpointsList}:0x44711e0 { __internal Event}:0x4917960 { __internal EventList}:0x4471420 { __internal LimitRange}:0x44718a0 { __internal LimitRangeList}:0x4471960 { __internal List}:0x4471a20 { __internal Namespace}:0x4565d60 { __internal NamespaceList}:0x4471ba0 { __internal Node}:0x4565e20 { __internal NodeList}:0x4471c60 { __internal NodeProxyOptions}:0x441b5c0 { __internal PersistentVolume}:0x4565fa0 { __internal PersistentVolumeClaim}:0x4566060 { __internal PersistentVolumeClaimList}:0x4471de0 { __internal PersistentVolumeList}:0x4471ea0 { __internal Pod}:0x45661e0 { __internal PodAttachOptions}:0x474e820 { __internal PodExecOptions}:0x47be240 { __internal PodList}:0x44721a0 { __internal PodLogOptions}:0x4877900 { __internal PodPortForwardOptions}:0x441b660 { __internal PodProxyOptions}:0x441b700 { __internal PodStatusResult}:0x4472260 { __internal PodTemplate}:0x4472320 { __internal PodTemplateList}:0x44723e0 { __internal RangeAllocation}:0x4566360 { __internal ReplicationController}:0x4566420 { __internal ReplicationControllerList}:0x4472560 { __internal ResourceQuota}:0x45665a0 { __internal ResourceQuotaList}:0x44726e0 { __internal Secret}:0x4566720 { __internal SecretList}:0x44729e0 { __internal SerializedReference}:0x441b840 { __internal Service}:0x45668a0 { __internal ServiceAccount}:0x4635520 { __internal ServiceAccountList}:0x4472b60 { __internal ServiceList}:0x4472ce0 { __internal ServiceProxyOptions}:0x441b8e0 { __internal WatchEvent}:0x43bbbc0 { v1 APIGroup}:0x46c1ce0 { v1 APIGroupList}:0x44249e0 { v1 APIResourceList}:0x455fce0 { v1 APIVersions}:0x4691160 { v1 Binding}:0x455a820 { v1 ComponentStatus}:0x455a8e0 { v1 ComponentStatusList}:0x455a9a0 { v1 ConfigMap}:0x461cae0 { v1 ConfigMapList}:0x455ab20 { v1 CreateOptions}:0x455fda0 { v1 DeleteOptions}:0x4752320 { v1 Endpoints}:0x455aee0 { v1 EndpointsList}:0x455afa0 { v1 Event}:0x49274a0 { v1 EventList}:0x455b1e0 { v1 ExportOptions}:0x455fe60 { v1 GetOptions}:0x4424a80 { v1 LimitRange}:0x455b660 { v1 LimitRangeList}:0x455b720 { v1 List}:0x44e6960 { v1 ListOptions}:0x48208e0 { v1 Namespace}:0x461d100 { v1 NamespaceList}:0x455b8a0 { v1 Node}:0x461d1e0 { v1 NodeList}:0x455b960 { v1 NodeProxyOptions}:0x44221e0 { v1 PatchOptions}:0x461fb00 { v1 PersistentVolume}:0x461d3a0 { v1 PersistentVolumeClaim}:0x461d480 { v1 PersistentVolumeClaimList}:0x455bae0 { v1 PersistentVolumeList}:0x455bba0 { v1 Pod}:0x461d640 { v1 PodAttachOptions}:0x4751820 { v1 PodExecOptions}:0x47c00a0 { v1 PodList}:0x455bea0 { v1 PodLogOptions}:0x48791c0 { v1 PodPortForwardOptions}:0x44226e0 { v1 PodProxyOptions}:0x4422780 { v1 PodStatusResult}:0x455bf60 { v1 PodTemplate}:0x455c020 { v1 PodTemplateList}:0x455c0e0 { v1 RangeAllocation}:0x461d800 { v1 ReplicationController}:0x461d8e0 { v1 ReplicationControllerList}:0x455c260 { v1 ResourceQuota}:0x461daa0 { v1 ResourceQuotaList}:0x455c3e0 { v1 Secret}:0x46c0de0 { v1 SecretList}:0x455c6e0 { v1 SerializedReference}:0x4422c80 { v1 Service}:0x461dd40 { v1 ServiceAccount}:0x46c0ee0 { v1 ServiceAccountList}:0x455c860 { v1 ServiceList}:0x455c9e0 { v1 ServiceProxyOptions}:0x4422d20 { v1 Status}:0x47c02e0 { v1 UpdateOptions}:0x45602e0 { v1 WatchEvent}:0x43bbda0] map[0x43bbbc0:[{ __internal WatchEvent}] 0x43bbda0:[{ v1 WatchEvent}] 0x441b5c0:[{ __internal NodeProxyOptions}] 0x441b660:[{ __internal PodPortForwardOptions}] 0x441b700:[{ __internal PodProxyOptions}] 0x441b840:[{ __internal SerializedReference}] 0x441b8e0:[{ __internal ServiceProxyOptions}] 0x44221e0:[{ v1 NodeProxyOptions}] 0x44226e0:[{ v1 PodPortForwardOptions}] 0x4422780:[{ v1 PodProxyOptions}] 0x4422c80:[{ v1 SerializedReference}] 0x4422d20:[{ v1 ServiceProxyOptions}] 0x44249e0:[{ v1 APIGroupList}] 0x4424a80:[{ v1 GetOptions}] 0x4470a60:[{ __internal Binding}] 0x4470b20:[{ __internal ComponentStatus}] 0x4470be0:[{ __internal ComponentStatusList}] 0x4470d60:[{ __internal ConfigMapList}] 0x4471120:[{ __internal Endpoints}] 0x44711e0:[{ __internal EndpointsList}] 0x4471420:[{ __internal EventList}] 0x44718a0:[{ __internal LimitRange}] 0x4471960:[{ __internal LimitRangeList}] 0x4471a20:[{ __internal List}] 0x4471ba0:[{ __internal NamespaceList}] 0x4471c60:[{ __internal NodeList}] 0x4471de0:[{ __internal PersistentVolumeClaimList}] 0x4471ea0:[{ __internal PersistentVolumeList}] 0x44721a0:[{ __internal PodList}] 0x4472260:[{ __internal PodStatusResult}] 0x4472320:[{ __internal PodTemplate}] 0x44723e0:[{ __internal PodTemplateList}] 0x4472560:[{ __internal ReplicationControllerList}] 0x44726e0:[{ __internal ResourceQuotaList}] 0x44729e0:[{ __internal SecretList}] 0x4472b60:[{ __internal ServiceAccountList}] 0x4472ce0:[{ __internal ServiceList}] 0x44e6960:[{ v1 List}] 0x455a820:[{ v1 Binding}] 0x455a8e0:[{ v1 ComponentStatus}] 0x455a9a0:[{ v1 ComponentStatusList}] 0x455ab20:[{ v1 ConfigMapList}] 0x455aee0:[{ v1 Endpoints}] 0x455afa0:[{ v1 EndpointsList}] 0x455b1e0:[{ v1 EventList}] 0x455b660:[{ v1 LimitRange}] 0x455b720:[{ v1 LimitRangeList}] 0x455b8a0:[{ v1 NamespaceList}] 0x455b960:[{ v1 NodeList}] 0x455bae0:[{ v1 PersistentVolumeClaimList}] 0x455bba0:[{ v1 PersistentVolumeList}] 0x455bea0:[{ v1 PodList}] 0x455bf60:[{ v1 PodStatusResult}] 0x455c020:[{ v1 PodTemplate}] 0x455c0e0:[{ v1 PodTemplateList}] 0x455c260:[{ v1 ReplicationControllerList}] 0x455c3e0:[{ v1 ResourceQuotaList}] 0x455c6e0:[{ v1 SecretList}] 0x455c860:[{ v1 ServiceAccountList}] 0x455c9e0:[{ v1 ServiceList}] 0x455fce0:[{ v1 APIResourceList}] 0x455fda0:[{ v1 CreateOptions}] 0x455fe60:[{ v1 ExportOptions}] 0x45602e0:[{ v1 UpdateOptions}] 0x4565820:[{ __internal ConfigMap}] 0x4565d60:[{ __internal Namespace}] 0x4565e20:[{ __internal Node}] 0x4565fa0:[{ __internal PersistentVolume}] 0x4566060:[{ __internal PersistentVolumeClaim}] 0x45661e0:[{ __internal Pod}] 0x4566360:[{ __internal RangeAllocation}] 0x4566420:[{ __internal ReplicationController}] 0x45665a0:[{ __internal ResourceQuota}] 0x4566720:[{ __internal Secret}] 0x45668a0:[{ __internal Service}] 0x461cae0:[{ v1 ConfigMap}] 0x461d100:[{ v1 Namespace}] 0x461d1e0:[{ v1 Node}] 0x461d3a0:[{ v1 PersistentVolume}] 0x461d480:[{ v1 PersistentVolumeClaim}] 0x461d640:[{ v1 Pod}] 0x461d800:[{ v1 RangeAllocation}] 0x461d8e0:[{ v1 ReplicationController}] 0x461daa0:[{ v1 ResourceQuota}] 0x461dd40:[{ v1 Service}] 0x461fb00:[{ v1 PatchOptions}] 0x4635520:[{ __internal ServiceAccount}] 0x4691160:[{ v1 APIVersions}] 0x46c0de0:[{ v1 Secret}] 0x46c0ee0:[{ v1 ServiceAccount}] 0x46c1ce0:[{ v1 APIGroup}] 0x474e820:[{ __internal PodAttachOptions}] 0x4751820:[{ v1 PodAttachOptions}] 0x4752320:[{ v1 DeleteOptions}] 0x47be240:[{ __internal PodExecOptions}] 0x47c00a0:[{ v1 PodExecOptions}] 0x47c02e0:[{ v1 Status}] 0x48208e0:[{ v1 ListOptions}] 0x4877900:[{ __internal PodLogOptions}] 0x48791c0:[{ v1 PodLogOptions}] 0x4917960:[{ __internal Event}] 0x49274a0:[{ v1 Event}]] map[0x44249e0:{ v1 APIGroupList} 0x455fce0:{ v1 APIResourceList} 0x4691160:{ v1 APIVersions} 0x46c1ce0:{ v1 APIGroup} 0x47c02e0:{ v1 Status}] map[APIGroup:0x46c1ce0 APIGroupList:0x44249e0 APIResourceList:0x455fce0 APIVersions:0x4691160 Status:0x47c02e0] map[{ v1 Event}:0x117c610 { v1 Namespace}:0x117cb70 { v1 Node}:0x117c1e0 { v1 Pod}:0x117bcc0 { v1 ReplicationController}:0x117c400 { v1 Secret}:0x117cd20] map[0x48d8d80:0x1199990 0x48d8f40:0x1199a70 0x48d92c0:0x1199b50 0x48d9480:0x1199c70 0x48d9640:0x1199d50 0x48d9800:0x1199ea0 0x48d99c0:0x1199f10 0x48d9b80:0x1199ff0 0x48d9d40:0x119a0d0 0x48d9f00:0x119a1b0 0x48da0c0:0x119a290 0x48da280:0x119a3b0 0x48da600:0x119a490 0x499e820:0x11998e0 0x499eba0:0x1199a00 0x499f2a0:0x1199ae0 0x499f620:0x1199bc0 0x499f9a0:0x1199ce0 0x499fd20:0x1199dc0 0x49a00a0:0x1199e30 0x49a0420:0x1199f80 0x49a0b20:0x119a060 0x49a1220:0x119a140 0x49a15a0:0x119a220 0x49a1920:0x119a300 0x49a1ca0:0x119a420] 0xc000296c60 map[:[v1]] [{ v1}] k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29}

oomichi / try-kubernetes

kubernetes/pull/75616 失敗調査 #81