numaproj / numaplane

Control Plane for Numaproj
Apache License 2.0
8 stars 4 forks source link

Fix the failure of ISBServiceRollout and PipelineRollout after applying the sample CRs #35

Closed chandankumar4 closed 3 months ago

chandankumar4 commented 3 months ago

Describe the bug After applying the sample yaml of ISBServiceRollout and PipelineRollout, numaplane-controller-manager is throwing below error

  1. ISBServiceRollout Error
    {"level":"error","error":"unexpected end of JSON input","logger":"numaplane","caller":"controller/isbservicerollout_controller.go:185","ts":"2024-06-05T13:42:32.225935678Z","msg":"failed to parse Status from InterstepBuffer CR: &{TypeMeta:{Kind:InterStepBufferService APIVersion:numaflow.numaproj.io/v1alpha1} ObjectMeta:{Name:my-isbsvc GenerateName: Namespace:example-namespace SelfLink: UID:e51aee47-6134-41bd-8f9a-5d3e5169e0ce ResourceVersion:2155 Generation:1 CreationTimestamp:2024-06-05 13:39:03 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:numaplane.numaproj.io/v1alpha1 Kind:ISBServiceRollout Name:my-isbsvc UID:ee097242-31a4-49d4-a6a5-4d15b56fb75b Controller:0x40028aba50 BlockOwnerDeletion:0x40028aba4f}] Finalizers:[isbsvc-controller] ManagedFields:[{Manager:manager Operation:Update APIVersion:numaflow.numaproj.io/v1alpha1 Time:2024-06-05 13:39:03 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:metadata\":{\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"ee097242-31a4-49d4-a6a5-4d15b56fb75b\\\"}\":{}}},\"f:spec\":{\".\":{},\"f:jetstream\":{\".\":{},\"f:persistence\":{\".\":{},\"f:volumeSize\":{}},\"f:replicas\":{},\"f:version\":{}}}} Subresource:} {Manager:numaflow Operation:Update APIVersion:numaflow.numaproj.io/v1alpha1 Time:2024-06-05 13:42:32 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"isbsvc-controller\\\"\":{}}}} Subresource:}]} Spec:{Raw:[123 34 106 101 116 115 116 114 101 97 109 34 58 123 34 112 101 114 115 105 115 116 101 110 99 101 34 58 123 34 118 111 108 117 109 101 83 105 122 101 34 58 34 49 71 105 34 125 44 34 114 101 112 108 105 99 97 115 34 58 51 44 34 118 101 114 115 105 111 110 34 58 34 108 97 116 101 115 116 34 125 125] Object:<nil>} Status:{Raw:[] Object:<nil>}}, unexpected end of JSON input"}
  2. PipelineRollout Error
    {"level":"error","pipelinerollout":{"Namespace":"example-namespace","Name":"my-pipeline"},"error":"unexpected end of JSON input","logger":"numaplane.controller-manager.reconciler","caller":"controller/pipelinerollout_controller.go:189","ts":"2024-06-05T13:42:31.588088386Z","msg":"failed to parse Pipeline Status from pipeline CR: &{TypeMeta:{Kind:Pipeline APIVersion:numaflow.numaproj.io/v1alpha1} ObjectMeta:{Name:my-pipeline GenerateName: Namespace:example-namespace SelfLink: UID:6fc61d8a-d3ec-4fce-8820-5dfeed25180b ResourceVersion:2133 Generation:3 CreationTimestamp:2024-06-05 13:41:28 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:numaplane.numaproj.io/v1alpha1 Kind:PipelineRollout Name:my-pipeline UID:39e4bedd-a838-435c-87b5-2c5cc3dbf402 Controller:0x400278d7c7 BlockOwnerDeletion:0x400278d7c6}] Finalizers:[pipeline-controller] ManagedFields:[{Manager:manager Operation:Update APIVersion:numaflow.numaproj.io/v1alpha1 Time:2024-06-05 13:42:31 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:metadata\":{\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"39e4bedd-a838-435c-87b5-2c5cc3dbf402\\\"}\":{}}},\"f:spec\":{\".\":{},\"f:edges\":{},\"f:interStepBufferServiceName\":{},\"f:lifecycle\":{\".\":{},\"f:deleteGracePeriodSeconds\":{},\"f:desiredPhase\":{},\"f:pauseGracePeriodSeconds\":{}},\"f:limits\":{\".\":{},\"f:bufferMaxLength\":{},\"f:bufferUsageLimit\":{},\"f:readBatchSize\":{},\"f:readTimeout\":{}},\"f:vertices\":{},\"f:watermark\":{\".\":{},\"f:disabled\":{},\"f:maxDelay\":{}}}} Subresource:} {Manager:numaflow Operation:Update APIVersion:numaflow.numaproj.io/v1alpha1 Time:2024-06-05 13:42:31 +0000 UTC FieldsType:FieldsV1 FieldsV1:{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"pipeline-controller\\\"\":{}}}} Subresource:}]} Spec:{Raw:[123 34 101 100 103 101 115 34 58 91 123 34 102 114 111 109 34 58 34 105 110 34 44 34 116 111 34 58 34 99 97 116 34 125 44 123 34 102 114 111 109 34 58 34 99 97 116 34 44 34 116 111 34 58 34 111 117 116 34 125 93 44 34 105 110 116 101 114 83 116 101 112 66 117 102 102 101 114 83 101 114 118 105 99 101 78 97 109 101 34 58 34 109 121 45 105 115 98 115 118 99 34 44 34 108 105 102 101 99 121 99 108 101 34 58 123 34 100 101 108 101 116 101 71 114 97 99 101 80 101 114 105 111 100 83 101 99 111 110 100 115 34 58 51 48 44 34 100 101 115 105 114 101 100 80 104 97 115 101 34 58 34 82 117 110 110 105 110 103 34 44 34 112 97 117 115 101 71 114 97 99 101 80 101 114 105 111 100 83 101 99 111 110 100 115 34 58 51 48 125 44 34 108 105 109 105 116 115 34 58 123 34 98 117 102 102 101 114 77 97 120 76 101 110 103 116 104 34 58 51 48 48 48 48 44 34 98 117 102 102 101 114 85 115 97 103 101 76 105 109 105 116 34 58 56 48 44 34 114 101 97 100 66 97 116 99 104 83 105 122 101 34 58 53 48 48 44 34 114 101 97 100 84 105 109 101 111 117 116 34 58 34 49 115 34 125 44 34 118 101 114 116 105 99 101 115 34 58 91 123 34 110 97 109 101 34 58 34 105 110 34 44 34 115 111 117 114 99 101 34 58 123 34 103 101 110 101 114 97 116 111 114 34 58 123 34 100 117 114 97 116 105 111 110 34 58 34 49 115 34 44 34 106 105 116 116 101 114 34 58 34 48 115 34 44 34 109 115 103 83 105 122 101 34 58 56 44 34 114 112 117 34 58 53 125 125 125 44 123 34 110 97 109 101 34 58 34 99 97 116 34 44 34 117 100 102 34 58 123 34 98 117 105 108 116 105 110 34 58 123 34 110 97 109 101 34 58 34 99 97 116 34 125 125 125 44 123 34 110 97 109 101 34 58 34 111 117 116 34 44 34 115 105 110 107 34 58 123 34 108 111 103 34 58 123 125 125 125 93 44 34 119 97 116 101 114 109 97 114 107 34 58 123 34 100 105 115 97 98 108 101 100 34 58 102 97 108 115 101 44 34 109 97 120 68 101 108 97 121 34 58 34 48 115 34 125 125] Object:<nil>} Status:{Raw:[] Object:<nil>}}, unexpected end of JSON input"}

To Reproduce Steps to reproduce the behavior:

  1. Deploy numaplane-controller-manager, can be done via make start
  2. Apply sample yaml of ISBServiceRollout and PipelineRollout from config/samples dir
  3. Check the logs of numaplane-controller-manager

Expected behavior Numaplane controller should not throw this error

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

Additional context Add any other context about the problem here.


Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

juliev0 commented 3 months ago

This happens because we're trying to parse the Pipeline's Status but there's no Numaflow Controller running to write any Status. We should change it to not error out just because of that.

juliev0 commented 3 months ago

Oh, I think @afugazzotto may have already fixed it in his Unit test PR on this line: https://github.com/numaproj/numaplane/blob/rolloutTests/internal/kubernetes/dynamic_client.go#L51

xdevxy commented 3 months ago

@afugazzotto this is fixed by this PR?

afugazzotto commented 3 months ago

yes, it should be fixed with this change: https://github.com/numaproj/numaplane/pull/41/files#diff-afcfe9771d970942a2fb57fdd48aa356fa484f93923e0b823497348ba16dcb80R50-R53