tmshort
commented
1 year ago Took 8 attempts to fail; 12.5% occurrance.
Open tmshort opened 1 year ago
tmshort
commented
1 year ago Took 8 attempts to fail; 12.5% occurrance.
tmshort
commented
1 year ago This is likely due to an error updating the subscription, possibly at a critical time. But, I do see similar errors during a success.
This is from the catalog-operator pod:
2023-11-07T19:43:34.379154025Z stderr F time="2023-11-07T19:43:34Z" level=debug msg="resolving subscriptions in namespace" id=TjnDL namespace=fail-forward-e2e-t25w6
2023-11-07T19:43:34.387130717Z stderr F time="2023-11-07T19:43:34Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"mc-csv-failed-qn8mm-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
2023-11-07T19:43:34.388610715Z stderr F E1107 19:43:34.388510 1 queueinformer_operator.go:319] sync {"update" "fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "mc-csv-failed-qn8mm-sub": the object has been modified; please apply your changes to the latest version and try again
2023-11-07T19:43:34.388813015Z stderr F time="2023-11-07T19:43:34Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=From the e2e log:
19:44:38.6016: subscription fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub state: AtLatestKnown (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-t25w6", Name:"install-kktrz", UID:"0666549d-ad6f-4d65-abd7-2ab374236509", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"1122", FieldPath:""}
19:49:38.5972: subscription fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"example-operator.v0.2.0", InstalledCSV:"example-operator.v0.2.0", Install:(*v1alpha1.InstallPlanReference)(0xc000fc3f40), State:"AtLatestKnown", Reason:"", InstallPlanGeneration:2, InstallPlanRef:(*v1.ObjectReference)(0xc0005fc690), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0005fc5b0), LastUpdated:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0005fc620), LastUpdated:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"BundleUnpacking", Status:"False", Reason:"", Message:"", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, @existing/fail-forward-e2e-t25w6//example-operator.v0.2.0, @existing/fail-forward-e2e-t25w6//example-operator.v0.1.0 and mc-csv-failed-qn8mm/fail-forward-e2e-t25w6/stable/example-operator.v0.3.0 originate from package packageA, subscription mc-csv-failed-qn8mm-sub exists, subscription mc-csv-failed-qn8mm-sub requires at least one of mc-csv-failed-qn8mm/fail-forward-e2e-t25w6/stable/example-operator.v0.3.0 or @existing/fail-forward-e2e-t25w6//example-operator.v0.2.0", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 7, 19, 44, 38, 0, time.Local)}
stevekuznetsov
commented
12 months ago Fail Forward Upgrades when a CSV resource is in a failed state eventually reports a successful state when using skips
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:402
STEP: creating the testing namespace with an OG that enabled fail forward behavior @ 11/14/23 02:22:51.908
created the fail-forward-e2e-hm8m2 testing namespace
STEP: deploying the testing catalog @ 11/14/23 02:22:51.999
Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
Catalog source mc-csv-failed-b47k7 created
waiting 148.179373ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 590.956103ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - IDLE
waiting 200.019269ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 602.234668ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 197.393456ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 15.656604807s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
STEP: creating the testing subscription @ 11/14/23 02:23:09.484
STEP: waiting until the subscription has an IP reference @ 11/14/23 02:23:09.526
02:23:10.8861: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: UpgradePending (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
waiting 1.359020737s for subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
STEP: waiting for the v0.1.0 CSV to report a succeeded phase @ 11/14/23 02:23:10.886
waiting for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0 to reach condition
waited 199.608397ms for csv fail-forward-e2e-hm8m2/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
waited 199.697425ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
waited 1.601659075s for csv fail-forward-e2e-hm8m2/example-operator.v0.1.0 - Succeeded (InstallSucceeded): install strategy completed with no errors
waited 1.601733023s for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0: to be in phases [Succeeded], in phase Succeeded (InstallSucceeded): install strategy completed with no errors
STEP: updating the catalog with a broken v0.2.0 csv @ 11/14/23 02:23:12.688
STEP: removing the existing catalog source @ 11/14/23 02:23:12.688
STEP: removing the previous catalog source pod(s) @ 11/14/23 02:23:12.892
waiting for the catalog source mc-csv-failed-b47k7-8ckq6 pod to be deleted...
STEP: updating the catalog with a new bundle images @ 11/14/23 02:23:13.976
Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
Catalog source mc-csv-failed-b47k7 created
waiting 138.410284ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 2.26910633s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 2.598714655s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 200.99772ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 12.400568791s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 199.285203ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
STEP: verifying the subscription has installed the current csv @ 11/14/23 02:23:31.885
02:23:32.0872: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: AtLatestKnown (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
02:23:32.2868: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: UpgradePending (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-m4jh9", UID:"4d3d9467-23cd-4055-884c-34ffc0c08b28", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128987", FieldPath:""}
STEP: waiting for the bad CSV to report a failed state @ 11/14/23 02:23:32.286
waiting for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0 to reach condition
waited 207.233924ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - Pending (RequirementsUnknown): requirements not yet checked
waited 1m12.23119933s for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase Pending (RequirementsUnknown): requirements not yet checked
waited 593.392252ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - InstallReady (AllRequirementsMet): all requirements found, attempting install
waited 593.428936ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase InstallReady (AllRequirementsMet): all requirements found, attempting install
waited 398.113032ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
waited 398.154075ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
STEP: patching the catalog with a fixed version @ 11/14/23 02:23:33.486
STEP: removing the existing catalog source @ 11/14/23 02:23:33.486
STEP: removing the previous catalog source pod(s) @ 11/14/23 02:23:33.698
waiting for the catalog source mc-csv-failed-b47k7-297mr pod to be deleted...
STEP: updating the catalog with a new bundle images @ 11/14/23 02:23:34.793
Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
Catalog source mc-csv-failed-b47k7 created
waiting 138.763895ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 4.050054318s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 19.996456256s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 1.000994366s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
STEP: waiting for the subscription to have the example-operator.v0.3.0 status.currentCSV @ 11/14/23 02:24:00.085
02:24:00.2865: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: AtLatestKnown (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-m4jh9", UID:"4d3d9467-23cd-4055-884c-34ffc0c08b28", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128987", FieldPath:""}
02:29:00.4869: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"example-operator.v0.2.0", InstalledCSV:"example-operator.v0.2.0", Install:(*v1alpha1.InstallPlanReference)(0xc001b13e40), State:"AtLatestKnown", Reason:"", InstallPlanGeneration:2, InstallPlanRef:(*v1.ObjectReference)(0xc00087ec40), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087e930), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087e9a0), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087ea80), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087eaf0), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087eb60), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"BundleUnpacking", Status:"False", Reason:"", Message:"", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: @existing/fail-forward-e2e-hm8m2//example-operator.v0.2.0, @existing/fail-forward-e2e-hm8m2//example-operator.v0.1.0 and mc-csv-failed-b47k7/fail-forward-e2e-hm8m2/stable/example-operator.v0.3.0 originate from package packageA, clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, subscription mc-csv-failed-b47k7-sub exists, subscription mc-csv-failed-b47k7-sub requires at least one of mc-csv-failed-b47k7/fail-forward-e2e-hm8m2/stable/example-operator.v0.3.0 or @existing/fail-forward-e2e-hm8m2//example-operator.v0.2.0", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 14, 2, 28, 34, 0, time.Local)}
02:29:00.4871: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub spec: &v1alpha1.SubscriptionSpec{CatalogSource:"mc-csv-failed-b47k7", CatalogSourceNamespace:"fail-forward-e2e-hm8m2", Package:"packageA", Channel:"stable", StartingCSV:"", InstallPlanApproval:"", Config:(*v1alpha1.SubscriptionConfig)(nil)}
[FAILED] in [It] - /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:412 @ 11/14/23 02:29:00.488
STEP: removing the testing catalog resources @ 11/14/23 02:29:00.489
Deleting config map mc-csv-failed-mkq2q-configmap...
Deleting catalog source mc-csv-failed-mkq2q...
Deleting config map mc-csv-failed-mkq2q-configmap...
Deleting catalog source mc-csv-failed-mkq2q...
Deleting config map mc-csv-failed-mkq2q-configmap...
Deleting catalog source mc-csv-failed-mkq2q...
Deleting config map mc-csv-failed-b47k7-configmap...
Deleting catalog source mc-csv-failed-b47k7...
waiting for the catalog source mc-csv-failed-b47k7-29zm8 pod to be deleted...
Deleting config map mc-csv-failed-b47k7-configmap...
Deleting catalog source mc-csv-failed-b47k7...
Deleting config map mc-csv-failed-b47k7-configmap...
Deleting catalog source mc-csv-failed-b47k7...
STEP: deleting the testing namespace @ 11/14/23 02:29:02.588
collecting the fail-forward-e2e-hm8m2 namespace artifacts as the 'eventually reports a successful state when using skips' test case failed
collecting logs in the /logs/artifacts artifacts directory
unable to determine kubeconfig path so defaulting to the $KUBECONFIG value
Using the oc kubectl binary
Using the /logs/artifacts/fail-forward-e2e-hm8m2 output directory
Storing the test artifact output in the /logs/artifacts/fail-forward-e2e-hm8m2 directory
Collecting get catalogsources -o yaml output...
Collecting get subscriptions -o yaml output...
Collecting get operatorgroups -o yaml output...
Collecting get clusterserviceversions -o yaml output...
Collecting get installplans -o yaml output...
Collecting get pods -o wide output...
Collecting get events --sort-by .lastTimestamp output...
tearing down the fail-forward-e2e-hm8m2 namespace
resetting e2e kube client
deleting fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub
deleting fail-forward-e2e-hm8m2/fail-forward-e2e-hm8m2-operatorgroup
deleting <global>/fail-forward-e2e-hm8m2
garbage collecting CRDs
• [FAILED] [375.072 seconds]
Fail Forward Upgrades when a CSV resource is in a failed state [It] eventually reports a successful state when using skips
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:402
[FAILED] Expected
<wait.errInterrupted>:
timed out waiting for the condition
{
cause: <*errors.errorString | 0xc000250190>{
s: "timed out waiting for the condition",
},
}
to be nil
In [It] at: /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:412 @ 11/14/23 02:29:00.488 likely similar culprit in different test
Flaky Test Report
Failure Log Link Failure Log
Relevant Failure Log