Closed anshrma closed 1 year ago
hi @elamaran11 - Adding the TestJob.
@shapirov103 The installation works fine all 4 variants of EKSA vSphere, BareMetal, Snow and Nutanix. Test Job works fine in vSphere and BareMetal but fails in Snow and Nutanix. But i can see all data of all 4 clusters in NewRelic console. We should be good to pass this PR.
@elamaran11 if the test jobs fails in a couple of environments, then approval will mean that New Relic should be marked as FAIL for environments where it fails, and SUCCESS where it works. We have no capacity to manually validate UI across environments, the whole point is to have the full flow automated. Let's also agree on continuity of this contribution: who, when and how it will be maintained. Since this is not a point in time validation (must be continuous) we expect partners to support their products, including version upgrades, configuration upgrades, responses to issues (if any).
Agreed @shapirov103 @anshrma Please check on the feedback from Mikhail. We need this to work on all envs.
@shapirov103 Now it works in Snow and Nutanix after introducting 60s sleep in the test-container
. The problem is slowness in admission web hook from control plane side. Im good with this now. Please check from your end.
@elamaran11 -
1) Clustername - This need not fixed or addressed now as the tests look for a pod name directly and the intention is to have this in place via automation not by UI. 2) Tightened the SA used by the test job with the latest commit.
- Clustername @anshrma Please remove this field if we done need it for conformance test job.
Hi @anshrma The installation now works fine for vSphere and BareMetal installation but fails for Local Clusters and Snow.
Remember in Local Clusters CP in NotReady
by design.
Please see the details from Local Clusters :
ubuntu@ip-10-0-0-178:~/eks-anywhere-conformance-testing$ kubectl get all -n newrelic
NAME READY STATUS RESTARTS AGE
pod/newrelic-newrelic-kube-state-metrics-78f4849688-q5j96 1/1 Running 0 15m
pod/newrelic-newrelic-nri-kube-events-6bf76bddf7-7zkcz 2/2 Running 0 15m
pod/newrelic-newrelic-nri-metadata-injection-6b7b47669d-88m8g 0/1 Pending 0 15m
pod/newrelic-newrelic-nrk8s-controlplane-4k9pm 1/2 CrashLoopBackOff 12 (38s ago) 15m
pod/newrelic-newrelic-nrk8s-controlplane-mqwfq 1/2 CrashLoopBackOff 12 (62s ago) 15m
pod/newrelic-newrelic-nrk8s-controlplane-qcf78 1/2 CrashLoopBackOff 12 (56s ago) 15m
pod/newrelic-newrelic-nrk8s-ksm-85d6bfc46b-wqdng 0/2 ContainerCreating 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-2svq7 0/2 ContainerCreating 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-5k5r5 0/2 ContainerCreating 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-62kjv 0/2 Pending 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-cs7nq 0/2 Pending 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-lg2wd 0/2 ContainerCreating 0 15m
pod/newrelic-newrelic-nrk8s-kubelet-nhjkk 0/2 Pending 0 15m
pod/newrelic-testjob-001-9rfcp 1/1 Running 0 2m51s
pod/newrelic-testjob-001-cs5m8 0/1 Error 0 9m34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/newrelic-newrelic-kube-state-metrics ClusterIP 172.20.134.13 <none> 8080/TCP 15m
service/newrelic-newrelic-nri-metadata-injection ClusterIP 172.20.153.12 <none> 443/TCP 15m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/newrelic-newrelic-nrk8s-controlplane 3 3 0 3 0 <none> 15m
daemonset.apps/newrelic-newrelic-nrk8s-kubelet 6 6 0 6 0 <none> 15m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/newrelic-newrelic-kube-state-metrics 1/1 1 1 15m
deployment.apps/newrelic-newrelic-nri-kube-events 1/1 1 1 15m
deployment.apps/newrelic-newrelic-nri-metadata-injection 0/1 1 0 15m
deployment.apps/newrelic-newrelic-nrk8s-ksm 0/1 1 0 15m
NAME DESIRED CURRENT READY AGE
replicaset.apps/newrelic-newrelic-kube-state-metrics-78f4849688 1 1 1 15m
replicaset.apps/newrelic-newrelic-nri-kube-events-6bf76bddf7 1 1 1 15m
replicaset.apps/newrelic-newrelic-nri-metadata-injection-6b7b47669d 1 1 0 15m
replicaset.apps/newrelic-newrelic-nrk8s-ksm-85d6bfc46b 1 1 0 15m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/newrelic-testjob 10 10 * * * False 0 <none> 11m
NAME COMPLETIONS DURATION AGE
job.batch/newrelic-testjob-001 0/1 9m34s 9m34s
ubuntu@ip-10-0-0-178:~/eks-anywhere-conformance-testing$ k logs newrelic-newrelic-nrk8s-controlplane-4k9pm -n newrelic
Defaulted container "controlplane" out of: controlplane, forwarder
time="2023-04-18T21:11:08Z" level=info msg="Waiting for agent container to be ready..."
time="2023-04-18T21:12:39Z" level=error msg="creating integration wrapper: applying option: timeout waiting for agent: probe timed out after 1m30s"
ubuntu@ip-10-0-0-178:~/eks-anywhere-conformance-testing$ k logs newrelic-testjob-001-cs5m8 -n newrelic
Defaulted container "test-container" out of: test-container, kubectl (init)
Cloning into 'newrelic-integration-e2e-action'...
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading github.com/newrelic/newrelic-client-go v0.91.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8
go: downloading github.com/imdario/mergo v0.3.12
go: downloading github.com/google/go-querystring v1.1.0
go: downloading github.com/hashicorp/go-retryablehttp v0.7.0
go: downloading github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go: downloading github.com/valyala/fastjson v1.6.3
go: downloading github.com/hashicorp/go-cleanhttp v0.5.1
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
time="2023-04-18T21:08:59Z" level=info msg="running e2e"
time="2023-04-18T21:08:59Z" level=debug msg="parsing the content of the spec file"
time="2023-04-18T21:08:59Z" level=debug msg="return with settings"
time="2023-04-18T21:08:59Z" level=debug msg="validating the spec definition"
time="2023-04-18T21:08:59Z" level=debug msg="[scenario]: This scenario will verify that metrics from a k8s Cluster are correctly collected without privileges\n, [Tag]: newrelic-testjob-001-cs5m8"
time="2023-04-18T21:09:00Z" level=warning msg="Error detected" iteration=0
time="2023-04-18T21:09:00Z" level=error msg="querying: query did not return a valid result: SELECT latest(k8s.pod.startTime) FROM Metric SINCE 5 MINUTES AGO WHERE k8s.podName = 'newrelic-testjob-001-cs5m8'"
time="2023-04-18T21:09:05Z" level=fatal msg="after 1 attempts, last errors: [querying: query did not return a valid result: SELECT latest(k8s.pod.startTime) FROM Metric SINCE 5 MINUTES AGO WHERE k8s.podName = 'newrelic-testjob-001-cs5m8']"
exit status 1
Please see below on Snow. Installation is not working and also job fails :
ubuntu@ip-34-223-14-199:~/eks-anywhere-conformance-testing$ kubectl logs newrelic-testjob-001-4pqzl -n newrelic
Error from server (BadRequest): container "test-container" in pod "newrelic-testjob-001-4pqzl" is waiting to start: PodInitializing
ubuntu@ip-34-223-14-199:~/eks-anywhere-conformance-testing$ kubectl get all -n newrelic
NAME READY STATUS RESTARTS AGE
pod/newrelic-newrelic-nri-metadata-injection-admission-create-srckw 0/1 Completed 0 59m
pod/newrelic-testjob-001-4pqzl 0/1 Terminating 0 30m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/newrelic-testjob 10 10 * * * False 0 <none> 33m
NAME COMPLETIONS DURATION AGE
job.batch/newrelic-newrelic-nri-metadata-injection-admission-create 1/1 34m 60m
job.batch/newrelic-testjob-001 0/1 30m 30m
Good News though is the solution is validated on vSphere and BareMetal.
Hi @anshrma After working on our environment a little now, we can confirm that EKS-A Conformance framework works fine on EKS-A on SnowBall, please see your test job results below :
I would recommend a next step to do the following :
[ec2-user@ip-34-223-14-193 eksa-conformance-snow]$ kubectl logs newrelic-testjob-01-f7k5s -n newrelic -f
Cloning into 'newrelic-integration-e2e-action'...
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading github.com/newrelic/newrelic-client-go v0.91.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8
go: downloading github.com/imdario/mergo v0.3.12
go: downloading github.com/hashicorp/go-retryablehttp v0.7.0
go: downloading github.com/google/go-querystring v1.1.0
go: downloading github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go: downloading github.com/valyala/fastjson v1.6.3
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/hashicorp/go-cleanhttp v0.5.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
time="2023-05-19T19:44:14Z" level=info msg="running e2e"
time="2023-05-19T19:44:14Z" level=debug msg="parsing the content of the spec file"
time="2023-05-19T19:44:14Z" level=debug msg="return with settings"
time="2023-05-19T19:44:14Z" level=debug msg="validating the spec definition"
time="2023-05-19T19:44:14Z" level=debug msg="[scenario]: This scenario will verify that metrics from a k8s Cluster are correctly collected without privileges\n, [Tag]: newrelic-testjob-01-f7k5s"
time="2023-05-19T19:44:14Z" level=info msg="execution completed successfully!"
@elamaran11 - For outpost (mentioned as local cluster in this thread,
Running
when you ran kubectl get all -n newrelic
(for the ones, which are not newrelic-testjob
) . Error
Remember in Local Clusters CP in NotReady by design
?@anshrma Conformance framework has successfully validated NewRelic for three deployment models namely - EKS-A on BM, EKS-A on Snow and EKS-A on vSphere. One pre-req that is required to merge the PR is NEWR Commenting on the PR to say they can support ongoing maintenance with previously shared verbiage.
GH Action failure can be ignored as it was from an experiment. We have disabled the GH actions now for future PRs until we have the experiment fully working. Approving and Merging this.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.