search

kubeshop / testkube

☸️ Kubernetes-native testing framework for test execution and orchestration
https://testkube.io
Other
1.35k stars 133 forks source link

Tests passing on backend show as failed in Dashboard, no Dashboard logs #4194

Closed AtomicMapache closed 1 year ago

AtomicMapache commented 1 year ago

Version / Cluster AWS EKS Kubernetes (1.27) Fargate cluster. Testkube v. 1.8.39-1.13.12

Describe the bug Passing tests (viewed via logs on the backend) show up as failed in the testkube UI dashboard, no logs are displayed.

To Reproduce Install testkube with the following command and chart values, minus ingress and persistent volume configs helm install -f testkube-values-new.yaml testkube kubeshop/testkube --version 1.13.12 --timeout 10m0s --namespace shared

testkube-values-new.zip

Backend Test Output

[ec2-user@ip-10-0-192-135 k8s-manifests]$ kubectl logs 64b6e3f0530875ebaf73794f-lhf6t -n shared Defaulted container "64b6e3f0530875ebaf73794f" out of: 64b6e3f0530875ebaf73794f, 64b6e3f0530875ebaf73794f-init (init) {"type":"line","content":"🌍 Reading environment variables...","time":"2023-07-18T19:12:44.065916596Z"} {"type":"line","content":"✅ Environment variables read successfully","time":"2023-07-18T19:12:44.066174662Z"} {"type":"line","content":"RUNNER_ENDPOINT=\"testkube-minio-service-shared:9000\"","time":"2023-07-18T19:12:44.066185893Z"} {"type":"line","content":"RUNNER_ACCESSKEYID=\"********\"","time":"2023-07-18T19:12:44.066190525Z"} {"type":"line","content":"RUNNER_SECRETACCESSKEY=\"********\"","time":"2023-07-18T19:12:44.066198451Z"} {"type":"line","content":"RUNNER_REGION=\"\"","time":"2023-07-18T19:12:44.066202295Z"} {"type":"line","content":"RUNNER_TOKEN=\"\"","time":"2023-07-18T19:12:44.066206054Z"} {"type":"line","content":"RUNNER_BUCKET=\"testkube-artifacts\"","time":"2023-07-18T19:12:44.066209522Z"} {"type":"line","content":"RUNNER_SSL=false","time":"2023-07-18T19:12:44.066215367Z"} {"type":"line","content":"RUNNER_SCRAPPERENABLED=\"true\"","time":"2023-07-18T19:12:44.066218487Z"} {"type":"line","content":"RUNNER_GITUSERNAME=\"********\"","time":"2023-07-18T19:12:44.066222184Z"} {"type":"line","content":"RUNNER_GITTOKEN=\"********\"","time":"2023-07-18T19:12:44.066226231Z"} {"type":"line","content":"RUNNER_DATADIR=\"/data\"","time":"2023-07-18T19:12:44.066229687Z"} {"type":"line","content":"RUNNER_CLUSTERID=\"clusterf83c3172f255fad68c2ab1a59be52916\"","time":"2023-07-18T19:12:44.066233268Z"} {"type":"line","content":"RUNNER_CDEVENTS_TARGET=\"\"","time":"2023-07-18T19:12:44.066237063Z"} {"type":"line","content":"RUNNER_DASHBOARD_URI=\"\"","time":"2023-07-18T19:12:44.066240323Z"} {"type":"line","content":"RUNNER_CLOUD_MODE=\"false\"","time":"2023-07-18T19:12:44.066244115Z"} {"type":"line","content":"RUNNER_CLOUD_API_TLS_INSECURE=\"false\"","time":"2023-07-18T19:12:44.066247795Z"} {"type":"line","content":"RUNNER_CLOUD_API_URL=\"agent.testkube.io:443\"","time":"2023-07-18T19:12:44.066251778Z"} {"type":"line","content":"RUNNER_CLOUD_API_KEY=\"\"","time":"2023-07-18T19:12:44.066255395Z"} {"type":"line","content":"RUNNER_CLOUD_CONNECTION_TIMEOUT=10","time":"2023-07-18T19:12:44.066259384Z"} {"type":"line","content":"🚚 Preparing test runner","time":"2023-07-18T19:12:44.066263179Z"} {"type":"line","content":"✅ Uploading artifacts using MinIO Uploader","time":"2023-07-18T19:12:44.066269065Z"} {"type":"event","content":"running test [64b6e3f0530875ebaf73794f]","time":"2023-07-18T19:12:44.06658648Z"} {"type":"line","content":"🚚 Preparing for test run","time":"2023-07-18T19:12:44.066599252Z"} {"type":"line","content":"🚀 Test run command newman run /data/repo/Core App Tests - WebPlayer.postman_collection.json -e /tmp/testkube-tmp1897306484 --reporters cli,json --reporter-json-export /tmp/testkube-tmp4151526495.json","time":"2023-07-18T19:12:44.066856036Z"} {"type":"line","content":"🔬 Executing in directory : \n $ newman run /data/repo/Core App Tests - WebPlayer.postman_collection.json -e /tmp/testkube-tmp1897306484 --reporters cli,json --reporter-json-export /tmp/testkube-tmp4151526495.json","time":"2023-07-18T19:12:44.066872166Z"} {"type":"line","content":"newman\n\n","time":"2023-07-18T19:12:45.654031992Z"} {"type":"line","content":"Core App Tests - WebPlayer\n","time":"2023-07-18T19:12:45.654989975Z"} {"type":"line","content":"\n→ na.com client=testdb sign=testct1 company=41574150-b952-413b-898b-dc5336b4bd12\n","time":"2023-07-18T19:12:45.658607412Z"} {"type":"line","content":" GET https://na.com/v6-wplt/?client=testdb\u0026sign=testct1\u0026company=41574150-b952-413b-898b-dc5336b4bd12 ","time":"2023-07-18T19:12:45.685011667Z"} {"type":"line","content":"[200 OK, 33.9kB, 326ms]\n","time":"2023-07-18T19:12:46.014247794Z"} {"type":"line","content":" ✓ Status code is 200\n","time":"2023-07-18T19:12:46.076176552Z"} {"type":"line","content":"\n┌─────────────────────────┬────────────────────┬───────────────────┐\n│ │ executed │ failed │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ iterations │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ requests │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ test-scripts │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ prerequest-scripts │ 0 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ assertions │ 1 │ 0 │\n├─────────────────────────┴────────────────────┴───────────────────┤\n│ total run duration: 429ms │\n├──────────────────────────────────────────────────────────────────┤\n│ total data received: 33.45kB (approx) │\n├──────────────────────────────────────────────────────────────────┤\n│ average response time: 326ms [min: 326ms, max: 326ms, s.d.: 0µs] │\n└──────────────────────────────────────────────────────────────────┘\n","time":"2023-07-18T19:12:46.104174132Z"} {"type":"line","content":"✅ Execution succeeded","time":"2023-07-18T19:12:46.114565626Z"} {"type":"line","content":"✅ Got Newman result successfully","time":"2023-07-18T19:12:46.126116248Z"} {"type":"line","content":"✅ Mapped Newman result successfully","time":"2023-07-18T19:12:46.126152021Z"} {"type":"result","result":{"status":"passed","output":"newman\n\nCore App Tests - WebPlayer\n\n→ core-eks-test.poppcore.co client=testdb sign=testct1 company=41574150-b952-413b-898b-dc5336b4bd12\n GET https://na.com/v6-wplt/?client=testdb\u0026sign=testct1\u0026company=41574150-b952-413b-898b-dc5336b4bd12 [200 OK, 33.9kB, 326ms]\n ✓ Status code is 200\n\n┌─────────────────────────┬────────────────────┬───────────────────┐\n│ │ executed │ failed │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ iterations │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ requests │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ test-scripts │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ prerequest-scripts │ 0 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ assertions │ 1 │ 0 │\n├─────────────────────────┴────────────────────┴───────────────────┤\n│ total run duration: 429ms │\n├──────────────────────────────────────────────────────────────────┤\n│ total data received: 33.45kB (approx) │\n├──────────────────────────────────────────────────────────────────┤\n│ average response time: 326ms [min: 326ms, max: 326ms, s.d.: 0µs] │\n└──────────────────────────────────────────────────────────────────┘\n","outputType":"text/plain","steps":[{"name":"na.com client=testdb sign=testct1 company=41574150-b952-413b-898b-dc5336b4bd12","duration":"326ms","status":"passed","assertionResults":[{"name":"Status code is 200","status":"passed"}]}]},"time":"2023-07-18T19:12:46.12615853Z"}

Expected behavior Expect tests that passed on the backend to show as passed in the dashboard UI, with logs.

Screenshots Pods listing: image

Dashboard Output: image

Additional context The Postman test collection passes in Postman. Using mongodb as default logs / artifacts db. Minio is disabled. Dashboard and API ingresses appear to be working correctly. I've tested this with Postman, Curl, and Artillery executors. Artillery returns logs to the dashboard UI, but doesn't show as passed in the dash. Postman and curl do not show passed or logs.

vsukhin commented 1 year ago

thank yuo @AtomicMapache sounds like test result was not properly detected

AtomicMapache commented 1 year ago

@vsukhin That is correct. Does this appear to be a configuration issue, perhaps something to do with the use of mongodb over minio? I would imagine that, as long as the api and default db is working properly, this should work out of the box?

vsukhin commented 1 year ago

it should, but we're parsing this result from log output, so it might be wrong parsing as well

vsukhin commented 1 year ago

@AtomicMapache so, I tested out the provided logs, and it was parsed successfully, detected that test passed. But what I could see in your screenshot, that testkube api server was restarted, if it restarted during test execution, it will not be able to catch test execution result prooperly

AtomicMapache commented 1 year ago

@vsukhin I believe that restart only happens on install. I've never seen the api server restart on execution of those tests, but always restarts twice on initial installation. I think it has to do with the mongodb server taking a while to startup, but once it's running, the api server appears to stabilize.

vsukhin commented 1 year ago

can you give provide an exmaple of the test CRD you run above to reproduce it on our side?

AtomicMapache commented 1 year ago

@vsukhin As a simple smoke test, one of the postman requests I'm running simply hits a "health check" endpoint (GET request to www.google.com for example) and then runs the associated test script:

PostmanCollection { "info": { "_postman_id": "04d0a718-d7cf-4d76-8ec8-47f01d7aa58a", "name": "SmokeTests", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json" }, "item": [ { "name": "google-smoke-test", "event": [ { "listen": "test", "script": { "exec": [ "pm.test(\"Status code is 200\", function () {\r", " pm.response.to.have.status(200);\r", "});" ], "type": "text/javascript" } } ], "request": { "method": "GET", "header": [], "url": { "raw": "https://core-eks-test.poppcore.co/v6-wplt/", "protocol": "https", "host": [ "core-eks-test", "poppcore", "co" ], "path": [ "v6-wplt", "" ], "query": [ { "key": "client", "value": "testdb", "disabled": true }, { "key": "sign", "value": "testct1", "disabled": true }, { "key": "company", "value": "41574150-b952-413b-898b-dc5336b4bd12", "disabled": true } ] } }, "response": [] } ] }

Nothing extraordinary

Test output from this morning: [ec2-user@ip-10-0-192-135 k8s-manifests]$ kubectl logs 64b8129f530875ebaf737959-hrnd2 -n shared Defaulted container "64b8129f530875ebaf737959" out of: 64b8129f530875ebaf737959, 64b8129f530875ebaf737959-init (init) {"type":"line","content":"🌍 Reading environment variables...","time":"2023-07-19T16:44:11.360852857Z"} {"type":"line","content":"✅ Environment variables read successfully","time":"2023-07-19T16:44:11.361331067Z"} {"type":"line","content":"RUNNER_ENDPOINT=\"testkube-minio-service-shared:9000\"","time":"2023-07-19T16:44:11.361346331Z"} {"type":"line","content":"RUNNER_ACCESSKEYID=\"********\"","time":"2023-07-19T16:44:11.361352374Z"} {"type":"line","content":"RUNNER_SECRETACCESSKEY=\"********\"","time":"2023-07-19T16:44:11.361357092Z"} {"type":"line","content":"RUNNER_REGION=\"\"","time":"2023-07-19T16:44:11.361360648Z"} {"type":"line","content":"RUNNER_TOKEN=\"\"","time":"2023-07-19T16:44:11.361365082Z"} {"type":"line","content":"RUNNER_BUCKET=\"testkube-artifacts\"","time":"2023-07-19T16:44:11.361369338Z"} {"type":"line","content":"RUNNER_SSL=false","time":"2023-07-19T16:44:11.361374508Z"} {"type":"line","content":"RUNNER_SCRAPPERENABLED=\"true\"","time":"2023-07-19T16:44:11.361379132Z"} {"type":"line","content":"RUNNER_GITUSERNAME=\"core-eks-testkube-git-at-041962984387\"","time":"2023-07-19T16:44:11.361383498Z"} {"type":"line","content":"RUNNER_GITTOKEN=\"********\"","time":"2023-07-19T16:44:11.361387545Z"} {"type":"line","content":"RUNNER_DATADIR=\"/data\"","time":"2023-07-19T16:44:11.361391923Z"} {"type":"line","content":"RUNNER_CLUSTERID=\"clusterf83c3172f255fad68c2ab1a59be52916\"","time":"2023-07-19T16:44:11.361396477Z"} {"type":"line","content":"RUNNER_CDEVENTS_TARGET=\"\"","time":"2023-07-19T16:44:11.361403164Z"} {"type":"line","content":"RUNNER_DASHBOARD_URI=\"\"","time":"2023-07-19T16:44:11.361407723Z"} {"type":"line","content":"RUNNER_CLOUD_MODE=\"false\"","time":"2023-07-19T16:44:11.361412994Z"} {"type":"line","content":"RUNNER_CLOUD_API_TLS_INSECURE=\"false\"","time":"2023-07-19T16:44:11.361417359Z"} {"type":"line","content":"RUNNER_CLOUD_API_URL=\"agent.testkube.io:443\"","time":"2023-07-19T16:44:11.361435434Z"} {"type":"line","content":"RUNNER_CLOUD_API_KEY=\"\"","time":"2023-07-19T16:44:11.3614398Z"} {"type":"line","content":"RUNNER_CLOUD_CONNECTION_TIMEOUT=10","time":"2023-07-19T16:44:11.361443755Z"} {"type":"line","content":"🚚 Preparing test runner","time":"2023-07-19T16:44:11.361451804Z"} {"type":"line","content":"✅ Uploading artifacts using MinIO Uploader","time":"2023-07-19T16:44:11.361457967Z"} {"type":"event","content":"running test [64b8129f530875ebaf737959]","time":"2023-07-19T16:44:11.361811953Z"} {"type":"line","content":"🚚 Preparing for test run","time":"2023-07-19T16:44:11.361824258Z"} {"type":"line","content":"🚀 Test run command newman run /data/repo/SmokeTests.postman_collection.json -e /tmp/testkube-tmp3646309856 --reporters cli,json --reporter-json-export /tmp/testkube-tmp3734072744.json","time":"2023-07-19T16:44:11.362051049Z"} {"type":"line","content":"🔬 Executing in directory : \n $ newman run /data/repo/SmokeTests.postman_collection.json -e /tmp/testkube-tmp3646309856 --reporters cli,json --reporter-json-export /tmp/testkube-tmp3734072744.json","time":"2023-07-19T16:44:11.362067135Z"} {"type":"line","content":"newman\n\n","time":"2023-07-19T16:44:12.721707112Z"} {"type":"line","content":"SmokeTests\n","time":"2023-07-19T16:44:12.722290308Z"} {"type":"line","content":"\n→ google-smoke-test\n","time":"2023-07-19T16:44:12.726032766Z"} {"type":"line","content":" GET https://core-eks-test.poppcore.co/v6-wplt/ ","time":"2023-07-19T16:44:12.742082021Z"} {"type":"line","content":"[200 OK, 5.77kB, 177ms]\n","time":"2023-07-19T16:44:12.924856642Z"} {"type":"line","content":" ✓ Status code is 200\n","time":"2023-07-19T16:44:12.979816076Z"} {"type":"line","content":"\n┌─────────────────────────┬────────────────────┬───────────────────┐\n│ │ executed │ failed │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ iterations │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ requests │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ test-scripts │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ prerequest-scripts │ 0 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ assertions │ 1 │ 0 │\n├─────────────────────────┴────────────────────┴───────────────────┤\n│ total run duration: 273ms │\n├──────────────────────────────────────────────────────────────────┤\n│ total data received: 5.38kB (approx) │\n├──────────────────────────────────────────────────────────────────┤\n│ average response time: 177ms [min: 177ms, max: 177ms, s.d.: 0µs] │\n└──────────────────────────────────────────────────────────────────┘\n","time":"2023-07-19T16:44:13.012340811Z"} {"type":"line","content":"✅ Execution succeeded","time":"2023-07-19T16:44:13.028885065Z"} {"type":"line","content":"✅ Got Newman result successfully","time":"2023-07-19T16:44:13.031586156Z"} {"type":"line","content":"✅ Mapped Newman result successfully","time":"2023-07-19T16:44:13.031609991Z"} {"type":"result","result":{"status":"passed","output":"newman\n\nSmokeTests\n\n→ google-smoke-test\n GET https://core-eks-test.poppcore.co/v6-wplt/ [200 OK, 5.77kB, 177ms]\n ✓ Status code is 200\n\n┌─────────────────────────┬────────────────────┬───────────────────┐\n│ │ executed │ failed │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ iterations │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ requests │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ test-scripts │ 1 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ prerequest-scripts │ 0 │ 0 │\n├─────────────────────────┼────────────────────┼───────────────────┤\n│ assertions │ 1 │ 0 │\n├─────────────────────────┴────────────────────┴───────────────────┤\n│ total run duration: 273ms │\n├──────────────────────────────────────────────────────────────────┤\n│ total data received: 5.38kB (approx) │\n├──────────────────────────────────────────────────────────────────┤\n│ average response time: 177ms [min: 177ms, max: 177ms, s.d.: 0µs] │\n└──────────────────────────────────────────────────────────────────┘\n","outputType":"text/plain","steps":[{"name":"google-smoke-test","duration":"177ms","status":"passed","assertionResults":[{"name":"Status code is 200","status":"passed"}]}]},"time":"2023-07-19T16:44:13.031615971Z"}

image

Note: API server did not restart during this test. API server logs: {"level":"info","ts":1689784991.760531,"caller":"event/emitter.go:131","msg":"event published","id":"1162e9a1-8426-4683-8b79-f0837878920d","type":"start-test","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"test-type=postman-collection executor=postman-executor ","topic":"events.all","error":null} {"level":"info","ts":1689784991.7620232,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e171a30d-11ea-49e1-9200-6f0143d8aaa7","event":"start-test"} {"level":"info","ts":1689784991.7620661,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"a71efe8c-e8ad-409b-b93c-b0cb1fb1ad23","event":"start-test"} {"level":"info","ts":1689784991.7620747,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"79dbed14-ed93-4345-b275-ef382a0c0ab4","event":"start-test"} {"level":"info","ts":1689784991.7620819,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d1d60650-82f8-4dce-9ec9-07ba6b5c3d74","event":"start-test"} {"level":"info","ts":1689784991.7620916,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"10ebf9d5-bf62-4411-9552-3f4843445b9d","event":"start-test"} {"level":"info","ts":1689784991.7620986,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d437b7ef-2b93-40f8-93a5-9b6cd78d1a3d","event":"start-test"} {"level":"info","ts":1689784991.762105,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"7a6bd3c9-dfbc-474a-8787-cf4be5120f20","event":"start-test"} {"level":"info","ts":1689784991.7621114,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"484d2af1-f7a1-445b-8b12-e118e79ba15f","event":"start-test"} {"level":"info","ts":1689784991.7621176,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"685778ea-ba84-41de-8edc-3c8597b4381d","event":"start-test"} {"level":"info","ts":1689784991.762126,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"df16bc26-446a-431a-a522-23d60f8f16db","event":"start-test"} {"level":"info","ts":1689784991.7621336,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5bba6a2e-d1c7-474a-8af8-a0388d135ee4","event":"start-test"} {"level":"info","ts":1689784991.7621403,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"584a82ca-a89a-4ab1-98db-fe71056de9e2","event":"start-test"} {"level":"info","ts":1689784991.7621472,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"b47ba67f-f7ef-4579-9b2c-de92d5c01965","event":"start-test"} {"level":"info","ts":1689784991.7621572,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"844577d0-b3f8-4398-9834-33f1828045ee","event":"start-test"} {"level":"info","ts":1689784991.7621639,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"a34ba239-d20a-45aa-9814-9edd1fd71784","event":"start-test"} {"level":"info","ts":1689784991.7621708,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"c6d2ba05-8172-48db-8d5d-b461f71e43af","event":"start-test"} {"level":"info","ts":1689784991.7621777,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"4afefea1-92c2-424d-ade6-23b0441eb7f3","event":"start-test"} {"level":"info","ts":1689784991.7621856,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"7dee262b-3e70-46eb-ac3e-e2b8e00c9478","event":"start-test"} {"level":"info","ts":1689784991.7621927,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e326bc58-e4cc-4512-a399-f623f52892fa","event":"start-test"} {"level":"info","ts":1689784991.7621999,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e50a1bde-3f18-448c-b743-2a2fe475c78a","event":"start-test"} {"level":"info","ts":1689784991.7622068,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5f55be9e-11e8-4819-9f0d-6128f31d3980","event":"start-test"} {"level":"info","ts":1689784991.7622135,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"1d431e10-6ce6-48e0-8234-5dca573401eb","event":"start-test"} {"level":"info","ts":1689784991.7622201,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"4f8527d5-4f2f-4aef-84d0-4aa021f488ad","event":"start-test"} {"level":"info","ts":1689784991.7622297,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"703ce5b3-61a2-4c29-9ab3-9d17a121a211","event":"start-test"} {"level":"info","ts":1689784991.762239,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5e40b969-25d9-4a05-aa07-f6a2567ef567","event":"start-test"} {"level":"info","ts":1689784991.7622457,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"349dc18e-e880-46d3-8dc3-77c1793d40fd","event":"start-test"} {"level":"info","ts":1689784991.7622528,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"0ef3f43c-28f6-44d8-b032-dff4b3e549c9","event":"start-test"} {"level":"info","ts":1689784991.7622597,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"24b010e2-73d3-47ed-a520-530bcb78307d","event":"start-test"} {"level":"info","ts":1689784991.7622666,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5cba9634-cf56-4bc2-b15c-10d0bda27c7e","event":"start-test"} {"level":"info","ts":1689784991.7622726,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"31e9913c-b592-4312-91e1-9a6e2756ac41","event":"start-test"} {"level":"info","ts":1689784991.7622786,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d55a6596-a585-4872-90f4-784c55ae3755","event":"start-test"} {"level":"info","ts":1689784991.7623684,"caller":"event/emitter.go:176","msg":"notification result","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"websocket.allevents","selector":"","metadata":{"clients":"[]","events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"websocket.allevents","selector":""},"error":"message sent to not all clients, failed: e171a30d-11ea-49e1-9200-6f0143d8aaa7, a71efe8c-e8ad-409b-b93c-b0cb1fb1ad23, 79dbed14-ed93-4345-b275-ef382a0c0ab4, d1d60650-82f8-4dce-9ec9-07ba6b5c3d74, 10ebf9d5-bf62-4411-9552-3f4843445b9d, d437b7ef-2b93-40f8-93a5-9b6cd78d1a3d, 7a6bd3c9-dfbc-474a-8787-cf4be5120f20, 484d2af1-f7a1-445b-8b12-e118e79ba15f, 685778ea-ba84-41de-8edc-3c8597b4381d, df16bc26-446a-431a-a522-23d60f8f16db, 5bba6a2e-d1c7-474a-8af8-a0388d135ee4, 584a82ca-a89a-4ab1-98db-fe71056de9e2, b47ba67f-f7ef-4579-9b2c-de92d5c01965, 844577d0-b3f8-4398-9834-33f1828045ee, a34ba239-d20a-45aa-9814-9edd1fd71784, c6d2ba05-8172-48db-8d5d-b461f71e43af, 4afefea1-92c2-424d-ade6-23b0441eb7f3, 7dee262b-3e70-46eb-ac3e-e2b8e00c9478, e326bc58-e4cc-4512-a399-f623f52892fa, e50a1bde-3f18-448c-b743-2a2fe475c78a, 5f55be9e-11e8-4819-9f0d-6128f31d3980, 1d431e10-6ce6-48e0-8234-5dca573401eb, 4f8527d5-4f2f-4aef-84d0-4aa021f488ad, 703ce5b3-61a2-4c29-9ab3-9d17a121a211, 5e40b969-25d9-4a05-aa07-f6a2567ef567, 349dc18e-e880-46d3-8dc3-77c1793d40fd, 0ef3f43c-28f6-44d8-b032-dff4b3e549c9, 24b010e2-73d3-47ed-a520-530bcb78307d, 5cba9634-cf56-4bc2-b15c-10d0bda27c7e, 31e9913c-b592-4312-91e1-9a6e2756ac41"} {"level":"info","ts":1689784991.7623937,"caller":"event/emitter.go:177","msg":"listener notified","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"websocket.allevents","selector":"","metadata":{"clients":"[]","events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"websocket.allevents","selector":""},"id":"1162e9a1-8426-4683-8b79-f0837878920d","type":"start-test","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"test-type=postman-collection executor=postman-executor ","topic":"events.all"} {"level":"info","ts":1689784991.7625437,"caller":"slack/slack.go:225","msg":"Execution changed","status":"running"} {"level":"warn","ts":1689784991.8311045,"caller":"slack/slack.go:123","msg":"error while getting bot channels","error":"not_authed"} {"level":"info","ts":1689784991.8311281,"caller":"event/emitter.go:176","msg":"notification result","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"slack","selector":"","metadata":{"events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"slack","selector":""},"error":"not_authed"} {"level":"info","ts":1689784991.8311644,"caller":"event/emitter.go:177","msg":"listener notified","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"slack","selector":"","metadata":{"events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"slack","selector":""},"id":"1162e9a1-8426-4683-8b79-f0837878920d","type":"start-test","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"executor=postman-executor test-type=postman-collection ","topic":"events.all"} {"level":"info","ts":1689784992.298226,"caller":"scheduler/test_scheduler.go:115","msg":"test started","executionId":"64b8129f530875ebaf737959","status":"running"} {"level":"error","ts":1689785055.5249324,"caller":"client/job.go:341","msg":"get pod logs error","executionID":"64b8129f530875ebaf737959","type":"async","error":"Get \"https://10.0.131.244:10250/containerLogs/shared/64b8129f530875ebaf737959-hrnd2/64b8129f530875ebaf737959-init\": remote error: tls: internal error","stacktrace":"github.com/kubeshop/testkube/pkg/executor/client.(*JobExecutor).updateResultsFromPod\n\t/home/runner/work/testkube/testkube/pkg/executor/client/job.go:341\ngithub.com/kubeshop/testkube/pkg/executor/client.(*JobExecutor).Execute.func1\n\t/home/runner/work/testkube/testkube/pkg/executor/client/job.go:204"} {"level":"info","ts":1689785055.5267868,"caller":"client/job.go:410","msg":"execution ended, saving result","executionID":"64b8129f530875ebaf737959","type":"async","executionId":"64b8129f530875ebaf737959","status":"failed"} {"level":"info","ts":1689785055.54196,"caller":"event/emitter.go:131","msg":"event published","id":"62bd32fe-acc9-4d6f-8227-83939c4d9a9a","type":"end-test-failed","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"test-type=postman-collection executor=postman-executor ","topic":"events.all","error":null} {"level":"info","ts":1689785055.5433874,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e171a30d-11ea-49e1-9200-6f0143d8aaa7","event":"end-test-failed"} {"level":"info","ts":1689785055.5434732,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"a71efe8c-e8ad-409b-b93c-b0cb1fb1ad23","event":"end-test-failed"} {"level":"info","ts":1689785055.543499,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"79dbed14-ed93-4345-b275-ef382a0c0ab4","event":"end-test-failed"} {"level":"info","ts":1689785055.5435195,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d1d60650-82f8-4dce-9ec9-07ba6b5c3d74","event":"end-test-failed"} {"level":"info","ts":1689785055.5435598,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"10ebf9d5-bf62-4411-9552-3f4843445b9d","event":"end-test-failed"} {"level":"info","ts":1689785055.5435925,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d437b7ef-2b93-40f8-93a5-9b6cd78d1a3d","event":"end-test-failed"} {"level":"info","ts":1689785055.5436373,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"7a6bd3c9-dfbc-474a-8787-cf4be5120f20","event":"end-test-failed"} {"level":"info","ts":1689785055.5436676,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"484d2af1-f7a1-445b-8b12-e118e79ba15f","event":"end-test-failed"} {"level":"info","ts":1689785055.5436924,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"685778ea-ba84-41de-8edc-3c8597b4381d","event":"end-test-failed"} {"level":"info","ts":1689785055.5437489,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"df16bc26-446a-431a-a522-23d60f8f16db","event":"end-test-failed"} {"level":"info","ts":1689785055.5437758,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5bba6a2e-d1c7-474a-8af8-a0388d135ee4","event":"end-test-failed"} {"level":"info","ts":1689785055.5437968,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"584a82ca-a89a-4ab1-98db-fe71056de9e2","event":"end-test-failed"} {"level":"info","ts":1689785055.5438704,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"b47ba67f-f7ef-4579-9b2c-de92d5c01965","event":"end-test-failed"} {"level":"info","ts":1689785055.5439215,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"844577d0-b3f8-4398-9834-33f1828045ee","event":"end-test-failed"} {"level":"info","ts":1689785055.5438936,"caller":"slack/slack.go:225","msg":"Execution changed","status":"failed"} {"level":"info","ts":1689785055.5439513,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"a34ba239-d20a-45aa-9814-9edd1fd71784","event":"end-test-failed"} {"level":"info","ts":1689785055.544027,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"c6d2ba05-8172-48db-8d5d-b461f71e43af","event":"end-test-failed"} {"level":"info","ts":1689785055.5440962,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"4afefea1-92c2-424d-ade6-23b0441eb7f3","event":"end-test-failed"} {"level":"info","ts":1689785055.5441422,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"7dee262b-3e70-46eb-ac3e-e2b8e00c9478","event":"end-test-failed"} {"level":"info","ts":1689785055.5441964,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e326bc58-e4cc-4512-a399-f623f52892fa","event":"end-test-failed"} {"level":"info","ts":1689785055.544237,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"e50a1bde-3f18-448c-b743-2a2fe475c78a","event":"end-test-failed"} {"level":"info","ts":1689785055.54431,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5f55be9e-11e8-4819-9f0d-6128f31d3980","event":"end-test-failed"} {"level":"info","ts":1689785055.5443223,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"1d431e10-6ce6-48e0-8234-5dca573401eb","event":"end-test-failed"} {"level":"info","ts":1689785055.5443294,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"4f8527d5-4f2f-4aef-84d0-4aa021f488ad","event":"end-test-failed"} {"level":"info","ts":1689785055.5443833,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"703ce5b3-61a2-4c29-9ab3-9d17a121a211","event":"end-test-failed"} {"level":"info","ts":1689785055.544396,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5e40b969-25d9-4a05-aa07-f6a2567ef567","event":"end-test-failed"} {"level":"info","ts":1689785055.544439,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"349dc18e-e880-46d3-8dc3-77c1793d40fd","event":"end-test-failed"} {"level":"info","ts":1689785055.5444922,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"0ef3f43c-28f6-44d8-b032-dff4b3e549c9","event":"end-test-failed"} {"level":"info","ts":1689785055.5445,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"24b010e2-73d3-47ed-a520-530bcb78307d","event":"end-test-failed"} {"level":"info","ts":1689785055.5445487,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"5cba9634-cf56-4bc2-b15c-10d0bda27c7e","event":"end-test-failed"} {"level":"info","ts":1689785055.5446007,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"31e9913c-b592-4312-91e1-9a6e2756ac41","event":"end-test-failed"} {"level":"info","ts":1689785055.5446143,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d55a6596-a585-4872-90f4-784c55ae3755","event":"end-test-failed"} {"level":"info","ts":1689785055.5447857,"caller":"event/emitter.go:176","msg":"notification result","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"websocket.allevents","selector":"","metadata":{"clients":"[]","events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"websocket.allevents","selector":""},"error":"message sent to not all clients, failed: e171a30d-11ea-49e1-9200-6f0143d8aaa7, a71efe8c-e8ad-409b-b93c-b0cb1fb1ad23, 79dbed14-ed93-4345-b275-ef382a0c0ab4, d1d60650-82f8-4dce-9ec9-07ba6b5c3d74, 10ebf9d5-bf62-4411-9552-3f4843445b9d, d437b7ef-2b93-40f8-93a5-9b6cd78d1a3d, 7a6bd3c9-dfbc-474a-8787-cf4be5120f20, 484d2af1-f7a1-445b-8b12-e118e79ba15f, 685778ea-ba84-41de-8edc-3c8597b4381d, df16bc26-446a-431a-a522-23d60f8f16db, 5bba6a2e-d1c7-474a-8af8-a0388d135ee4, 584a82ca-a89a-4ab1-98db-fe71056de9e2, b47ba67f-f7ef-4579-9b2c-de92d5c01965, 844577d0-b3f8-4398-9834-33f1828045ee, a34ba239-d20a-45aa-9814-9edd1fd71784, c6d2ba05-8172-48db-8d5d-b461f71e43af, 4afefea1-92c2-424d-ade6-23b0441eb7f3, 7dee262b-3e70-46eb-ac3e-e2b8e00c9478, e326bc58-e4cc-4512-a399-f623f52892fa, e50a1bde-3f18-448c-b743-2a2fe475c78a, 5f55be9e-11e8-4819-9f0d-6128f31d3980, 1d431e10-6ce6-48e0-8234-5dca573401eb, 4f8527d5-4f2f-4aef-84d0-4aa021f488ad, 703ce5b3-61a2-4c29-9ab3-9d17a121a211, 5e40b969-25d9-4a05-aa07-f6a2567ef567, 349dc18e-e880-46d3-8dc3-77c1793d40fd, 0ef3f43c-28f6-44d8-b032-dff4b3e549c9, 24b010e2-73d3-47ed-a520-530bcb78307d, 5cba9634-cf56-4bc2-b15c-10d0bda27c7e, 31e9913c-b592-4312-91e1-9a6e2756ac41"} {"level":"info","ts":1689785055.5448792,"caller":"event/emitter.go:177","msg":"listener notified","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"websocket.allevents","selector":"","metadata":{"clients":"[]","events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"websocket.allevents","selector":""},"id":"62bd32fe-acc9-4d6f-8227-83939c4d9a9a","type":"end-test-failed","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"executor=postman-executor test-type=postman-collection ","topic":"events.all"} {"level":"warn","ts":1689785055.6134856,"caller":"slack/slack.go:123","msg":"error while getting bot channels","error":"not_authed"} {"level":"info","ts":1689785055.6135156,"caller":"event/emitter.go:176","msg":"notification result","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"slack","selector":"","metadata":{"events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"slack","selector":""},"error":"not_authed"} {"level":"info","ts":1689785055.6135297,"caller":"event/emitter.go:177","msg":"listener notified","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"slack","selector":"","metadata":{"events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"slack","selector":""},"id":"62bd32fe-acc9-4d6f-8227-83939c4d9a9a","type":"end-test-failed","resource":"","resourceId":"","executionName":"google-smoke-test-3","executionId":"64b8129f530875ebaf737959","labels":"executor=postman-executor test-type=postman-collection ","topic":"events.all"} {"level":"error","ts":1689785055.8546896,"caller":"client/job.go:206","msg":"update results from jobs pod error","executionID":"64b8129f530875ebaf737959","type":"async","error":"Get \"https://10.0.131.244:10250/containerLogs/shared/64b8129f530875ebaf737959-hrnd2/64b8129f530875ebaf737959-init\": remote error: tls: internal error","stacktrace":"github.com/kubeshop/testkube/pkg/executor/client.(*JobExecutor).Execute.func1\n\t/home/runner/work/testkube/testkube/pkg/executor/client/job.go:206"}

This error at the end of the API logs might be key? update results from jobs pod error

vsukhin commented 1 year ago

yes, execution result was not updated. remote error: tls: internal error. do you use istio or any mesh?

AtomicMapache commented 1 year ago

@vsukhin I was originally using AWS AppMesh to integrate prometheus for collection of pod metrics to AWS CloudWatch. I wonder if this could have caused some issues? I have since uninstalled the prometheus chart and am still having the above documented problems. Worst case I might just nuke the cluster and do a fresh install with testkube first.

vsukhin commented 1 year ago

it looks like mesh, because of tls calls, which we don't have. So yes, better to try on clear cluster

AtomicMapache commented 1 year ago

Originally, my AWS EKS cluster was being deployed via terraform, but after redeploying a new cluster via "eksctl create cluster ...", things appear to be working. I didn't explicitly denote the use of App Mesh or any other mesh in my .tf scripts, but nonetheless this was the likely culprit given the above mentioned logs. Thanks for the help @vsukhin ! Closing the ticket.