kubeshop / testkube

☸️ Kubernetes-native testing framework for test execution and orchestration
https://testkube.io
Other
1.36k stars 131 forks source link

PodInitializing error #1502

Closed damien-m-abrahams closed 1 year ago

damien-m-abrahams commented 2 years ago

I've added a Postman collection test to an Azure DevOps repository and created a test in TestKube. When tying to run the test the following error occurs:

Latest execution
ID:        627b867c720181823b77c58e
Name:      easily-electric-chow
Type:      postman/collection
Duration:  7.563438451s

Status test execution failed:

⨯ container "627b867c720181823b77c58e" in pod "627b867c720181823b77c58e-8bdfh" is waiting to start: PodInitializing

Error: exit status 1

I've tried to create the test with and without a token and the same error occurs. I'm not sure whether I've missed something in terms of creating the test in TestKube or my K8 environment is the problem?

vsukhin commented 2 years ago

Hey @damien-m-abrahams So, the work of the pod initialization container is to prepare data for a main test executor pod. Technically, simply to store data in a pod volume. I never used Azure Devops repos, do they support token based access similar to github https git access? Also, the message itself is about an awaiting container. Can you please provide more details? kubectl get pods -n testkube kubectl describe pod 627b867c720181823b77c58e-8bdfh -n testkube kubectl logs 627b867c720181823b77c58e-8bdfh -n testkube

damien-m-abrahams commented 2 years ago

Hi again,

Here are the results of the commands: C:\Development\testkube_1.0.16_Windows_x86_64>kubectl get pods -n testkube NAME READY STATUS RESTARTS AGE 627caa34720181823b77c59a-jb5cl 0/1 Init:Error 0 18s testkube-api-server-664766b7bb-sz9tv 1/1 Running 0 5d18h testkube-dashboard-7f6f6dc6f-5k6sr 1/1 Running 0 5d18h testkube-minio-testkube-79d99f9746-khl9c 1/1 Running 0 5d18h testkube-mongodb-5d766b7d58-h79r5 1/1 Running 0 5d18h testkube-operator-controller-manager-78bd65f589-cfk9z 2/2 Running 0 5d18h

C:\Development\testkube_1.0.16_Windows_x86_64>kubectl describe pod 627caa34720181823b77c59a-jb5cl -n testkube Name: 627caa34720181823b77c59a-jb5cl Namespace: testkube Priority: 0 Node: worker1/10.3.11.11 Start Time: Thu, 12 May 2022 14:33:24 +0800 Labels: controller-uid=cf6c638f-6551-4043-8ac4-2b46dab76c32 job-name=627caa34720181823b77c59a Annotations: cni.projectcalico.org/containerID: 5ac715114d0e946b6b87d5a5c09e786afaffcebdc941e6cd1db1aedf72034410 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs: Status: Failed IP: 10.244.235.179 IPs: IP: 10.244.235.179 Controlled By: Job/627caa34720181823b77c59a Init Containers: 627caa34720181823b77c59a-init: Container ID: containerd://4ca7929e47608fdc784d6434cebfa2f2d06287d948f1a6151855b4de78df2b12 Image: kubeshop/testkube-executor-init:0.7.8 Image ID: docker.io/kubeshop/testkube-executor-init@sha256:0233a7a9b6a27c9fa4df042e33ba76275b56dc388cb3a3b7c6aa7bb4f02d7667 Port: Host Port: Command: /bin/runner {"id":"627caa34720181823b77c59a","testName":"white-flowers","testType":"postman/collection","name":"moderately-direct-condor","content":{"type":"git-dir","repository":{"type":"git","uri":"https://dev.azure.com/damienabrahams/tests/_git/testkube","branch":"master","path":"Services/Test"}},"startTime":"2022-05-12T06:33:24.156881961Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Terminated Reason: Error Exit Code: 1 Started: Thu, 12 May 2022 14:33:28 +0800 Finished: Thu, 12 May 2022 14:33:28 +0800 Ready: False Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'white-flowers-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'white-flowers-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rh56m (ro) Containers: 627caa34720181823b77c59a: Container ID: Image: kubeshop/testkube-postman-executor:0.7.8 Image ID: Port: Host Port: Command: /bin/runner {"id":"627caa34720181823b77c59a","testName":"white-flowers","testType":"postman/collection","name":"moderately-direct-condor","content":{"type":"git-dir","repository":{"type":"git","uri":"https://dev.azure.com/damienabrahams/tests/_git/testkube","branch":"master","path":"Services/Test"}},"startTime":"2022-05-12T06:33:24.156881961Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'white-flowers-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'white-flowers-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rh56m (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: data-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-rh56m: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message


Normal Scheduled 49s default-scheduler Successfully assigned testkube/627caa34720181823b77c59a-jb5cl to worker1 Normal Pulling 49s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 47s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 2.578894091s Normal Created 47s kubelet Created container 627caa34720181823b77c59a-init Normal Started 46s kubelet Started container 627caa34720181823b77c59a-init

The describe command doesn't show anything more...

C:\Development\testkube_1.0.16_Windows_x86_64>kubectl logs 627caa34720181823b77c59a-jb5cl -n testkube Defaulted container "627caa34720181823b77c59a" out of: 627caa34720181823b77c59a, 627caa34720181823b77c59a-init (init) Error from server (BadRequest): container "627caa34720181823b77c59a" in pod "627caa34720181823b77c59a-jb5cl" is waiting to start: PodInitializing

damien-m-abrahams commented 2 years ago

I did specify a token when I created the test. I've also added the test to a private GitHub repo and I get the same result.

The create command I use is this:

kubectl testkube create test --git-uri https://dev.azure.com/damienabrahams/tests/_git/testkube --git-branch master --git-path Services/Test --git-token XXXXXXXXXXXXXXXXXX --name white-flowers --type postman/collection

vsukhin commented 2 years ago

Thank you @damien-m-abrahams ! Yes, it really looks like git repo checkout issue. Need to test it for azure dev

vsukhin commented 2 years ago

Hey, @damien-m-abrahams I created test account and test repo in Azure Devops

You can use this command to create and then run a test: kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token jmzqsaiblvoxyo2275hqgifxi6filj3vpmco37hbae5dx27amuja --name red-flowers --type postman/collection --test-content-type git-file

Pay attention to test content type parameter. We don't support git-dir for postman tests (Token will be alive for a few days)

Also, you didn't mention k8s cluster you use, but in my personal experience I frequently see that Minikube sometimes fails to solve dns names like dev.azure.com or github.com till you don't stop/start it. That's probably why git checkout fails in your k8s cluster

damien-m-abrahams commented 2 years ago

Hi @vsukhin I am using Minikube but I do have access to another K8 environment, which isn't Minikube. I'll try out the different create command and let you know

vsukhin commented 2 years ago

hey @damien-m-abrahams did it help?

bode-praetorian commented 2 years ago

Having a similar problem. When trying to add a cypress or k6 test from private git repo will fail to initialize though I provide the git-username and git-token (PAT) in the test creation. Looking at the pod describe below, you see the runner never gets the git-username and token set.

kubectl testkube update test --git-uri https://github.com/praetorian-inc/chariot-cypress.git --git-branch main --git-path e2e --name chariot-cypress --type cypress/project --git-username XXX --git-token XXX

Name: 62853cf96fd2e48c9d3d241e-mzfd7 Namespace: testkube Priority: 0 Node: k3d-dev-server-0/172.18.0.2 Start Time: Wed, 18 May 2022 12:37:45 -0600 Labels: controller-uid=ae362f7c-99dd-435d-99e0-5ecbbdca3a82 job-name=62853cf96fd2e48c9d3d241e Annotations: Status: Failed IP: 10.42.0.210 IPs: IP: 10.42.0.210 Controlled By: Job/62853cf96fd2e48c9d3d241e Init Containers: 62853cf96fd2e48c9d3d241e-init: Container ID: containerd://d0e0d52bd4d92dd5d69319504282bc885e27f3e3655674eeb0ff24d1e04f2530 Image: kubeshop/testkube-executor-init:0.7.8 Image ID: docker.io/kubeshop/testkube-executor-init@sha256:0233a7a9b6a27c9fa4df042e33ba76275b56dc388cb3a3b7c6aa7bb4f02d7667 Port: Host Port: Command: /bin/runner {"id":"62853cf96fd2e48c9d3d241e","testName":"chariot-cypress","testNamespace":"testkube","testType":"cypress/project","name":"brightly-immense-raccoon","content":{"type":"git-file","repository":{"type":"git","uri":"https://github.com/praetorian-inc/chariot-cypress.git","branch":"main","path":"e2e"}},"startTime":"2022-05-18T18:37:45.433751866Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Terminated Reason: Error Exit Code: 1 Started: Wed, 18 May 2022 12:37:47 -0600 Finished: Wed, 18 May 2022 12:37:47 -0600 Ready: False Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'chariot-cypress-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'chariot-cypress-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x5hzl (ro) Containers: 62853cf96fd2e48c9d3d241e: Container ID: Image: kubeshop/testkube-cypress-executor:0.7.8 Image ID: Port: Host Port: Command: /bin/runner {"id":"62853cf96fd2e48c9d3d241e","testName":"chariot-cypress","testNamespace":"testkube","testType":"cypress/project","name":"brightly-immense-raccoon","content":{"type":"git-file","repository":{"type":"git","uri":"https://github.com/praetorian-inc/chariot-cypress.git","branch":"main","path":"e2e"}},"startTime":"2022-05-18T18:37:45.433751866Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'chariot-cypress-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'chariot-cypress-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x5hzl (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: data-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-x5hzl: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message


Normal Scheduled 53s default-scheduler Successfully assigned testkube/62853cf96fd2e48c9d3d241e-mzfd7 to k3d-dev-server-0 Normal Pulling 53s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 52s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 724.283561ms Normal Created 52s kubelet Created container 62853cf96fd2e48c9d3d241e-init Normal Started 52s kubelet Started container 62853cf96fd2e48c9d3d241e-init

vsukhin commented 2 years ago

Thank you @bode-praetorian Let us check it

vsukhin commented 2 years ago

hey @bode-praetorian. Yes, from the logs it looks like an error for checking out your private git repo. A bit too hard to say, what is an exact reason, but sometimes Kubernetes emulators like Minikube, Kind, K3D have issues with dns solving for external websites or https cert validation. Can we try to check the logs of init container kubectl logs 62853cf96fd2e48c9d3d241e-mzfd7 62853cf96fd2e48c9d3d241e-init -n testkube?

vsukhin commented 2 years ago

Also, have you tried to run a test with public repo?

bode-praetorian commented 2 years ago

Great question. So, I resolved the SSO git access bit. Seems like I am getting into the private repos now. That said, I am getting some other errors. I did try: kubectl testkube create test --git-uri https://github.com/kubeshop/testkube-executor-k6.git --git-branch main --git-path examples --type "k6/script" --name k6-test-script-git

That failed with: k6 test script /data/repo/run not found. So there still seems to be some oddness. So, I also ran: kubectl testkube create test --git-uri https://github.com/kubeshop/testkube-executor-cypress.git --git-branch main --git-path examples --name kubeshop-cypress --type cypress/project

That one worked fine. Yet when I run my own cypress repo I get "passing cypress test as single file not implemented yet. " Though I am not calling in the docker a single spec but running cypress run. Perhaps it isn't pathing right in the container; still investigating.

On Thu, May 19, 2022 at 11:15 AM Vladislav Sukhin @.***> wrote:

Also, have you tried to run a test with public repo?

— Reply to this email directly, view it on GitHub https://github.com/kubeshop/testkube/issues/1502#issuecomment-1131975767, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVPYRPCMF2L66LEB6GGTNV3VKZZK3ANCNFSM5VULVHKA . You are receiving this because you were mentioned.Message ID: @.***>

damien-m-abrahams commented 2 years ago

I tried to use TestKube with vanilla K8 and Azure DevOps. I specify the username and the PAT token with this command:

kubectl testkube create test --git-uri https://dev.azure.com/damienabrahams/tests/_git/testkube --git-branch master --git-path Services/Test/WeatherForecast.postman_collection.json --git-username damienabrahams --git-token XXXXXXXXXXXXXXXXXX --name white-flowers --type postman/collection --test-content-type git-file

kubectl logs 62831f67dbcef024925039ff-fkpzk -n testkube {"type":"event","content":"running test [62831f67dbcef024925039ff]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nfatal: could not read Password for 'https://dev.azure.com': No such device or address\n"}

I also tried different URIs and Tokens which had different errors.

Interestingly I also tried your repo last week (sorry I've been sidetracked with other things this week). And it didn't work:

kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token jmzqsaiblvoxyo2275hqgifxi6filj3vpmco37hbae5dx27amuja --name red-flowers --type postman/collection --test-content-type git-file

The error was similar to not being able to resolve https://dev.azure/com. I'm wondering if a K8 egress rule needs to be set up?

vsukhin commented 2 years ago

hey @bode-praetorian Yeah, indeed, git authentication should setup properly to access private repos :)

Please check examples for exact test parameters there https://github.com/kubeshop/testkube-executor-k6#examples

You should run it as kubectl testkube run test --args examples/k6-test-script.js --watch k6-test-script-git

We support only directory based cypress tests right now. I don't know which parameters you use for your own cypress test, but it can't be a single file

vsukhin commented 2 years ago

hey @damien-m-abrahams Created one more 1 week token for you. Can you try it and give me a precise error? ae3snb5u5pcjttymk34yyvd6xq3ie7uuo62g2mvxbjzse3ql2alq

The error above looks like the PAT was not provided, and git couldn't read it from terminal

Also can you just try to run test for a public github repo. Does it work for you?

I don't think, that you need to setup k8 egress, still it really depends from your k8s config

damien-m-abrahams commented 2 years ago

Hey @vsukhin here's the output of the commands and responses:

`C:\Users\DamienAbrahams>kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token ae3snb5u5pcjttymk34yyvd6xq3ie7uuo62g2mvxbjzse3ql2alq --name blue-flowers --type postman/collection --test-content-type git-file

████████ ███████ ███████ ████████ ██ ██ ██ ██ ██████ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████ ███████ ██ █████ ██ ██ ██████ █████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ███████ ███████ ██ ██ ██ ██████ ██████ ███████ /tɛst kjub/ by Kubeshop

Test created testkube / blue-flowers 🥇

C:\Users\DamienAbrahams>kubectl testkube get tests

NAME | TYPE | CREATED | LABELS | SCHEDULE | STATUS | EXECUTION ID
+----------------+--------------------+-------------------------------+--------+----------+--------+--------------------------+ black-flowers | postman/collection | 2022-05-17 04:06:56 +0000 UTC | | | failed | 62831f67dbcef024925039ff blue-flowers | postman/collection | 2022-05-23 05:27:54 +0000 UTC | | | |

C:\Users\DamienAbrahams>kubectl testkube run test blue-flowers

████████ ███████ ███████ ████████ ██ ██ ██ ██ ██████ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████ ███████ ██ █████ ██ ██ ██████ █████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ███████ ███████ ██ ██ ██ ██████ ██████ ███████ /tɛst kjub/ by Kubeshop

Type : postman/collection Name : blue-flowers Execution ID : 628b1b82dbcef02492503a21 Execution name: secondly-relieved-arachnid

Test execution started

Watch test execution until complete: $ kubectl testkube watch execution 628b1b82dbcef02492503a21

Use following command to get test execution details: $ kubectl testkube get execution 628b1b82dbcef02492503a21

C:\Users\DamienAbrahams>kubectl get pods -n testkube NAME READY STATUS RESTARTS AGE 628b1b82dbcef02492503a21-568w5 0/1 Init:0/1 0 8s testkube-api-server-664766b7bb-sz9tv 1/1 Running 1 (6d3h ago) 16d testkube-dashboard-7f6f6dc6f-5k6sr 1/1 Running 1 (6d3h ago) 16d testkube-minio-testkube-79d99f9746-khl9c 1/1 Running 1 (6d3h ago) 16d testkube-mongodb-5d766b7d58-h79r5 1/1 Running 1 (6d3h ago) 16d testkube-operator-controller-manager-78bd65f589-cfk9z 2/2 Running 20 (111m ago) 16d

C:\Users\DamienAbrahams>kubectl describe pod 628b1b82dbcef02492503a21-568w5 -n testkube Name: 628b1b82dbcef02492503a21-568w5 Namespace: testkube Priority: 0 Node: worker1/10.3.11.11 Start Time: Mon, 23 May 2022 13:28:36 +0800 Labels: controller-uid=d5e9c797-c617-47e8-a469-549a6e4790e0 job-name=628b1b82dbcef02492503a21 Annotations: cni.projectcalico.org/containerID: 0b0d1dd5406cb3dbbadccb2c258a432265ab19ad5d509510f36056dad0b94f63 cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs: Status: Failed IP: 10.244.235.159 IPs: IP: 10.244.235.159 Controlled By: Job/628b1b82dbcef02492503a21 Init Containers: 628b1b82dbcef02492503a21-init: Container ID: containerd://d151bbad105d2ab84117fae6638429d7908416a7ed03835e8e2397a82a1ebbc4 Image: kubeshop/testkube-executor-init:0.7.8 Image ID: docker.io/kubeshop/testkube-executor-init@sha256:0233a7a9b6a27c9fa4df042e33ba76275b56dc388cb3a3b7c6aa7bb4f02d7667 Port: Host Port: Command: /bin/runner {"id":"628b1b82dbcef02492503a21","testName":"blue-flowers","testType":"postman/collection","name":"secondly-relieved-arachnid","content":{"type":"git-file","repository":{"type":"git","uri":"https://dev.azure.com/vladislavsukhin/Test/_git/Test","branch":"master","path":"Service/Test/TODO.postman_collection.json"}},"startTime":"2022-05-23T05:28:34.089760335Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 23 May 2022 13:28:40 +0800 Finished: Mon, 23 May 2022 13:28:50 +0800 Ready: True Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'blue-flowers-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'blue-flowers-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7r6lf (ro) Containers: 628b1b82dbcef02492503a21: Container ID: containerd://244971de8b15f5e05c9db3f8c41a9e2a36183bd72f985c0744ae2de3fd2bcac8 Image: kubeshop/testkube-postman-executor:0.7.8 Image ID: docker.io/kubeshop/testkube-postman-executor@sha256:206a992410b54a1983eccc48a89ee71e1e238974175ed67cd3a57aa296ce3b1f Port: Host Port: Command: /bin/runner {"id":"628b1b82dbcef02492503a21","testName":"blue-flowers","testType":"postman/collection","name":"secondly-relieved-arachnid","content":{"type":"git-file","repository":{"type":"git","uri":"https://dev.azure.com/vladislavsukhin/Test/_git/Test","branch":"master","path":"Service/Test/TODO.postman_collection.json"}},"startTime":"2022-05-23T05:28:34.089760335Z","endTime":"0001-01-01T00:00:00Z","executionResult":{"status":"running"}} State: Terminated Reason: Error Exit Code: 1 Started: Mon, 23 May 2022 13:28:54 +0800 Finished: Mon, 23 May 2022 13:28:54 +0800 Ready: False Restart Count: 0 Environment: DEBUG: RUNNER_ENDPOINT: testkube-minio-service-testkube:9000 RUNNER_ACCESSKEYID: minio RUNNER_SECRETACCESSKEY: minio123 RUNNER_LOCATION: RUNNER_TOKEN: RUNNER_SSL: false RUNNER_SCRAPPERENABLED: true RUNNER_DATADIR: /data RUNNER_GITUSERNAME: <set to the key 'git-username' in secret 'blue-flowers-secrets'> Optional: false RUNNER_GITTOKEN: <set to the key 'git-token' in secret 'blue-flowers-secrets'> Optional: false Mounts: /data from data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7r6lf (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-7r6lf: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message


Normal Scheduled 32s default-scheduler Successfully assigned testkube/628b1b82dbcef02492503a21-568w5 to worker1 Normal Pulling 31s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 29s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 2.644335553s Normal Created 28s kubelet Created container 628b1b82dbcef02492503a21-init Normal Started 28s kubelet Started container 628b1b82dbcef02492503a21-init Normal Pulling 17s kubelet Pulling image "kubeshop/testkube-postman-executor:0.7.8" Normal Pulled 14s kubelet Successfully pulled image "kubeshop/testkube-postman-executor:0.7.8" in 2.651650952s Normal Created 14s kubelet Created container 628b1b82dbcef02492503a21 Normal Started 14s kubelet Started container 628b1b82dbcef02492503a21

C:\Users\DamienAbrahams>kubectl logs 628b1b82dbcef02492503a21-568w5 -n testkube {"type":"event","content":"running test [628b1b82dbcef02492503a21]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nfatal: could not read Username for 'https://dev.azure.com': No such device or address\n"}`

vsukhin commented 2 years ago

Thank you, @damien-m-abrahams !

  1. Can you try to upgrade to latest testkube release using kubectl testkube upgrade?
  2. Can you create a test for public github repo (you can use one of our exampes) in order to be sure, that git works properly?
  3. The error itself claims about missed username and password, which is really weird, because you supplied them during test creation. Can we try to define them explicitely in uri, like https://vladislavsukhin:ae3snb5u5pcjttymk34yyvd6xq3ie7uuo62g2mvxbjzse3ql2alq @dev.azure.com/vladislavsukhin/Test/_git/Test ?
bode-praetorian commented 2 years ago

Yeah, after I posted I dug around a bit more and had a suspicion it was the git setup.

{"type":"event","content":"running test [628643ab6fd2e48c9d3d2428]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nremote: The `praetorian-inc' organization has enabled or enforced SAML SSO. To access\nremote: this repository, visit https://github.com/orgs/praetorian-inc/sso?authorization_request=AVPYRPFLCVNZLJUCWDLTBKTCQZI35A5PN5ZGOYLONF5GC5DJN5XF62LEZYAHZOG3VVRXEZLEMVXHI2LBNRPWSZGOGQG2KRFPMNZGKZDFNZ2GSYLML52HS4DFVNHWC5LUNBAWGY3FONZQ\nremote: and try your request again.\nfatal: unable to access ' https://github.com/praetorian-inc/chariot-cypress.git/': The requested URL returned error: 403\n"}

It would probably be good to have maybe a bit more of the error handling on the test run to point he user to auth issues rather than the general pod failure message, "contacontainer "628643ab6fd2e48c9d3d2428" in pod "628643ab6fd2e48c9d3d2428-nqz5r" is waiting to start: PodInitializinginer "628643ab6fd2e48c9d3d2428" in pod "628643ab6fd2e48c9d3d2428-nqz5r" is waiting to start: PodInitializing", assuming you can trap that.

On Thu, May 19, 2022 at 5:10 AM Vladislav Sukhin @.***> wrote:

hey @bode-praetorian https://github.com/bode-praetorian. Yes, from the logs it looks like an error of checking out your private git repo. A bit too hard to say, what is an exact reason, but sometimes Kubernetes emulators like Minikube, Kind, K3D have issues with dns solving for external websites or https cert validation. Can we try to check the logs of init container kubectl logs 62853cf96fd2e48c9d3d241e-mzfd7 62853cf96fd2e48c9d3d241e-init -n testkube?

— Reply to this email directly, view it on GitHub https://github.com/kubeshop/testkube/issues/1502#issuecomment-1131555120, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVPYRPAZCIMZ3JBNYSSDRFDVKYOSRANCNFSM5VULVHKA . You are receiving this because you were mentioned.Message ID: @.***>

vsukhin commented 2 years ago

Thank you, @bode-praetorian. Good point, we have an issue about improving init pod logs #2396