Closed damien-m-abrahams closed 1 year ago
Hey @damien-m-abrahams So, the work of the pod initialization container is to prepare data for a main test executor pod. Technically, simply to store data in a pod volume. I never used Azure Devops repos, do they support token based access similar to github https git access? Also, the message itself is about an awaiting container. Can you please provide more details? kubectl get pods -n testkube kubectl describe pod 627b867c720181823b77c58e-8bdfh -n testkube kubectl logs 627b867c720181823b77c58e-8bdfh -n testkube
Hi again,
Here are the results of the commands: C:\Development\testkube_1.0.16_Windows_x86_64>kubectl get pods -n testkube NAME READY STATUS RESTARTS AGE 627caa34720181823b77c59a-jb5cl 0/1 Init:Error 0 18s testkube-api-server-664766b7bb-sz9tv 1/1 Running 0 5d18h testkube-dashboard-7f6f6dc6f-5k6sr 1/1 Running 0 5d18h testkube-minio-testkube-79d99f9746-khl9c 1/1 Running 0 5d18h testkube-mongodb-5d766b7d58-h79r5 1/1 Running 0 5d18h testkube-operator-controller-manager-78bd65f589-cfk9z 2/2 Running 0 5d18h
C:\Development\testkube_1.0.16_Windows_x86_64>kubectl describe pod 627caa34720181823b77c59a-jb5cl -n testkube
Name: 627caa34720181823b77c59a-jb5cl
Namespace: testkube
Priority: 0
Node: worker1/10.3.11.11
Start Time: Thu, 12 May 2022 14:33:24 +0800
Labels: controller-uid=cf6c638f-6551-4043-8ac4-2b46dab76c32
job-name=627caa34720181823b77c59a
Annotations: cni.projectcalico.org/containerID: 5ac715114d0e946b6b87d5a5c09e786afaffcebdc941e6cd1db1aedf72034410
cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
Status: Failed
IP: 10.244.235.179
IPs:
IP: 10.244.235.179
Controlled By: Job/627caa34720181823b77c59a
Init Containers:
627caa34720181823b77c59a-init:
Container ID: containerd://4ca7929e47608fdc784d6434cebfa2f2d06287d948f1a6151855b4de78df2b12
Image: kubeshop/testkube-executor-init:0.7.8
Image ID: docker.io/kubeshop/testkube-executor-init@sha256:0233a7a9b6a27c9fa4df042e33ba76275b56dc388cb3a3b7c6aa7bb4f02d7667
Port:
Normal Scheduled 49s default-scheduler Successfully assigned testkube/627caa34720181823b77c59a-jb5cl to worker1 Normal Pulling 49s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 47s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 2.578894091s Normal Created 47s kubelet Created container 627caa34720181823b77c59a-init Normal Started 46s kubelet Started container 627caa34720181823b77c59a-init
The describe command doesn't show anything more...
C:\Development\testkube_1.0.16_Windows_x86_64>kubectl logs 627caa34720181823b77c59a-jb5cl -n testkube Defaulted container "627caa34720181823b77c59a" out of: 627caa34720181823b77c59a, 627caa34720181823b77c59a-init (init) Error from server (BadRequest): container "627caa34720181823b77c59a" in pod "627caa34720181823b77c59a-jb5cl" is waiting to start: PodInitializing
I did specify a token when I created the test. I've also added the test to a private GitHub repo and I get the same result.
The create command I use is this:
kubectl testkube create test --git-uri https://dev.azure.com/damienabrahams/tests/_git/testkube --git-branch master --git-path Services/Test --git-token XXXXXXXXXXXXXXXXXX --name white-flowers --type postman/collection
Thank you @damien-m-abrahams ! Yes, it really looks like git repo checkout issue. Need to test it for azure dev
Hey, @damien-m-abrahams I created test account and test repo in Azure Devops
You can use this command to create and then run a test: kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token jmzqsaiblvoxyo2275hqgifxi6filj3vpmco37hbae5dx27amuja --name red-flowers --type postman/collection --test-content-type git-file
Pay attention to test content type parameter. We don't support git-dir for postman tests (Token will be alive for a few days)
Also, you didn't mention k8s cluster you use, but in my personal experience I frequently see that Minikube sometimes fails to solve dns names like dev.azure.com or github.com till you don't stop/start it. That's probably why git checkout fails in your k8s cluster
Hi @vsukhin I am using Minikube but I do have access to another K8 environment, which isn't Minikube. I'll try out the different create command and let you know
hey @damien-m-abrahams did it help?
Having a similar problem. When trying to add a cypress or k6 test from private git repo will fail to initialize though I provide the git-username and git-token (PAT) in the test creation. Looking at the pod describe below, you see the runner never gets the git-username and token set.
kubectl testkube update test --git-uri https://github.com/praetorian-inc/chariot-cypress.git --git-branch main --git-path e2e --name chariot-cypress --type cypress/project --git-username XXX --git-token XXX
Name: 62853cf96fd2e48c9d3d241e-mzfd7
Namespace: testkube
Priority: 0
Node: k3d-dev-server-0/172.18.0.2
Start Time: Wed, 18 May 2022 12:37:45 -0600
Labels: controller-uid=ae362f7c-99dd-435d-99e0-5ecbbdca3a82
job-name=62853cf96fd2e48c9d3d241e
Annotations:
Normal Scheduled 53s default-scheduler Successfully assigned testkube/62853cf96fd2e48c9d3d241e-mzfd7 to k3d-dev-server-0 Normal Pulling 53s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 52s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 724.283561ms Normal Created 52s kubelet Created container 62853cf96fd2e48c9d3d241e-init Normal Started 52s kubelet Started container 62853cf96fd2e48c9d3d241e-init
Thank you @bode-praetorian Let us check it
hey @bode-praetorian. Yes, from the logs it looks like an error for checking out your private git repo. A bit too hard to say, what is an exact reason, but sometimes Kubernetes emulators like Minikube, Kind, K3D have issues with dns solving for external websites or https cert validation. Can we try to check the logs of init container kubectl logs 62853cf96fd2e48c9d3d241e-mzfd7 62853cf96fd2e48c9d3d241e-init -n testkube
?
Also, have you tried to run a test with public repo?
Great question. So, I resolved the SSO git access bit. Seems like I am getting into the private repos now. That said, I am getting some other errors. I did try: kubectl testkube create test --git-uri https://github.com/kubeshop/testkube-executor-k6.git --git-branch main --git-path examples --type "k6/script" --name k6-test-script-git
That failed with: k6 test script /data/repo/run not found. So there still seems to be some oddness. So, I also ran: kubectl testkube create test --git-uri https://github.com/kubeshop/testkube-executor-cypress.git --git-branch main --git-path examples --name kubeshop-cypress --type cypress/project
That one worked fine. Yet when I run my own cypress repo I get "passing cypress test as single file not implemented yet. " Though I am not calling in the docker a single spec but running cypress run. Perhaps it isn't pathing right in the container; still investigating.
On Thu, May 19, 2022 at 11:15 AM Vladislav Sukhin @.***> wrote:
Also, have you tried to run a test with public repo?
— Reply to this email directly, view it on GitHub https://github.com/kubeshop/testkube/issues/1502#issuecomment-1131975767, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVPYRPCMF2L66LEB6GGTNV3VKZZK3ANCNFSM5VULVHKA . You are receiving this because you were mentioned.Message ID: @.***>
I tried to use TestKube with vanilla K8 and Azure DevOps. I specify the username and the PAT token with this command:
kubectl testkube create test --git-uri https://dev.azure.com/damienabrahams/tests/_git/testkube --git-branch master --git-path Services/Test/WeatherForecast.postman_collection.json --git-username damienabrahams --git-token XXXXXXXXXXXXXXXXXX --name white-flowers --type postman/collection --test-content-type git-file
kubectl logs 62831f67dbcef024925039ff-fkpzk -n testkube {"type":"event","content":"running test [62831f67dbcef024925039ff]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nfatal: could not read Password for 'https://dev.azure.com': No such device or address\n"}
I also tried different URIs and Tokens which had different errors.
Interestingly I also tried your repo last week (sorry I've been sidetracked with other things this week). And it didn't work:
kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token jmzqsaiblvoxyo2275hqgifxi6filj3vpmco37hbae5dx27amuja --name red-flowers --type postman/collection --test-content-type git-file
The error was similar to not being able to resolve https://dev.azure/com
. I'm wondering if a K8 egress rule needs to be set up?
hey @bode-praetorian Yeah, indeed, git authentication should setup properly to access private repos :)
Please check examples for exact test parameters there https://github.com/kubeshop/testkube-executor-k6#examples
You should run it as kubectl testkube run test --args examples/k6-test-script.js --watch k6-test-script-git
We support only directory based cypress tests right now. I don't know which parameters you use for your own cypress test, but it can't be a single file
hey @damien-m-abrahams Created one more 1 week token for you. Can you try it and give me a precise error? ae3snb5u5pcjttymk34yyvd6xq3ie7uuo62g2mvxbjzse3ql2alq
The error above looks like the PAT was not provided, and git couldn't read it from terminal
Also can you just try to run test for a public github repo. Does it work for you?
I don't think, that you need to setup k8 egress, still it really depends from your k8s config
Hey @vsukhin here's the output of the commands and responses:
`C:\Users\DamienAbrahams>kubectl testkube create test --git-uri https://dev.azure.com/vladislavsukhin/Test/_git/Test --git-branch master --git-path Service/Test/TODO.postman_collection.json --git-username vladislavsukhin --git-token ae3snb5u5pcjttymk34yyvd6xq3ie7uuo62g2mvxbjzse3ql2alq --name blue-flowers --type postman/collection --test-content-type git-file
████████ ███████ ███████ ████████ ██ ██ ██ ██ ██████ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████ ███████ ██ █████ ██ ██ ██████ █████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ███████ ███████ ██ ██ ██ ██████ ██████ ███████ /tɛst kjub/ by Kubeshop
Test created testkube / blue-flowers 🥇
C:\Users\DamienAbrahams>kubectl testkube get tests
NAME | TYPE | CREATED | LABELS | SCHEDULE | STATUS | EXECUTION ID
+----------------+--------------------+-------------------------------+--------+----------+--------+--------------------------+
black-flowers | postman/collection | 2022-05-17 04:06:56 +0000 UTC | | | failed | 62831f67dbcef024925039ff
blue-flowers | postman/collection | 2022-05-23 05:27:54 +0000 UTC | | | |
C:\Users\DamienAbrahams>kubectl testkube run test blue-flowers
████████ ███████ ███████ ████████ ██ ██ ██ ██ ██████ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████ ███████ ██ █████ ██ ██ ██████ █████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ███████ ███████ ██ ██ ██ ██████ ██████ ███████ /tɛst kjub/ by Kubeshop
Type : postman/collection Name : blue-flowers Execution ID : 628b1b82dbcef02492503a21 Execution name: secondly-relieved-arachnid
Test execution started
Watch test execution until complete: $ kubectl testkube watch execution 628b1b82dbcef02492503a21
Use following command to get test execution details: $ kubectl testkube get execution 628b1b82dbcef02492503a21
C:\Users\DamienAbrahams>kubectl get pods -n testkube NAME READY STATUS RESTARTS AGE 628b1b82dbcef02492503a21-568w5 0/1 Init:0/1 0 8s testkube-api-server-664766b7bb-sz9tv 1/1 Running 1 (6d3h ago) 16d testkube-dashboard-7f6f6dc6f-5k6sr 1/1 Running 1 (6d3h ago) 16d testkube-minio-testkube-79d99f9746-khl9c 1/1 Running 1 (6d3h ago) 16d testkube-mongodb-5d766b7d58-h79r5 1/1 Running 1 (6d3h ago) 16d testkube-operator-controller-manager-78bd65f589-cfk9z 2/2 Running 20 (111m ago) 16d
C:\Users\DamienAbrahams>kubectl describe pod 628b1b82dbcef02492503a21-568w5 -n testkube
Name: 628b1b82dbcef02492503a21-568w5
Namespace: testkube
Priority: 0
Node: worker1/10.3.11.11
Start Time: Mon, 23 May 2022 13:28:36 +0800
Labels: controller-uid=d5e9c797-c617-47e8-a469-549a6e4790e0
job-name=628b1b82dbcef02492503a21
Annotations: cni.projectcalico.org/containerID: 0b0d1dd5406cb3dbbadccb2c258a432265ab19ad5d509510f36056dad0b94f63
cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
Status: Failed
IP: 10.244.235.159
IPs:
IP: 10.244.235.159
Controlled By: Job/628b1b82dbcef02492503a21
Init Containers:
628b1b82dbcef02492503a21-init:
Container ID: containerd://d151bbad105d2ab84117fae6638429d7908416a7ed03835e8e2397a82a1ebbc4
Image: kubeshop/testkube-executor-init:0.7.8
Image ID: docker.io/kubeshop/testkube-executor-init@sha256:0233a7a9b6a27c9fa4df042e33ba76275b56dc388cb3a3b7c6aa7bb4f02d7667
Port:
Normal Scheduled 32s default-scheduler Successfully assigned testkube/628b1b82dbcef02492503a21-568w5 to worker1 Normal Pulling 31s kubelet Pulling image "kubeshop/testkube-executor-init:0.7.8" Normal Pulled 29s kubelet Successfully pulled image "kubeshop/testkube-executor-init:0.7.8" in 2.644335553s Normal Created 28s kubelet Created container 628b1b82dbcef02492503a21-init Normal Started 28s kubelet Started container 628b1b82dbcef02492503a21-init Normal Pulling 17s kubelet Pulling image "kubeshop/testkube-postman-executor:0.7.8" Normal Pulled 14s kubelet Successfully pulled image "kubeshop/testkube-postman-executor:0.7.8" in 2.651650952s Normal Created 14s kubelet Created container 628b1b82dbcef02492503a21 Normal Started 14s kubelet Started container 628b1b82dbcef02492503a21
C:\Users\DamienAbrahams>kubectl logs 628b1b82dbcef02492503a21-568w5 -n testkube {"type":"event","content":"running test [628b1b82dbcef02492503a21]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nfatal: could not read Username for 'https://dev.azure.com': No such device or address\n"}`
Thank you, @damien-m-abrahams !
Yeah, after I posted I dug around a bit more and had a suspicion it was the git setup.
{"type":"event","content":"running test [628643ab6fd2e48c9d3d2428]"} {"type":"error","content":"process error: exit status 128\noutput: Cloning into 'repo'...\nremote: The `praetorian-inc' organization has enabled or enforced SAML SSO. To access\nremote: this repository, visit https://github.com/orgs/praetorian-inc/sso?authorization_request=AVPYRPFLCVNZLJUCWDLTBKTCQZI35A5PN5ZGOYLONF5GC5DJN5XF62LEZYAHZOG3VVRXEZLEMVXHI2LBNRPWSZGOGQG2KRFPMNZGKZDFNZ2GSYLML52HS4DFVNHWC5LUNBAWGY3FONZQ\nremote: and try your request again.\nfatal: unable to access ' https://github.com/praetorian-inc/chariot-cypress.git/': The requested URL returned error: 403\n"}
It would probably be good to have maybe a bit more of the error handling on the test run to point he user to auth issues rather than the general pod failure message, "contacontainer "628643ab6fd2e48c9d3d2428" in pod "628643ab6fd2e48c9d3d2428-nqz5r" is waiting to start: PodInitializinginer "628643ab6fd2e48c9d3d2428" in pod "628643ab6fd2e48c9d3d2428-nqz5r" is waiting to start: PodInitializing", assuming you can trap that.
On Thu, May 19, 2022 at 5:10 AM Vladislav Sukhin @.***> wrote:
hey @bode-praetorian https://github.com/bode-praetorian. Yes, from the logs it looks like an error of checking out your private git repo. A bit too hard to say, what is an exact reason, but sometimes Kubernetes emulators like Minikube, Kind, K3D have issues with dns solving for external websites or https cert validation. Can we try to check the logs of init container kubectl logs 62853cf96fd2e48c9d3d241e-mzfd7 62853cf96fd2e48c9d3d241e-init -n testkube?
— Reply to this email directly, view it on GitHub https://github.com/kubeshop/testkube/issues/1502#issuecomment-1131555120, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVPYRPAZCIMZ3JBNYSSDRFDVKYOSRANCNFSM5VULVHKA . You are receiving this because you were mentioned.Message ID: @.***>
Thank you, @bode-praetorian. Good point, we have an issue about improving init pod logs #2396
I've added a Postman collection test to an Azure DevOps repository and created a test in TestKube. When tying to run the test the following error occurs:
I've tried to create the test with and without a token and the same error occurs. I'm not sure whether I've missed something in terms of creating the test in TestKube or my K8 environment is the problem?