mit-dci / opencbdc-tx

A transaction processor for a hypothetical, general-purpose, central bank digital currency
Other
895 stars 200 forks source link

Errors in running "E2E Testing with Kubernetes" #194

Open Cbdc2022 opened 1 year ago

Cbdc2022 commented 1 year ago

Hi, I tried to run "E2E Testing with Kubernetes" , but all test cases failed with the following error: Events: Type Reason Age From Message


Normal Scheduled 7m17s default-scheduler Successfully assigned opencbdc-tx-qg7ep1/shard-0-1-0 to opencbdc Warning Failed 5m17s (x12 over 7m15s) kubelet Error: ErrImageNeverPull Warning ErrImageNeverPull 2m6s (x27 over 7m15s) kubelet Container image "opencbdc-tx:latest" is not present with pull policy of Never

Not sure where to get the correct Image, would you please help when have time. thank you.

File "testrun.log" attached. testrun.log Yes, I have run the ./scripts/build-docker.sh

FYI: OS: ubuntu 22.04

tried to run:

E2E Testing with Kubernetes Requirements Go (go test library used to run tests) Minikube Helm Kubectl Running tests ./scripts/test-e2e-minikube.sh Review results and logs at testruns//

FYI: ./scripts/build-docker.sh Sending build context to Docker daemon 106.7MB Step 1/10 : ARG IMAGE_VERSION="ubuntu:20.04" Step 2/10 : ARG BASE_IMAGE="ghcr.io/mit-dci/opencbdc-tx-base:latest" Step 3/10 : FROM $IMAGE_VERSION AS base 20.04: Pulling from library/ubuntu eaead16dc43b: Pull complete Digest: sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd Status: Downloaded newer image for ubuntu:20.04 ---> 680e5dfb52c7 Step 4/10 : ENV DEBIAN_FRONTEND noninteractive .......

$ docker images -a --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE

f217f881db26 20 minutes ago 119MB 5c8c1273d9d3 20 minutes ago 94.7MB opencbdc-tx-twophase latest b08cde7306d1 20 minutes ago 119MB 87c3318f7a11 20 minutes ago 117MB d10a8be8a553 20 minutes ago 116MB f5bef2400852 20 minutes ago 117MB 92f60702b416 20 minutes ago 1.81GB 202277689b37 20 minutes ago 73.6MB 94790781d5dc 20 minutes ago 72.8MB 6b14885d4292 21 minutes ago 1.64GB opencbdc-tx-base latest 676b02da7baa 21 minutes ago 1.53GB 18a322dc3633 23 minutes ago 72.8MB 65fb0c173c7f 23 minutes ago 72.8MB ac60d424e32b 23 minutes ago 72.8MB 1932112415d7 23 minutes ago 72.8MB fea9d7cc6b92 23 minutes ago 72.8MB 1b2a63db5764 23 minutes ago 72.8MB ubuntu 20.04 sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd 680e5dfb52c7 8 days ago 72.8MB [ghcr.io/mit-dci/opencbdc-tx-base](http://ghcr.io/mit-dci/opencbdc-tx-base) latest sha256:2076bdd505b9a003acebc733c1981ec4ef1ed3d3f2cdea1ba8abb3e1d2250720 49fc6337a4eb 13 days ago 1.54GB [ghcr.io/mit-dci/opencbdc-tx-base](http://ghcr.io/mit-dci/opencbdc-tx-base) sha-16bd61a sha256:2076bdd505b9a003acebc733c1981ec4ef1ed3d3f2cdea1ba8abb3e1d2250720 49fc6337a4eb 13 days ago 1.54GB opencbdc-tx latest 929938b6244b 2 weeks ago 2.88GB [gcr.io/k8s-minikube/kicbase](http://gcr.io/k8s-minikube/kicbase) v0.0.35 sha256:e6f9b2700841634f3b94907f9cfa6785ca4409ef8e428a0322c1781e809d311b 7fb60d0ea30e 4 weeks ago 1.12GB
HalosGhost commented 1 year ago

CC @kylecrawshaw

Cbdc2022 commented 1 year ago

FYI: OS: ubuntu 22.04 running on virtual machine - VMware® Workstation 16 Pro (16.2.4 build-20089737)

$ docker version Client: Docker Engine - Community Version: 20.10.21 API version: 1.41 Go version: go1.18.7 Git commit: baeda1f Built: Tue Oct 25 18:01:58 2022 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.21 API version: 1.41 (minimum version 1.12) Go version: go1.18.7 Git commit: 3056208 Built: Tue Oct 25 17:59:49 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.9 GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0


~$ minikube version minikube version: v1.27.1 commit: fe869b5d4da11ba318eb84a3ac00f336411de7ba


$ helm version version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}


$ kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.25.3 Kustomize Version: v4.5.7 Server Version: v1.25.2


~$ go version go version go1.18.1 linux/amd64

Cbdc2022 commented 1 year ago

It works after the following change:

  1. values.yaml line 53: repository: ghcr.io/mit-dci/opencbdc-tx --> repositoryVal: opencbdc-tx-twophase
  2. client.yaml line 11: $.Values.image.repository --> $.Values.image.repositoryVal
  3. 2pc-component.yaml line 151, 193: $.Values.image.repository --> $.Values.image.repositoryVal

The problem is using variable name "repository" will cause issue, maybe "repository" is a keyword or something. I need to change "repository" to "repositoryVal" ( "repository2" also works)

Cbdc2022 commented 1 year ago

must run ./scripts/build-docker.sh first to create local docker images , then run /scripts/test-e2e-minikube.sh

HalosGhost commented 1 year ago

Interesting. Needing to run ./scripts/build-docker.sh is known and required before running the E2E test suite, but I'm surprised that repository is causing an issue for you in the yaml files (since it's worked for us several times now). Can you tell me more about the setup you have so we can figure out what might be causing that?

Cbdc2022 commented 1 year ago

sorry for the delay response. not sure what is special on my end. I suspect it is helm. I am using helm with Version:"v3.7.0". when using variable name "repository", kube pod created for shard (and other components) is looking for image/tag ="opencbdc-tx:latest". However, the image/tag created by "./scripts/build-docker.sh " is "opencbdc-tx-twophase:latest". Basically, line "repository: opencbdc-tx-twophase" in 'values.yaml" will always create kube pod with image/tag ="opencbdc-tx:latest" regardless what value I assign to variable "repository".

FYI: trader@trader-virtual-machine:~/Documents/code/opencbdc/opencbdc-tx-trunk$ kubectl describe pod -n opencbdc-tx-lt1ptx shard-0-0-0 Name: shard-0-0-0 Namespace: opencbdc-tx-lt1ptx Priority: 0 Service Account: opencbdc-tx-updox9-opencbdc-2pc Node: opencbdc/192.168.49.2 Start Time: Thu, 05 Jan 2023 22:35:38 -0500 Labels: app.kubernetes.io/instance=opencbdc-tx-updox9 app.kubernetes.io/name=shard controller-revision-hash=shard-0-0-88c8d7f46 opencbdc.hmltn.io/clusterId=0 opencbdc.hmltn.io/id=0 opencbdc.hmltn.io/role=shard statefulset.kubernetes.io/pod-name=shard-0-0-0 Annotations: Status: Pending IP: 172.17.0.36 IPs: IP: 172.17.0.36 Controlled By: StatefulSet/shard-0-0 Containers: shard: Container ID:
Image: opencbdc-tx:latest Image ID:
Ports: 6666/TCP, 6667/TCP, 6767/TCP


trader@trader-virtual-machine:~/Documents/code/opencbdc/opencbdc-tx-trunk$ docker image ls -a REPOSITORY TAG IMAGE ID CREATED SIZE

76fa079d53b9 30 minutes ago 119MB **opencbdc-tx-twophase latest** 900b435e3bdc 30 minutes ago 119MB 212e859ccdc1 30 minutes ago 117MB 4fd10e616171 30 minutes ago 117MB 7885982b0419 30 minutes ago 116MB 57a719dbc368 31 minutes ago 72.8MB 047ca6a55824 31 minutes ago 2.22GB 2d9bc56ee18b 31 minutes ago 73.6MB 76c166be6354 31 minutes ago 94.7MB ac8f926e8b6c 32 minutes ago 2.05GB opencbdc-tx-base latest faf9f9b099b0 32 minutes ago 1.54GB 0bc94695397c 37 minutes ago 72.8MB e8c395469e4b 37 minutes ago 72.8MB 01811d8d9869 37 minutes ago 72.8MB 298053d1dd17 37 minutes ago 72.8MB 8924c488eb7f 37 minutes ago 72.8MB f2089c38c8cd 37 minutes ago 72.8MB ubuntu 20.04 d5447fc01ae6 4 weeks ago 72.8MB gcr.io/k8s-minikube/kicbase v0.0.35 7fb60d0ea30e 3 months ago 1.12GB
HalosGhost commented 1 year ago

As it's working for me, several other people, and especially because it works in the github CI, I'm very hesitant to change this without knowing the actual cause (or at least knowing for sure that changing the name wouldn't break something else).