strangelove-ventures / interchaintest

e2e testing framework for the interchain
https://interchaintest-docs.vercel.app
Apache License 2.0
195 stars 125 forks source link

feat: kubernetes integration (w/ kind) / local-ic dind #1075

Open Reecepbcups opened 7 months ago

Reecepbcups commented 7 months ago

I think having the cosmos-operator as an option for local-interchain persistent testnet deployments is where it would be most useful in the near term. Mark's idea of using kind could get us to a place where we have feature parity with existing local-interchain and then giving users the ability to point to an external kube cluster would allow testing of more real-world conditions

https://kind.sigs.k8s.io/ is the reference docs

By doing kube in docker, users get the same experience as now without the k8 dep. Keeping UX the same machine side

Reecepbcups commented 3 months ago

local-interchain integration from Mateusz Mielewczyk in the cosmoshub discord :D

apiVersion: batch/v1
kind: Job
metadata:
  name: cosmos-e2e-test-local-ic
spec:
  ttlSecondsAfterFinished: 60
  backoffLimit: 0
  template:
    spec:
      # Init containers to have a docker deamon and local-ic side by side to e2 tests
      initContainers:
      - name: dind
        image: docker:19.03.13-dind
        command: ["dockerd-entrypoint.sh"]
        env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
        securityContext:
          privileged: true
        volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        restartPolicy: Always
      - name: local-ic
      # This image is a custom image that has local-ic and golang, base image is docker:19.03.13
        image: example_cosmos_e2e:1
        command: [sh, -c]
        args: ["until docker ps; do sleep 3; done; /root/go/bin/local-ic start cosmoshub.json"]
        restartPolicy: Always
        env:
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        # Configmap with cosmoshub.json configs
        - name: cosmoshub-config-volume
          mountPath: /interchaintest/local-interchain/chains
      # As init contianers do not have readiness probe, we need to wait for them to be ready
      - name: wait-for-cosmos
        image: busybox
        command: ['sh', '-c', 'until wget -q --spider http://localhost:26657; do echo waiting for 26657; sleep 3; done; echo Cosmos is up']
      containers:
      # Main container running tests
        - name: e2e-tests
          image: e2e_example_service:1
          command: [sh, -c]
          args: ["npm run test:e2e"]
          imagePullPolicy: Always
          startupProbe:
            httpGet:
              path: /health
              port: 26657
            failureThreshold: 30
            periodSeconds: 3
      restartPolicy: Never
      volumes:
      - name: docker-graph-storage
        emptyDir: {}
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock
      - name: cosmoshub-config-volume
        configMap:
          name: e2e-cosmoshub-config