Init containers migration failing

[mar1n3r0]

I have locked migrate/migrate to v4.7.1 and managed to get the specific errors from the init containers:

error: open /migrations/auth: no such file or directory
error: open /migrations/user: no such file or directory

The migrations directory is not copied to the final image. Expected migration files for example auth service https://github.com/vardius/go-api-boilerplate/blob/20fe02936f82db7d723f33ebfbc69092dc42226d/helm/app/values.yaml#L184 Copied files are: https://github.com/vardius/go-api-boilerplate/blob/20fe02936f82db7d723f33ebfbc69092dc42226d/cmd/auth/Dockerfile#L38 Doing so should solve the issue, with migrations

FROM scratch
COPY --from=buildenv /go/bin/app /go/bin/app
+ COPY --from=buildenv /app/migrations/"$BIN" /migrations/"$BIN"
ENTRYPOINT ["/go/bin/app"]

This needs to be done for both auth and user service.

Originally posted by @vardius in https://github.com/vardius/go-api-boilerplate/issues/22#issuecomment-567684300

[vardius]

@mar1n3r0 do you want to create a pull request with a fix for it ? ofc if it works, i havent tested it yet.

[mar1n3r0]

@vardius same error still after rebuilding the images.

[vardius]

Ok, I will have a look tmr morning, now going to sleep. Let me know which parts you want to create pull requests for, so our work does not duplicates :)

[mar1n3r0]

Sure thing basically once this is resolved I will open a PR with all the changes I needed to make it work on Kubernetes v1.14.10 thus far so we can adjust the docs accordingly.

Edit: Tried moving the migrations directory directly in the migrate container and revert back to migrate:latest which weirdly caused Init:ImagePullBackOff even though it exists. unable to pull local image After running eval $(minikube docker-env) and tagging the local images with a tag local before building the auth migrations succeeded. The user one is still failing with: error: no migration found for version 1562371201error: file does not exist

[vardius]

I did some fixes at this pr https://github.com/vardius/go-api-boilerplate/pull/25. Please verify if it matches your changes.

[vardius]

I did some fixes at this pr https://github.com/vardius/go-api-boilerplate/pull/25. Please verify if it matches your changes.

My dev env versions

➜  go-api-boilerplate git:(hotfix/kubernetes-1.16) docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

➜  go-api-boilerplate git:(hotfix/kubernetes-1.16) kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:11:03Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Testing flow

The way I test is maybe not the fastest and optimal but the steps I follow are:

1. Build local images:

   make docker-build BIN=auth
   make docker-build BIN=migrate
   make docker-build BIN=user

2. Install Assuming you already have cert manager up to date https://github.com/vardius/go-api-boilerplate#install-cert-manager

   make helm-install

Then if I encounter errors i just delete whole cluster with

make helm-delete

And repeat all steps after doing some changes

[mar1n3r0]

Yep the testing flow is the same. I have opened a PR: #26.

[mar1n3r0]

@vardius The problem with user service remains though: no migration found for version 1562371201error: file does not exist. The strange thing is the version number is for the auth service but produced in the user init so I am not sure what's going on. Edit: Now I observe the opposite behavior where the user init migrations went fine but the auth init reports a wrong version: error: no migration found for version 1562371202error: file does not exist The kubernetes-dashboard service also fails to start with the following message: 2019/12/21 14:14:56 Storing encryption key in a secret panic: secrets is forbidden: User "system:serviceaccount:go-api-boilerplate:kubernetes-dashboard" cannot create resource "secrets" in API group "" in the namespace "kube-system"

[vardius]

https://github.com/vardius/go-api-boilerplate/pull/26#discussion_r360673069

Add kubernetes-dashboard we can disable it for now and w8 for official release

[vardius]

The problem with user service remains though: no migration found for version 1562371201error: file does not exist. The strange thing is the version number is for the auth service but produced in the user init so I am not sure what's going on. Edit: Now I observe the opposite behavior where the user init migrations went fine but the auth init reports a wrong version: error: no migration found for version 1562371202error: file does not exist

This is because migration version conflicts, both of them want to run migrations and if user service runs 1562371202 first then auth fails as current version is greater then 1562371201

@mar1n3r0 something like this would be perfect

• one (same) migration service for use as init container
• decoupled migration files (each service has its own)

Example setup

I think it should work with your change here https://github.com/vardius/go-api-boilerplate/blob/32f363c01b6b068b9d09a4bc18bf2c08238680de/helm/microservice/templates/deployment.yaml#L64L69

[vardius]

Fixed: https://github.com/vardius/go-api-boilerplate/pull/26 https://github.com/vardius/go-api-boilerplate/pull/27