⭐️ K8 modularize: deploy microservices in different namespace

[rivernews]

Currently we are using one namespace for all microservices (specified a fixed namespace in our custom terraform module). It would be great if we can separate them into different microservices.

The challenge is we are using a single wildcard certificate that intends to serve all ingress of every microservices:

ClusterIssuer -->(cert-manager)-->(Ing annotation)--> Certificate == Ingress == Service
!! Certificate & ingress should be in same namespace (at least one cert per ingress)
!! Ingress & Service should be in same namespace

If we are using different namespace for each service, we will end up having several certificates, even if we only have one ClusterIssuer, and should only need one wildcard certificate that covers all domains services will deploy to.

Multiple certificate might work as well, it's just a concern if this will cause unnecessary api calls to letsencrypt, since letsencrypt has a pretty strict quota limit. Needs further investigation into this.

Relevant posts

• Some suggest manually create the certificate. But then how to let cert-manager auto-manage expiration and renewal?
• Some suggest adding controller.extraArgs.default-ssl-certificate (also here) on helm_release ing controller. Indeed it's a valid parameter for kubernetes nginx ingress. Have to figure out how it work, could be a good start. Be aware that it may only work for hosts that do not match any ing rule, so might not work for our case, where we want to use the certificate for all ing hosts.
  • This issue also mentioned the same config default-ssl-certificate.
  • The K8 nginx ingress doc has a section to explain default-ssl-certificate flag.

[rivernews]

This is indeed a common problem. Wildcard certificate, different namespace. How?

https://itnext.io/using-wildcard-certificates-with-cert-manager-in-kubernetes-and-replicating-across-all-namespaces-5ed1ea30bb93

https://github.com/jetstack/cert-manager/issues/273

https://www.techdiction.com/2018/06/17/configuring-kubernetes-ingress-with-a-wildcard-dns-certificate-single-tls-secret-and-applications-in-multiple-namespaces/

[rivernews]

This isn’t probably a good start: https://scriptthe.net/2019/06/21/utilize-a-default-certificate-for-services-fronted-by-nginx-ingress/

1. Set default ssl certificate in extra args in Nginx controller. Also for the cert secret specified, you have to use the one w/ tls.crt and tls.key, which is the letsencrypt prod or staging itself, not the one created by cert-manager.Like in this example.
2. Ingress resources: remove annotations, remove secret name. So just the tls block with hosts.

[rivernews]

Implementation

• [x] Migrate appl tracky

  • Debug the current issue
  • OK, it's because of db connection issue. (some hanging migration connection)

• [x] Migrate slack

  • Change the conditions like in postgres

• [x] Migrate postgres cluster to its own namespace

  • You'll have to change the condition in installation module: for kubernetes_ingress and kubernetes_service_account.
  • You also have to change secret - the postgres sql host, the namespace part to reflect its local dns change.
  • This may involve destroying the entire volume since it's cross-namespace
  • In any case you have to wipe out data. do this:
    • In postgres you can: psql -U admin default_database and drop database <name>; then create database <name>.
    • To load data into each app, follow this link

• [x] Migrate redis cluster to its own namespace

  • Change the conditions like in postgres

• [x] Can we remove the iriversland2 block in ingress controller? To do a complete de-coupling.

• [x] Can we change microservice module's ingress tf name from project-* to app?

• [x] Lastly, we want to de-couple the installation tf module out of this repo. But let's make sure the above is done first.

[rivernews]

De-coupling the installation module

How should we approach this?

• Research: how to reference a module from url or git repository in terraform?
• We should now copy the installation module to a new repo. But don't delete the module in this K8 repo yet! So you have parallel two.

Imagine what the result should be like

• This K8 repository
  • Should only contain resources that are global, or across the system. Of course, Kafka and ES can be de-coupled too. However, ES & Kafka has a lot of stuff not using the microservice module, so in this case it cannot be decoupled. True, because it should be a shared resources across different project. But the real problem is that Kafka and ES references too many other resources, making the de-coupling a bit hard to do.
• The installation module repo
  • It'll really just be the terraform code, at least for now.
• Each microservice project repo
  • This will be a "real application example" for the installation microservice module.
  • Basically the microservice repo still has to scaffold a tf project, but referencing the installation module. Then it may require some credential. We'll try to minimize the number of credential required. Hopefully what we all need is just DO and AWS credentials. (AWS credential should give us all the credential, actually, thanks to parameter store.)

---

• [x] Implement the docker, digitalocean and K8 credential fetching logic in microservice module. We can largely refer to this k8 repo
• [x] Initiate a terraform project in one microservice repo, source the microservice module, and try to run the terraform in microservice repo. Can we spin up terraform? If we can, it's trivial to integrate to that microservice repo's travis or CI/CD pipeline.
• [x] Integrate the terraform logic into CI/CD pipeline. Then test it out!

[rivernews]

CI/CD through terraform

• [x] iriversland
• [x] appl tracky
• [x] slack
• [x] postgres
• [x] redis
• [x] code base clean up: k8

[rivernews]

Resolved K8 unauthorized error, see code comment in k8 provider block. Deployed module caller in each microservices repository.