Step function

[rivernews]

Resolve the infra issues first

• [x] Fix code-server pod - delete old one so new pod can provision while attaching to the same volume
• [x] Delete the entire cluster - seems MEM garbage collection is not working well, having memory pressure.
  • [x] Avoid this in the future - add capacity limit (CPU/MEM) for all microservices
• [x] Use code-server over remote-container-kubernetes
• [x] Microservice module: allow PVC size to be configurable
  • [x] Ways to make sure deployments that enables PVC are only stick to one node (attachment can travel over pod of same replicaset, but cannot transfer across nodes - when you have node pool of >= 2 nodes.
  • [x] This SO says we should use StatefulSet instead of Deployment when having PVC? -- put this in comment TODO.
  • [x] When PVC enabled, update strategy force use Recreate; otherwise RollingUpdate will cause multi-attach and error.
• Last resort is to increase node size to 4CPU/8G, but we should avoid this.

DO enforce using newer K8s.. now cert-manager is not working. Let's create a branch, and use it to update stuff to latest, then try again.

• [x] Need a local environment, don't need to have credentials; just need to git push
• [x] Upgrade helm -> arg has changed, see tf doc: https://registry.terraform.io/providers/hashicorp/helm/latest/docs
• [x] (helm) upgrade cert-manager - see cert-manager doc: https://cert-manager.io/docs/installation/kubernetes/
  • But after all, you may want to see tf solution for cert-manger, may try this: https://github.com/DNXLabs/terraform-aws-eks-cert-manager
  • If the change by upgrade is too much, we need to refactor it (code: https://github.com/rivernews/iriversland2-kubernetes/blob/master/terraform/cert-manager.tf). Mostly: set up route53 validation.

Then, when the node dev environment is ready again

• [x] Plz separate cluster out! Too slow every time waiting to create new
• [x] Error when creating the ClusterIssuer. Terraform outputs INFO: creating issuer... error: unable to recognize "STDIN": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused. Local testing by kubectl apply further reveals error: error validating "test.yaml": error validating data: [ValidationError(ClusterIssuer.spec.acme): unknown field "dns01" in io.cert-manager.v1.ClusterIssuer.spec.acme, ValidationError(ClusterIssuer.spec.acme.solvers[0].dns01): unknown field "selector" in io.cert-manager.v1.ClusterIssuer.spec.acme.solvers.dns01]; if you choose to ignore these errors, turn validation off with --validate=false. We need a working example of ClusterIssuer, using DNS challenge with Route53.
• [x] Unknown error: error: unable to recognize "STDIN": Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
  • Finally - indeed the problem is "localhost" - the command is missing kube config, that's why it defaults to 127.0.0.1 and thus failed.
• [ ] Run terraform apply! To provision skeleton step function.

In the future, to be able to fully use VScode remote-container-k8s...

• [ ] Maybe set up a dedicated machine for it...? Otherwise we'll need to upgrade to 40$ a month... worth it? Of course we can sun set iriversland and appl-tracky api but not in short term.

[rivernews]

Debugging step func creation

Terraform apply error, something in the ASL YAML:

 Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.step_function.aws_sfn_state_machine.this[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
│ produced an invalid new value for .definition: was cty.StringVal("\nStartAt: FirstState\nStates:\n  FirstState:\n    Type: Pass\n    Result: \"FirstStateResult\"\n    End: true\n"),
│ but now cty.StringVal("\nComment: |\n  # AWS State Machine Definition Supports YAML\n  #
│ https://aws.amazon.com/about-aws/whats-new/2021/03/aws-step-functions-adds-tooling-support-for-yaml/\n  # Based on\n  #
│ https://docs.aws.amazon.com/step-functions/latest/dg/development-options.html#development-options-format\n\nStartAt: FirstState\nStates:\n  FirstState:\n    Type: Pass\n    Result:
│ \"FirstStateResult\"\n    End: true\n").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.