Closed mogul closed 2 years ago
(I don't know why this closed.)
@nickumia-reisys,
I did a bunch of refactoring to separate the provision
module into provision-aws
and provision-k8s
. Currently:
docker-compose --env-file=.env.${ENV_NAME}.secrets run --rm terraform apply -target='module.brokerpak-eks-terraform.module.provision-aws' -var-file=terraform.${ENV_NAME}.tfvars
refreshes cleanly, with no complaints about the kubernetes provider. It shows 93 resources to add.docker-compose --env-file=.env.${ENV_NAME}.secrets run --rm terraform apply -target='module.brokerpak-eks-terraform.module.provision-k8s' -var-file=terraform.${ENV_NAME}.tfvars
refreshes cleanly, with no complaints about the kubernetes provider. It shows 51 resources to add (just the subset of provision-aws
that it depends on is included).You might try pushing this to the develop
branch to see if it deploys cleanly in development-ssb CF space / development TF workspace, and investigate if not.
However, we still have a ton of stuff provisioned in the production workspace under the former module, module.brokerpak-eks-terraform.module.provision
, and it won't destroy
without the kubernetes provider configured; it still complains about wanting to connect to 127.0.0.1. I don't know what to do about that... I guess we still just have to manually remove it...? It feels like there must be some other way, but I can't think of it. 😞
Still to do:
eks-service-definition.yml
to deal with all these changes; I think we will have to carefully factor the outputs and inputs between the modules out, then leave the files containing them out of the service definition...? I dunno. I'm choosing to focus on the static deployment from datagov-ssb first.Note Hashicorp now has a support page about this situation.
Also I think maybe applying without refreshing may get us out of our current bind... Worth a shot!
Relating to the apply without refresh... No dice 😞
However, we still have a ton of stuff provisioned in the production workspace under the former module,
module.brokerpak-eks-terraform.module.provision
, and it won'tdestroy
without the kubernetes provider configured; it still complains about wanting to connect to 127.0.0.1. I don't know what to do about that... I guess we still just have to manually remove it...? It feels like there must be some other way, but I can't think of it. 😞
After banging my head against this for yet another day, I am almost out of things to try... This is the next thing:
kubectl -n kube-system delete configmap aws-auth
, also then also remove it from the terraform state (with terraform state rm
), and then apply to see if that resolves the problem.Other than that, I think we're down to doing this in all three of development/staging/default
workspaces and corresponding ENV_NAME
:
terraform state rm
everything with the prefix module.brokerpak-eks-terraform
(so that Terraform forgets everything it knows about those resources)@mogul I started fixing this, but got stuck because I wasn't sure how to fix the problem of the provision-aws
outputs being inputs to the provision-k8s
module. Can you have more than one provision block and then pass from provision-1 to provision-2 and then to bind?
The above question is only about how to declare the service-definition.yml. If both provision-aws
and provision-k8s
inputs are put as inputs, it expects that terraform will receive all of inputs at once which isn't true with the double-module design. If we leave out the provision-aws
outputs, then provision-k8s
wouldn't know about them as inputs for itself. 😕
Hmm.. so I got the brokerpak to build by using only the outputs from provision-k8s
and only the inputs from provision-aws
. I'm testing to see if terraform will like it.. but I guess this works 😖
This is what I was afraid of.. Terraform is complaining that variables aren't declared as inputs because the brokerpak didn't pass it in as an input variable. But the brokerpak doesn't have access to those inputs yet because provision part 1 was not run yet,
nickumia@DL62-2-2MDD043:~/temp/datagov-brokerpak-eks$ make test
Provisioning aws-eks-service:raw:instance-nickumia
in progress | (0 seconds)...failed!
Error: Reference to undeclared input variable on admin-account.tf line 53, in data "template_file" "admin_kubeconfig": 53: certificate-authority-data: ${var.certificate_authority_data}An input variable with the name "certificate_authority_data" has not beendeclared. This variable can be declared with a variable"certificate_authority_data" {} block.Error: Reference to undeclared input variable on admin-account.tf line 54, in data "template_file" "admin_kubeconfig": 54: server: ${var.server}An input variable with the name "server" has not been declared. This variablecan be declared with a variable "server" {} block.Error: Reference to undeclared input variable on external-dns-k8s.tf line 9, in resource "kubernetes_service_account" "external_dns": 9: "eks.amazonaws.com/role-arn" = var.zone_role_arnAn input variable with the name "zone_role_arn" has not been declared. Thisvariable can be declared with a variable "zone_role_arn" {} block.Error: Reference to undeclared input variable on persistent-storage-k8s.tf line 7, in resource "kubernetes_storage_class" "ebs-sc": 7: kmsKeyId = var.persistent_storage_key_idAn input variable with the name "persistent_storage_key_id" has not beendeclared. This variable can be declared with a variable"persistent_storage_key_id" {} block. exit status 1
Yeah I'm not sure how to deal with this yet.
I did some initial refactoring in the commits above. It's a little messy to symlink when you're working locally, but I think this approach may work since it approximates what we will do in the template_refs
paths in the service definition to have everything work as before.
Not yet working, though... I still haven't got the kubernetes provider configured correctly, so it's still trying to talk to 127.0.0.1:80
.
OK, this is pretty much back to where it was, as far as https://github.com/gsa/data.gov/issues/3706. All that remains is properly waiting for DNS to be resolvable before proceeding with tests.
Fixed the DNS resolution waiting-loop in test.sh
. This would be passing if we hadn't somehow dropped the DS records from the data.gov domain when we moved the data.gov zone to Route53...! I will make a pull-request on 18f/dns
in the morning to fix that.
Now blocked waiting on https://github.com/18F/dns/pull/593
That was quick! Now just waiting on the test to pass.
Unless DNSSEC takes a long time to setup too based on DNS propagation, the test failed for no apparent reason.. Unless the dns server being requested to do dnssec validation was just a bad one 😕
See https://github.com/kubernetes-sigs/external-dns/pull/422
This also ensures that
prerequisite_binaries_present
works even if the binaries are available via aliases.