Closed armenr closed 6 years ago
I do the same thing when referencing the module I'm testing from the fixture path. Mine looks like this: source = "../../.."
- so the only difference I see is an extra dot at the beginning. Maybe try that?
Oh, crap. I hope that's not it, because that'd be stupid.
...aaaaaaaand that wasn't it.
Changed my main.tf to:
module "test_server" {
source = "../../../modules/aws_ec2"
environment = "${var.environment}"
total_instances = "${var.testbed_instance_count}"
instance_name = "test"
instance_type = "t2.medium"
key_pair_id = "redacted-key-name"
enable_instance_protection = true
aws_region = "${var.aws_region}"
aws_route53_zone_id = "${var.r53_zone_id}"
dns_zone = "${var.r53_zone_name}"
security_group_ids = ["sg-111111"]
subnet_ids = ["${module.testbed_vpc.public_subnets}"]
}
Didn't make a difference. The only thing that seems to resolve properly is if I hardcode the FULL path. Relative paths don't seem to work, at all.
This is mostly okay with me, since I've already done the Git magic to break my modules out into their own repo (and preserved their commit history...sheesh), but what is problematic is that I may want to work in a local branch, and hack on local copies of my modules - and run them through integration and unit tests before promoting them to master...and it would be nice to be able to do that by referencing my modules through relative paths.
Uncertain what the issue is :-\
I got kitchen-terraform working by passing it the GitHub reference for "source =" for the modules. That's because of Problem Pt. 1
This is actually a bit more problematic than I'd originally thought.
I've developed and have been testing a plugin for using Ansible as a sort of provisioner, and I expose a relative path variable for where the roles and the playbook to use which I have in my main.tf.
The variable declaration looks like this:
variable "playbook_path" {
description = "Relative path to Ansible roles"
default = "../../../../ansible"
}
variable "target_playbook" {`
description = "Relative path reference to the playbook that contains the plays I want to run"
default = "../../../../ansible/vpn_servers.yml"
}
When I try to converge with the relative path passed in, here's what I get:
╰─ kitchen converge
-----> Starting Kitchen (v1.16.0)
Terraform v0.10.8
$$$$$$ Terraform version 0.10.8 is supported
-----> Converging <default-ubuntu>...
Copying configuration from "/Users/armenr/Development/project/project-infra/terraform/environments/project-ops/testbed"...
Upgrading modules...
Get: git::ssh://git@github.com/redacted/project-tf-modules.git (update)
Get: git::ssh://git@github.com/redacted/project-tf-modules.git (update)
Get: git::ssh://git@github.com/redacted/project-tf-modules.git (update)
Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.2.0)...
- Downloading plugin for provider "template" (1.0.0)...
- Downloading plugin for provider "null" (1.0.0)...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 1.2"
* provider.null: version = "~> 1.0"
* provider.template: version = "~> 1.0"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
There are warnings and/or errors related to your configuration. Please
fix these before continuing.
Errors:
* module.test_server_ansible.null_resource.ansiformer: Path not valid: [/Users/armenr/Development/project/project-infra/terraform/environments/project-ops/ansible/vpn_servers.yml]
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Converge failed on instance <default-ubuntu>. Please see .kitchen/logs/default-ubuntu.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
BUT when I pass the FULL path in the variable, converge works, completely, and without error. E.g. - /Users/armenr/Development/project/project-infra/ansible/
I think I could explore a way to work around it by having the ansible module that uses the ansible privisioner plugin that I've been hacking on provide the full path to the playbook and the the roles folder as an output...but this is undesirable and shouldn't be necessary. I wouldn't want to add that work or code.
So, it seems to me, there's an inherent problem with the way kitchen-terraform is inferring relative paths - such that it is actually broken.
I can try to dig through your source code to see where relative path inference/expansion is happening, but I'd really appreciate any help or insights the authors and maintainers might be able to provide.
Thank you!
../../../../../../../ansible/vpn_servers.yml
<--- This abomination works. It breaks my actual provisioning logic and real-life terraform runs, but somehow seems to work for kitchen-terraform specifically.
The actual, accurate, relative path to where the file exists in real life is "../../../../ansible/vpn_servers.yml"
Similarly, when I use this local reference "source = "../../../../../../modules/aws_ec2""
- it manages to pull from my local module (and breaks my actual terraform plan/terraform apply) but the accurate, correct, real path is "source = "../../../modules/aws_ec2"
So, it would seem, with relative paths, it's always 3 directory levels off - To make kitchen-terraform work, I'd have to break my actual terraform runs.
This is very strange, let's figure this out.
Can you help me out with the details, maybe sending the output of a tree
command in the module with the test-kitchen tests would be a good start. I'm looking to see if I can mimic on my machine the same setup.
Is this the module folder you are testing? /Users/armenr/Development/project/project-infra/terraform/modules/aws_ec2
Is this the fixtures directory? /Users/armenr/Development/project/project-infra/terraform/environments/project-ops/testbed
And your trying to bring in this ansible folder: /Users/armenr/Development/project/project-infra/ansible/
Trying to determine if it looks similar to our example module with tests: https://github.com/newcontext-oss/kitchen-terraform/tree/master/examples/aws_provider
Greetings @armenr! There is a driver attribute called directory where test fixtures can be placed (http://www.rubydoc.info/gems/kitchen-terraform/Kitchen/Driver/Terraform)
The value of that attribute effects will effect your converge. Without it being set, you're entire codebase (as opposed to fixture directory) will be copied to .kitchen/kitchen-terraform/default-ubuntu and that is where terraform is running from. So that explains the extra levels of directory you are needing to "escape" out of your module and into other terraform projects.
@armenr: thanks for your interest in the project!
I'm sorry that you're having a frustrating experience.
As @nictrix and @edwardbartholomew have alluded to, I believe the problem that you are seeing is caused by our use of terraform init
with the "copy a source module" behaviour: https://www.terraform.io/docs/commands/init.html#copy-a-source-module. It appears that using that feature does not account for the source module including relative references. The work around that I am aware of is to place the source module under two additional directories relative to the directory in which Test Kitchen in being invoked, like $PWD/test/fixtures/source_module
, as this is copied to $PWD/.kitchen/kitchen-terraform/<suite>-<platform>
.
I am currently working on changes to the workflow which will drop the source module copying in favour of using a Terraform workspace. I believe these changes will solve your problem as no directories will be moved prior to Terraform resolving relative paths.
Wow. The response and care has been overwhelming. You guys are awesome!
I'm going to address responses in order they were posted.
Since I haven't used kitchen in a couple of years, I may be rusty when it comes to this idea of fixtures.
@nictrix - to answer your question:
╰─ pwd && tree
/Users/armenr/Development/project/project-infra
.
├── Gemfile
├── Gemfile.lock
├── README.md
├── ansible
│ ├── roles
│ │ ├── project.mongodb
│ │ │ ├── README.md
│ │ │ ├── default.yml
│ │ │ ├── defaults
│ │ │ │ └── main.yml
│ │ │ ├── files
│ │ │ │ └── mongodb.service
│ │ │ ├── handlers
│ │ │ │ └── main.yml
│ │ │ ├── meta
│ │ │ │ └── main.yml
│ │ │ ├── tasks
│ │ │ │ └── main.yml
│ │ │ ├── templates
│ │ │ ├── tests
│ │ │ │ ├── inventory
│ │ │ │ └── test.yml
│ │ │ └── vars
│ │ │ └── main.yml
│ │ ├── project.ntp
│ │ │ ├── README.md
│ │ │ ├── defaults
│ │ │ │ └── main.yml
│ │ │ ├── files
│ │ │ ├── handlers
│ │ │ │ └── main.yml
│ │ │ ├── meta
│ │ │ │ └── main.yml
│ │ │ ├── tasks
│ │ │ │ └── main.yml
│ │ │ ├── templates
│ │ │ ├── tests
│ │ │ │ ├── inventory
│ │ │ │ └── test.yml
│ │ │ └── vars
│ │ │ └── main.yml
│ │ └── project.pritunl
│ │ ├── README.md
│ │ ├── default.yml
│ │ ├── defaults
│ │ │ └── main.yml
│ │ ├── files
│ │ ├── handlers
│ │ │ └── main.yml
│ │ ├── meta
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ └── main.yml
│ │ ├── templates
│ │ ├── tests
│ │ │ ├── inventory
│ │ │ └── test.yml
│ │ └── vars
│ │ └── main.yml
│ └── vpn_servers.yml
├── possible_policy.json
├── terraform
│ ├── data_sources
│ ├── environments
│ │ ├── README.md
│ │ ├── project-ops
│ │ │ ├── pritunl_vpn
│ │ │ │ ├── main.tf
│ │ │ │ ├── outputs.tf
│ │ │ │ ├── pritunl_instances.tf
│ │ │ │ ├── secrets.tf
│ │ │ │ ├── terraform.tfstate
│ │ │ │ ├── terraform.tfstate.backup
│ │ │ │ ├── variables.tf
│ │ │ │ └── vpc.tf
│ │ │ └── testbed
│ │ │ ├── main.tf
│ │ │ ├── outputs.tf
│ │ │ ├── secrets.tf
│ │ │ ├── terraform.tfstate
│ │ │ ├── terraform.tfstate.backup
│ │ │ ├── test
│ │ │ │ └── integration
│ │ │ │ └── default
│ │ │ │ ├── controls
│ │ │ │ └── inspec.yml
│ │ │ ├── test1_instances.tf
│ │ │ ├── variables.tf
│ │ │ └── vpc.tf
│ │ └── s3.tf
│ └── modules
│ ├── ansible_thingy
│ │ ├── README.md
│ │ ├── ansible_thingy_inventory.tmpl
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── aws_ec2
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── aws_ec2_security_groups
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── aws_route53_record
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── aws_route53_zone
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── aws_security_groups
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ └── aws_vpc
│ ├── README.md
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── terraform.tfstate
└── utils
└── hopes.dreams
48 directories, 77 files
So the way I've been approaching this so far has been that I have a folder inside "project-ops" called testbed. That testbed is just a place for me to play with infra configs safely. It's just a sandbox. It is set up to mimic my already-in-production pritunl/vpn setup.
You could technically break that off into a directory called "dev" that would sit at the same level of nestedness as "project-ops", within the "environments" folder. It probably should go there anyway...
Anyhow, within the testbed directory, I describe an infra similar to my actually deployed VPN infra, and have a .kitchen.yml file there. I thought it would be as easy as firing up kitchen and having it converge, without realizing from the updates that you posted, as well from @edwardbartholomew and @ncs-alane that I may be missing a critical piece which is that driver attribute for fixtures. This is purely ID10T error on my part - not a fault of kitchen-terraform. :)
This is what my .kitchen.yml looks like:
---
driver:
name: terraform
# variable_files:
# - variables.tf
provisioner:
name: terraform
platforms:
- name: ubuntu
transport:
name: ssh
ssh_key: ~/.ssh/ops-integration_testing.pem
verifier:
name: terraform
format: doc
groups:
- name: default
controls:
- operating_system
hostnames: public_dns
username: ubuntu
#Our group's name is default, and we expect to find a control file called operating_system_spec.rb within that group.
suites:
- name: default
So, I guess I might need some guidance to understand how and where to put my fixtures, what they essentially represent, and how that translates into the directory structure and kitchen.yml config I've been setting up.
@edwardbartholomew - Thank you for the reference there! I'm going to dive into the documentation, realizing now that I completely missed that WHOLE aspect of properly getting set up and configured.
@ncs-alane - THAT sounds super useful. I think the logical switch to workspaces is actually a brilliant way to leverage terraform's underlying native way of doing that. As for the workaround you mentioned, do you mean that I need to copy my aws_ec2 module, aws_vpc module, etc to that fixture directory, or just the code for testbed instance module (which is a module built on top of the aws_ec2 module)? The clarification would be a great help, not just in terms of understanding the plumbing better, but also to understand just how much DRY-ness may be sacrificed. I'm happy to make the sacrifice anyhow, since I'd value accurate testing over code repetition - but the repetition would mean that when I've made changes to the actual infra code, I'd also have to always remember to duplicate those changes over to the copies in /fixtures. Or I may have completely confused the hell out of myself and misunderstood you.
The way I see it in my head is that there should be integration tests in the my testbed directory, or any directory where I'm creating full "stacks" of infra (vpc + security groups + ec2's + s3 buckets, etc...) that will converge with kitchen-terraform.
This is now just a follow-up, and now this is me just being greedy because you awesome project contributors and maintainers are paying attention to me -->
As a clearly separated concern, there should also be unit tests for each of my individual modules in the /modules/ directory.
I'm assuming there will need to fixtures/fixture directories for the unit tests in each of the modules folders, as well as fixtures/fixture directories for the integration-oriented tests in each of my deployed "environment" directories?
With the unit tests @ the module level, I understand I have to pass variables and write tests that assess some set of hypothetical sample variables which flatly verify things do what they are supposed to.
In the case of my integration approach in the "environments," my desire is to have kitchen converge with the exact variables and params that will be passed to terraform in real-life. That is to say, is it possible to converge the same vars and configs (intended for prod) in kitchen-terraform without having a separate set of fixtures, or a copy/paste of the exact vars and code from /testbed/ into some nested /fixtures folder?
If my questions indicate ignorance on my part, I apologize.
I am a bit confused as to what that should actually look like, both in terms of organizing things at the filesystem level, and also in terms of what the .kitchen.yml config should look like.
AGAIN - please let me restate my gratitude for everyone's very kind and prompt attention. You guys ROCK!
Hi @armenr thanks for the explanation and information about your working structure. I believe I understand what you're trying to accomplish with the testbed
directory.
The changes I would recommend are:
testbed
directory. - so the .kitchen.yml, Gemfile* should be moved into testbed
driver
stanza: directory: "test/fixtures/project_ops"
test/fixtures/project_ops
in testbed
*.tf
files in testbed
into that directory: test/fixtures/project_ops
testbed
directoryfinal tree output under testbed
directory:
.
├── Gemfile
├── Gemfile.lock
└── test
├── fixtures
│ └── project_ops
│ ├── main.tf
│ ├── outputs.tf
│ ├── secrets.tf
│ ├── test1_instances.tf
│ ├── variables.tf
│ └── vpc.tf
└── integration
└── default
├── controls
└── inspec.yml
Then
# cd into testbed directory
bundle exec kitchen list && \
bundle exec kitchen converge && \
bundle exec kitchen verify
I like what you're doing, testing all the modules put together in your live repo. And as you mentioned you definitely want to start testing each module separately. The setup for those should be similar to how the testbed
directory is now set up.
@nictrix - Thanks for that! Appreciate you taking the time to map that out for me.
So, if I understand correctly, There's no way around code duplication, right? That is to say, if what I want to test (as my fixture) is the actual code I've written for my testbed environment, I'd have to duplicate those .tf files into a fixtures directory to test against?
Ahh I see I thought testbed was just for testing, not something that stayed running for a long period of time. I just built a skeleton module to mimic testbed
and this seems to work: leave the *.tf
files where they are and have this point to it: directory: "./"
My .kitchen.yml:
---
driver:
name: terraform
directory: "./"
provisioner:
name: terraform
platforms:
- name: terraform
transport:
name: ssh
verifier:
name: terraform
format: doc
groups:
- name: default
controls:
- operating_system
suites:
- name: default
So, I actually tried that yesterday after reading your post and exploring the RubyDocs for the Driver API! I'm glad we're thinking along the same lines.
I tested that yesterday in the afternoon - it didn't help. :-\
In /testbed/:
source = "../../../modules/aws_ec2"
environment = "${var.environment}"
total_instances = "${var.testbed_instance_count}"
instance_name = "test"
instance_type = "t2.medium"
key_pair_id = "project-ops"
enable_instance_protection = false
aws_region = "${var.aws_region}"
aws_route53_zone_id = "${var.r53_zone_id}"
dns_zone = "${var.r53_zone_name}"
security_group_ids = ["sg-5f330a22"]
subnet_ids = ["${module.testbed_vpc.public_subnets}"]
}
In .kitchen.yml:
---
driver:
name: terraform
directory: "./"
provisioner:
name: terraform
platforms:
- name: ubuntu
transport:
name: ssh
ssh_key: ~/.ssh/ops-integration_testing.pem
verifier:
name: terraform
format: doc
groups:
- name: default
controls:
- operating_system
hostnames: public_dns
username: ubuntu
#Our group's name is default, and we expect to find a control file called operating_system_spec.rb within that group.
suites:
- name: default
Result of converge:
╰─ converge
-----> Starting Kitchen (v1.16.0)
Terraform v0.10.8
$$$$$$ Terraform version 0.10.8 is supported
-----> Converging <default-ubuntu>...
Copying configuration from "/Users/armenr/Development/project/project-infra/terraform/environments/project-ops/testbed"...
Upgrading modules...
Error downloading modules: Error loading modules: module test_server: invalid source string: ../../../modules/aws_ec2
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Converge failed on instance <default-ubuntu>. Please see .kitchen/logs/default-ubuntu.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
Even if I created a secondary variables file to use for resolving my ansible playbooks, you can't interpolate variables in module "source=" lines, so I can't parameterize that.
I think the only way to make this work (for now) is actually going to have to involve code duplication...unless I've missed something.
The reason I'm insistent is because I'd like to check out a branch of my ops repo, hack on my modules, and test their convergence both at the module's unit-test level, as well as the integration-test level of any systems that are built on top of those modules, but I'd want to do on local copies of my modules and infra.
And the reason I'm persistent/surfacing the issue so verbosely and actively here is not because I think the contributors/developers/maintainers OWE me custom features or anything...It's because I think this is an exceptionally powerful tool for use in safety-critical, security-critical, and regulated/compliance-framed deployments. Examples:
In all of these cases, one must document and show systems compliance. The happy road to "continuous compliance and security" is through config management and infrastructure as code. That's now evolved to "The code you write for your config management and infra should serve as self-documenting specification and proof."
The next logical leap from there would be to say "We can prove compliance because we've translated our security and compliance requirements matrix into tests that we've written and run on our configurations and environments as part of the test-driven process of developing all of our configs and infra as code."
With a tool like kitchen-terraform, the world of testing the FULL compliance spec (in the form of AWSSpec + InSpec tests) as integration tests becomes a very happy and useful reality...but it would be a workable reality when you can virtually simulate a full integration test on the complete, "as-it-would-be-in-production" set of code and variables/parameters without code duplication or differences in the vars you set.
I know it might sound like making a mountain out of a mole-hill, but I've spent my last 5 years in compliance-centric environments, and we've dreamt of ways to achieve this reality. In addition to the work I've been trying to do with your kitchen-terraform testing tool, I've also accidentally discovered a new paradigm for wielding and coupling Ansible and Terraform together...so I'd actually be able to test the infra, the security groups, and all the infrastructure specification in ADDITION to my configuration management spec at the services level, all from one run, from one converge, and from one set of tests.
It's something I've been quietly polishing and testing in the field, and am preparing to release as a set of open source plugins, modules, and an intro blog + detailed documentation.
I'm happy to discuss...even help, but I haven't written Ruby in some time, so I don't know whether you'd want me coming anywhere near your code lol.
@armenr: you are certainly welcome to hack away on the project!
We definitely appreciate users opening issues because they provide us with valuable information that informs the design going forward. Given that usage patterns for Terraform are plentiful, it can be challenging for us to consider and test all possibilities in isolation.
Allow me to reiterate that the next release of kitchen-terraform, featuring workspaces, should solve the problems you are facing with relative paths. I hope to publish the release branch today or tomorrow and I would be very appreciative if you could test your project with that branch to see if there are any additional issues that need to be addressed.
@ncs-alane - You are the MAN! I completely appreciate that different people have different ways of approaching and wielding their tools and their set up, and what you folks have to do, as a team, is make decisions that maximize utility for the greatest number of people, in the least "unique snowflake" kind of way.
I will happily and gladly test the release branch for you! :) I'll keep an eye out to see when it's published.
I've been able to sidestep code duplication and the ability to test branch-specific modules in the following way:
test1_instance.tf looks like this:
# Pritunl standalone server
module "test_server" {
source = "git::git@github.com:project-org/project-tf-modules.git//aws_ec2?ref=armenr/test-tagging"
environment = "${var.environment}"
total_instances = "${var.testbed_instance_count}"
instance_name = "test"
instance_type = "t2.medium"
key_pair_id = "project-ops"
enable_instance_protection = false
aws_region = "${var.aws_region}"
aws_route53_zone_id = "${var.r53_zone_id}"
dns_zone = "${var.r53_zone_name}"
security_group_ids = ["sg-5f330a22"]
subnet_ids = ["${module.testbed_vpc.public_subnets}"]
}
And then I added a test_variables.tfvars:
playbook_path = "/Users/armenr/Development/project/project-infra/ansible/vpn_servers.yml"
pritunl_playbook = "/Users/armenr/Development/project/project-infra/ansible/vpn_servers.yml"
And in my .kitchen.yml, I configured:
driver:
name: terraform
directory: "./"
variable_files:
- test_variables.tfvars
The fix required 2 things:
Referencing my modules from their git repo, by the tag for a given branch I'm working on. This is acceptable because I should be unit-testing modules individually before pushing them up and trying out an integration test on them in a ready-to-deploy stack. So, in terms of development and test-flow, I can live with this. It's a reasonable constraint. It actually necessitates that you test your changes on a per-module basis against fixtures before testing them against existing stacks...and that you should commit the meaningful changes when you think they're ready.
The special testing.tfvars file where I can hard-code the locations of things (other than modules) that are referenced by relative path (like my ansible stuff) to get around the relative path resolution/inference issue. This is suboptimal, but workable, and not an unreasonable addition to the development flow.
...Plus, it's all temporary anyhow because the new branch is going to kick all kinds of ass. :)
@armenr: you can try the new branch now!
You will need to adjust your gemfile accordingly, like:
gem(
"kitchen-terraform",
git: "https://github.com/newcontext-oss/kitchen-terraform.git",
branch: "ncs-alane-split-workflow"
)
Note that there are some changes to the driver configuration as outlined in https://github.com/newcontext-oss/kitchen-terraform/pull/152/commits/63d619a22ee7e96492ad900fb1cbab00b08412da.
Thanks man! :) Going to test drive this now.
@ncs-alane - > Ran into some issues with the converge.
Steps Taken:
---
driver:
name: terraform
provisioner:
name: terraform
platforms:
- name: ubuntu
transport:
name: ssh
ssh_key: ~/.ssh/ops-integration_testing.pem
verifier:
name: terraform
format: doc
groups:
- name: default
controls:
- operating_system
hostnames: public_dns
username: ubuntu
suites:
- name: default
EVERYTHING went off without a hitch, including relative path inference/resolution, so that is excellent!! 👍
BUT, my converge fails with the following message:
Error: Error applying plan:
1 error(s) occurred:
* module.testbed_vpc.aws_vpc.this: 1 error(s) occurred:
* aws_vpc.this: Error creating VPC: VpcLimitExceeded: The maximum number of VPCs has been reached.
status code: 400, request id: REDACTED
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Our account is mostly new, and mostly barren, but the previous person managing our infrastructure left behind some cruft in the form of VPC's that were never used. So, from what I can see, this is just us hitting the default limit of 5 VPC's per region. I've already asked for that to be increased. I didn't realize kitchen, when converging, would try to request an actual new VPC.
Is this always the case? In the previous iteration of the kitchen-terraform gem, I had no such problem when running converge. By switching to workspaces, how are we suddenly hitting this default limit when, if I switch back to the old gem without workspaces, that wasn't the case?
Alrightyroo - This was a terribly misguided assumption on my part, but I will most definitely need a testing.tfvars file. All else, however, is well squared away.
Hooray for looking like a complete and total newb.
@armenr no problem! :laughing: By the way, you can also declare variables as a driver configuration attribute if you prefer that to using the variables file.
EVERYTHING went off without a hitch, including relative path inference/resolution, so that is excellent!! :+1:
I'm glad to read this is the case; this was a fairly significant problem.
Thank you. So far, we're still cruising. Really happy about this - and very grateful!
@ncs-alane - It may be useful to also include something in the readme/documentation about how someone can manually pull an already existing state/terraform-managed infrastructure into kitchen-terraform, non-destructively. :)
@armenr: Can you open a new issue for that point with some more details?
Will do!
Also, I'm a pretty big fan of this idea of passing custom vars through the .kitchen.yml file. The idea that everything you need to see or know about the kitchen convergence is in one file (the .kitchen.yml) is very attractive. I'm glad that you exposed the ability to pass variables directly as k/v pairs, without necessitating that the user must provide a tfvars for testing is wonderful.
@ncs-alane --> Happy to open a separate Issue if you tell me to do so, I did run into this:
variables:
environment: project-testbed-kitchen
testbed_instance_name: project-kitchen
testbed_vpc_cidr: 10.101.0.0/16
testbed_vpc_name: testbed-vpc-kitchen
testbed_vpc_subnets_private: 10.101.1.0/24
testbed_vpc_subnets_public: 10.100.101.0/24, 10.100.102.0/24
$$$$$$ Running command `terraform validate -check-variables=true -var='environment=project-testbed-kitchen' -var='testbed_instance_name=testebed-kitchen' -var='testbed_vpc_cidr=10.101.0.0/16' -var='testbed_vpc_name=testbed-vpc-kitchen' -var='testbed_vpc_subnets_private=10.101.1.0/24' -var='testbed_vpc_subnets_public=10.100.101.0/24, 10.100.102.0/24' /Users/armenr/Development/project/project-infra/terraform/environments/project-ops/testbed`
There are warnings and/or errors related to your configuration. Please
fix these before continuing.
Errors:
* 2 error(s) occurred:
* variable testbed_vpc_subnets_public should be type list, got string
* variable testbed_vpc_subnets_private should be type list, got string
How would I pass a list to a var in the kitchen.yml file?
If that's not yet supported, I can just use a tfvars file instead.
@armenr: I believe I have a solution for you, but please open a separate issue so that this issue can remain focused on a single topic.
Problem Pt. 1
I've got modules I use to create EC2's. Makes for easy templating and repeatability.
I am very, very excited about the fact that we can now use the kitchen testing paradigm to test and verify infrastructure with InSpec. EXCELLENT WORK, FOLKS!! :)
Unfortunately, I'm getting errors on a completely vanilla, first-pass run for converge. From what I can tell, I think it's because I'm using relative paths to reference my modules directories.
Error reads:
Error downloading modules: Error loading modules: module test_server: invalid source string: ./../../../modules/aws_ec2
For the environment in which I want to try running some kitchen-terraform testing + InSpec verification, I've got the following in my main.tf
For now, modules are local to the repo - my next effort after getting testing set up will be to individualize and version-stamp my modules in a repository separate from the actual infrastructure code and reference them via git and release tag. For now, I reference my modules by relative paths:
With the minimal configuration suggested in the "Getting Started" documentation, I have the following in my .kitchen.yml
When I run kitchen converge, I get the following:
When I reference my modules by absolute path, the trouble goes away:
Any help or advice would be greatly appreciated. Thank you!