search

slalom / dataops-infra

Slalom Infrastructure Catalog for DataOps deployments
https://infra.dataops.tk
MIT License
20 stars 35 forks source link

One-time cleanup: Missing terraform variables and documentation #67

Open aaronsteers opened 4 years ago

aaronsteers commented 4 years ago

Thanks to the new command s-infra check_tf_metadata - we're now able to detect deviations from standard best practices, including: missing module docstrings, missing descriptions on variables, and missing standard input or output variables.

In combination with the new auto-documentation feature, these cleanup steps will drastically improve the quality of Infra Catalog documentation.

Once these are resolved, the next step would be to implement the below check in the CI/CD framework so that new contributors will have clear guidance on the degree to which new modules are meeting code standards.

Running this command:

C:\Files\Source\dataops-infra>python C:\Files\Source\dataops-tools\slalom\dataops\infra.py check_tf_metadata . --recursive

Generates this output:

  1. Blank module headers:

    • [x] ./components/aws/ecs-task/main.tf

  2. Missing required input variables:

    • [x] ./catalog/aws/environment/variables.tf:var.environment
    • [x] ./components/aws/ecr/variables.tf:var.environment
    • [x] ./components/aws/ecr/variables.tf:var.name_prefix
    • [x] ./components/aws/secrets-manager/variables.tf:var.environment
    • [x] ./components/aws/step-functions/variables.tf:var.environment
    • [x] ./components/aws/step-functions/variables.tf:var.name_prefix
    • [x] ./components/aws/vpc/variables.tf:var.environment

  3. Missing required output variables:

    • [ ] ./components/aws/ec2/outputs.tf:output.summary
    • [ ] ./components/aws/ecr/outputs.tf:output.summary
    • [ ] ./components/aws/ecs-cluster/outputs.tf:output.summary
    • [ ] ./components/aws/ecs-task/outputs.tf:output.summary
    • [ ] ./components/aws/lambda-python/outputs.tf:output.summary
    • [ ] ./components/aws/vpc/outputs.tf:output.summary

  4. Missing input variable descriptions:

    • [ ] ./catalog/aws/airflow/variables.tf:var.container_command
    • [ ] ./catalog/aws/airflow/variables.tf:var.container_image
    • [ ] ./catalog/aws/airflow/variables.tf:var.container_num_cores
    • [ ] ./catalog/aws/airflow/variables.tf:var.container_ram_gb
    • [ ] ./catalog/aws/airflow/variables.tf:var.environment_secrets
    • [ ] ./catalog/aws/airflow/variables.tf:var.environment_vars
    • [ ] ./catalog/aws/dbt/variables.tf:var.admin_cidr
    • [ ] ./catalog/aws/dbt/variables.tf:var.container_entrypoint
    • [ ] ./catalog/aws/dbt/variables.tf:var.container_image
    • [ ] ./catalog/aws/dbt/variables.tf:var.container_num_cores
    • [ ] ./catalog/aws/dbt/variables.tf:var.container_ram_gb
    • [ ] ./catalog/aws/dbt/variables.tf:var.dbt_project_git_repo
    • [ ] ./catalog/aws/dbt/variables.tf:var.dbt_run_command
    • [ ] ./catalog/aws/dbt/variables.tf:var.scheduled_timezone
    • [ ] ./catalog/aws/environment/variables.tf:var.aws_region
    • [ ] ./catalog/aws/environment/variables.tf:var.secrets_folder
    • [ ] ./catalog/aws/redshift/variables.tf:var.elastic_ip
    • [ ] ./catalog/aws/redshift/variables.tf:var.jdbc_port
    • [ ] ./catalog/aws/redshift/variables.tf:var.kms_key_id
    • [ ] ./catalog/aws/redshift/variables.tf:var.num_nodes
    • [ ] ./catalog/aws/redshift/variables.tf:var.s3_logging_bucket
    • [ ] ./catalog/aws/redshift/variables.tf:var.s3_logging_path
    • [ ] ./catalog/aws/redshift/variables.tf:var.skip_final_snapshot
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.container_command
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.container_entrypoint
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.container_image
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.container_num_cores
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.container_ram_gb
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.scheduled_timezone
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.source_code_folder
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.source_code_s3_bucket
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.source_code_s3_path
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.taps
    • [ ] ./catalog/aws/singer-taps/variables.tf:var.target
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.admin_cidr
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.default_cidr
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.ec2_instance_storage_gb
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.ec2_instance_type
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.linux_https_domain
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.linux_use_https
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.num_linux_instances
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.num_windows_instances
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.registration_file
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.windows_https_domain
    • [ ] ./catalog/aws/tableau-server/variables.tf:var.windows_use_https
    • [ ] ./components/aws/ec2/variables.tf:var.admin_cidr
    • [ ] ./components/aws/ec2/variables.tf:var.admin_ports
    • [ ] ./components/aws/ec2/variables.tf:var.ami_name_filter
    • [ ] ./components/aws/ec2/variables.tf:var.ami_owner
    • [ ] ./components/aws/ec2/variables.tf:var.default_cidr
    • [ ] ./components/aws/ec2/variables.tf:var.https_domain
    • [ ] ./components/aws/ec2/variables.tf:var.instance_storage_gb
    • [ ] ./components/aws/ec2/variables.tf:var.instance_type
    • [ ] ./components/aws/ec2/variables.tf:var.is_windows
    • [ ] ./components/aws/ec2/variables.tf:var.num_instances
    • [ ] ./components/aws/ec2/variables.tf:var.ssh_key_name
    • [ ] ./components/aws/ec2/variables.tf:var.ssh_private_key_filepath
    • [ ] ./components/aws/ec2/variables.tf:var.use_https
    • [ ] ./components/aws/ecr/variables.tf:var.image_name
    • [ ] ./components/aws/ecr/variables.tf:var.repository_name
    • [ ] ./components/aws/ecs-cluster/variables.tf:var.ec2_instance_count
    • [ ] ./components/aws/ecs-cluster/variables.tf:var.ec2_instance_type
    • [ ] ./components/aws/ecs-task/variables.tf:var.admin_ports
    • [ ] ./components/aws/ecs-task/variables.tf:var.always_on
    • [ ] ./components/aws/ecs-task/variables.tf:var.app_ports
    • [ ] ./components/aws/ecs-task/variables.tf:var.container_command
    • [ ] ./components/aws/ecs-task/variables.tf:var.container_entrypoint
    • [ ] ./components/aws/ecs-task/variables.tf:var.container_name
    • [ ] ./components/aws/ecs-task/variables.tf:var.container_num_cores
    • [ ] ./components/aws/ecs-task/variables.tf:var.container_ram_gb
    • [ ] ./components/aws/ecs-task/variables.tf:var.ecs_cluster_name
    • [ ] ./components/aws/ecs-task/variables.tf:var.load_balancer_arn
    • [ ] ./components/aws/ecs-task/variables.tf:var.secrets_manager_kms_key_id
    • [ ] ./components/aws/ecs-task/variables.tf:var.use_fargate
    • [ ] ./components/aws/ecs-task/variables.tf:var.use_load_balancer
    • [ ] ./components/aws/lambda-python/variables.tf:var.pip_path
    • [ ] ./components/aws/lambda-python/variables.tf:var.runtime
    • [ ] ./components/aws/lambda-python/variables.tf:var.s3_triggers
    • [ ] ./components/aws/lambda-python/variables.tf:var.timeout_seconds
    • [ ] ./components/aws/redshift/variables.tf:var.database_name
    • [ ] ./components/aws/redshift/variables.tf:var.elastic_ip
    • [ ] ./components/aws/redshift/variables.tf:var.jdbc_port
    • [ ] ./components/aws/redshift/variables.tf:var.kms_key_id
    • [ ] ./components/aws/redshift/variables.tf:var.num_nodes
    • [ ] ./components/aws/redshift/variables.tf:var.s3_logging_bucket
    • [ ] ./components/aws/redshift/variables.tf:var.s3_logging_path
    • [ ] ./components/aws/redshift/variables.tf:var.skip_final_snapshot
    • [ ] ./components/aws/step-functions/variables.tf:var.account_id
    • [ ] ./components/aws/step-functions/variables.tf:var.state_machine_definition
    • [ ] ./components/aws/step-functions/variables.tf:var.state_machine_name
    • [ ] ./components/aws/vpc/variables.tf:var.aws_region
    • [ ] ./components/aws/vpc/variables.tf:var.name_prefix
    • [ ] ./components/aws/vpc/variables.tf:var.resource_tags

  5. Missing output variable descriptions:

    • [ ] ./catalog/aws/airflow/outputs.tf:output.airflow_url
    • [ ] ./catalog/aws/airflow/outputs.tf:output.logging_url
    • [ ] ./catalog/aws/airflow/outputs.tf:output.server_launch_cli
    • [ ] ./catalog/aws/airflow/outputs.tf:output.summary
    • [ ] ./catalog/aws/data-lake/outputs.tf:output.s3_data_bucket
    • [ ] ./catalog/aws/data-lake/outputs.tf:output.s3_logging_bucket
    • [ ] ./catalog/aws/data-lake/outputs.tf:output.s3_metadata_bucket
    • [ ] ./catalog/aws/data-lake/outputs.tf:output.summary
    • [ ] ./catalog/aws/dbt/outputs.tf:output.summary
    • [ ] ./catalog/aws/environment/outputs.tf:output.aws_credentials_file
    • [ ] ./catalog/aws/environment/outputs.tf:output.environment
    • [ ] ./catalog/aws/environment/outputs.tf:output.is_windows_host
    • [ ] ./catalog/aws/environment/outputs.tf:output.ssh_private_key_filename
    • [ ] ./catalog/aws/environment/outputs.tf:output.ssh_public_key_filename
    • [ ] ./catalog/aws/environment/outputs.tf:output.summary
    • [ ] ./catalog/aws/environment/outputs.tf:output.user_home
    • [ ] ./catalog/aws/redshift/outputs.tf:output.endpoint
    • [ ] ./catalog/aws/redshift/outputs.tf:output.summary
    • [ ] ./catalog/aws/singer-taps/outputs.tf:output.summary
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_instance_ids
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_instance_private_ips
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_instance_public_ips
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_instance_states
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_remote_admin_commands
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ec2_windows_instance_passwords
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ssh_private_key_path
    • [ ] ./catalog/aws/tableau-server/outputs.tf:output.ssh_public_key_path
    • [ ] ./components/aws/ec2/outputs.tf:output.instance_id
    • [ ] ./components/aws/ec2/outputs.tf:output.instance_ids
    • [ ] ./components/aws/ec2/outputs.tf:output.instance_state
    • [ ] ./components/aws/ec2/outputs.tf:output.instance_states
    • [ ] ./components/aws/ec2/outputs.tf:output.private_ip
    • [ ] ./components/aws/ec2/outputs.tf:output.private_ips
    • [ ] ./components/aws/ec2/outputs.tf:output.public_ips
    • [ ] ./components/aws/ec2/outputs.tf:output.remote_admin_commands
    • [ ] ./components/aws/ec2/outputs.tf:output.ssh_key_name
    • [ ] ./components/aws/ec2/outputs.tf:output.ssh_private_key_path
    • [ ] ./components/aws/ec2/outputs.tf:output.ssh_public_key_path
    • [ ] ./components/aws/ec2/outputs.tf:output.windows_instance_passwords
    • [ ] ./components/aws/ecr/outputs.tf:output.ecr_image_url
    • [ ] ./components/aws/ecr/outputs.tf:output.ecr_repo_arn
    • [ ] ./components/aws/ecr/outputs.tf:output.ecr_repo_root
    • [ ] ./components/aws/ecs-cluster/outputs.tf:output.ecs_cluster_arn
    • [ ] ./components/aws/ecs-cluster/outputs.tf:output.ecs_cluster_name
    • [ ] ./components/aws/ecs-cluster/outputs.tf:output.ecs_instance_role
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_checklogs_cli
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_container_name
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_logging_url
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_runtask_cli
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_security_group
    • [ ] ./components/aws/ecs-task/outputs.tf:output.ecs_task_name
    • [ ] ./components/aws/ecs-task/outputs.tf:output.load_balancer_arn
    • [ ] ./components/aws/ecs-task/outputs.tf:output.load_balancer_dns
    • [ ] ./components/aws/lambda-python/outputs.tf:output.build_temp_dir
    • [ ] ./components/aws/redshift/outputs.tf:output.endpoint
    • [ ] ./components/aws/redshift/outputs.tf:output.summary
    • [ ] ./components/aws/secrets-manager/outputs.tf:output.secrets_ids
    • [ ] ./components/aws/secrets-manager/outputs.tf:output.summary
    • [ ] ./components/aws/step-functions/outputs.tf:output.summary
    • [ ] ./components/aws/vpc/outputs.tf:output.private_subnets
    • [ ] ./components/aws/vpc/outputs.tf:output.public_subnets
    • [ ] ./components/aws/vpc/outputs.tf:output.vpc_id
aaronsteers commented 4 years ago

Missing headers are not resolved (deleted from above).