Column ordering in .csv output is inconsistent

[ciaran-finnegan]

Describe the bug The column ordering in .csv output is inconsistent for different queries, account_id sometimes precedes region and vice versa.

Steampipe version (steampipe -v) Example: 0.14.0-alpha.0

Plugin version (steampipe plugin list) Example: 0.52.0

To reproduce Steps to reproduce the behavior (please include relevant code and/or commands). Run the following checks:

• steampipe check aws_compliance.control.ec2_instance_ebs_optimized --export ec2_instance_ebs_optimized.csv
• steampipe check aws_compliance.control.foundational_security_redshift_3 --export redshift_cluster_automatic_snapshots_min_7_days.csv
• steampipe check aws_compliance.control.elb_classic_lb_cross_zone_load_balancing_enabled --export elb_classic_lb_cross_zone_load_balancing_enabled.csv
• steampipe check aws_compliance.control.elasticache_redis_cluster_automatic_backup_retention_15_days --export elasticache_redis_cluster_automatic_backup_retention_15_days.csv --output=brief

Note the headers in .csv output are ordered as follows (account_id precedes region): group_id,title,description,control_id,control_title,control_description,reason,resource,status,account_id,region,audit_manager_control_tower,hipaa,nist_csf,service,soc_2

Expected behavior Expect headers in .csv output to be ordered as follows (region precedes account_id): group_id,title,description,control_id,control_title,control_description,reason,resource,status,region,account_id,hipaa,nist_800_53_rev_4,nist_csf,service

Additional context Expected behaviour is based on the observed .csv column ordering for the following control checks:

• steampipe check aws_compliance.control.foundational_security_dynamodb_1 --export dynamodb_table_auto_scaling_enabled.csv --output=brief
• steampipe check aws_compliance.control.lambda_function_concurrent_execution_limit_configured --export lambda_function_concurrent_execution_limit_configured.csv --output=brief
• steampipe check aws_compliance.control.lambda_function_dead_letter_queue_configured --export lambda_function_dead_letter_queue_configured.csv --output=brief
• steampipe check aws_compliance.control.vpc_vpn_tunnel_up --export vpc_vpn_tunnel_up.csv --output=brief
• steampipe check aws_compliance.control.pci_v321_es_1 --export es_domain_in_vpc.csv --output=brief
• steampipe check aws_compliance.control.ec2_instance_in_vpc --export ec2_instance_in_vpc.csv --output=brief
• steampipe check aws_compliance.control.lambda_function_in_vpc --export lambda_function_in_vpc.csv --output=brief
• steampipe check aws_compliance.control.autoscaling_group_with_lb_use_health_check --export autoscaling_group_with_lb_use_health_check.csv --output=brief
• steampipe check aws_compliance.control.foundational_security_elasticbeanstalk_1 --export elastic_beanstalk_enhanced_health_reporting_enabled.csv --output=brief
• steampipe check aws_compliance.control.pci_v321_cw_1 --export log_metric_filter_root_login.csv --output=brief
• steampipe check aws_compliance.control.ec2_instance_detailed_monitoring_enabled --export ec2_instance_detailed_monitoring_enabled.csv --output=brief
• steampipe check aws_compliance.control.guardduty_enabled --export guardduty_enabled.csv --output=brief
• steampipe check aws_compliance.control.securityhub_enabled --export securityhub_enabled.csv --output=brief
• steampipe check aws_compliance.control.dynamodb_table_auto_scaling_enabled --export dynamodb_table_auto_scaling_enabled.csv --output=brief
• steampipe check aws_compliance.control.autoscaling_launch_config_public_ip_disabled --export autoscaling_launch_config_public_ip_disabled.csv --output=brief
• steampipe check aws_compliance.control.redshift_cluster_maintenance_settings_check --export redshift_cluster_maintenance_settings_check.csv --output=brief
• steampipe check aws_compliance.control.pci_v321_ssm_1 --export ssm_managed_instance_compliance_patch_compliant.csv --output=brief
• steampipe check aws_compliance.control.dynamodb_table_protected_by_backup_plan --export dynamodb_table_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.foundational_security_dynamodb_2 --export dynamodb_table_point_in_time_recovery_enabled.csv --output=brief
• steampipe check aws_compliance.control.ebs_volume_protected_by_backup_plan --export ebs_volume_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.efs_file_system_protected_by_backup_plan --export efs_file_system_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.elasticache_redis_cluster_automatic_backup_retention_15_days --export elasticache_redis_cluster_automatic_backup_retention_15_days.csv --output=brief
• steampipe check aws_compliance.control.rds_db_instance_protected_by_backup_plan --export rds_db_instance_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.s3_bucket_cross_region_replication_enabled --export s3_bucket_cross_region_replication_enabled.csv --output=brief
• steampipe check aws_compliance.control.rds_db_instance_multiple_az_enabled --export rds_db_instance_multiple_az_enabled.csv --output=brief
• steampipe check aws_compliance.control.foundational_security_elb_6 --export elb_application_lb_deletion_protection_enabled.csv --output=brief
• steampipe check aws_compliance.control.rds_db_instance_deletion_protection_enabled --export rds_db_instance_deletion_protection_enabled.csv --output=brief
• steampipe check aws_compliance.control.s3_bucket_object_lock_enabled --export s3_bucket_object_lock_enabled.csv --output=brief
• steampipe check aws_compliance.control.rds_db_cluster_aurora_protected_by_backup_plan --export rds_db_cluster_aurora_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.backup_plan_min_retention_35_days --export backup_plan_min_retention_35_days.csv --output=brief
• steampipe check aws_compliance.control.backup_recovery_point_encryption_enabled --export backup_recovery_point_encryption_enabled.csv --output=brief
• steampipe check aws_compliance.control.backup_recovery_point_manual_deletion_disabled --export backup_recovery_point_manual_deletion_disabled.csv --output=brief
• steampipe check aws_compliance.control.ec2_instance_protected_by_backup_plan --export ec2_instance_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.fsx_file_system_protected_by_backup_plan --export fsx_file_system_protected_by_backup_plan.csv --output=brief
• steampipe check aws_compliance.control.foundational_security_es_7 --export es_domain_dedicated_master_nodes_min_3.csv --output=brief
• steampipe check aws_compliance.control.foundational_security_es_6 --export es_domain_data_nodes_min_3.csv --output=brief

[misraved]

Hello @slartibastfast, Welcome to Steampipe 👍 !! Great to see that you are using the AWS Compliance mod in such detail !!!

The issue that you have mentioned has more to do with the core Steampipe CLI and this is already being tracked/investigated by @kaidaguerre

https://steampipe.slack.com/archives/C01UECB59A7/p1647893391363249

Please feel free to add your thoughts and experiences to the above thread 👍

[cbruno10]

@slartibastfast Please also see the relevant issue in the Steampipe repo, https://github.com/turbot/steampipe/issues/1715, for more updates.

[ciaran-finnegan]

Thanks for the quick reply!

[ciaran-finnegan]

Hi @misraved & @cbruno10, thanks so much, can confirm that column ordering appears to be consistent in 0.14.0-alpha.3, keep up the great work, loving Steampipe!

[misraved]

Thanks @slartibastfast for the confirmation !!

Great to know that you love Steampipe 👍 . Please do let us know in case you run into issues or have questions n general.