search

djloude / cloudwatch_exporter_metrics_config_builder

CI/CD pipeline that generates aws cloudwatch metrics and dimensions then builds the cloudwatch.yml file.
2 stars 6 forks source link

Ansible playbook fails with "<filename>.json must be stored as a dictionary/hash" #4

Closed daniduhnev1 closed 1 year ago

daniduhnev1 commented 1 year ago

Ansible playbook returns error when ran. See example full error message:

failed: [localhost] (item=/cloudwatch_exporter_metrics_config_builder/cloudwatch_config_builder/files/cloudwatch_metrics/auto_scaling.json) => {
    "ansible_facts": {},
    "ansible_included_var_files": [],
    "ansible_loop_var": "item",
    "changed": false,
    "item": "cloudwatch_exporter_metrics_config_builder/cloudwatch_config_builder/files/cloudwatch_metrics/transit_gateway.json",
    "message": "cloudwatch_exporter_metrics_config_builder/cloudwatch_config_builder/files/cloudwatch_metrics/transit_gateway.json must be stored as a dictionary/hash"
}

Here's how the cloudwatch_config_builder/files/cloudwatch_metrics/transit_gateway.json generated from the pipeline.sh script looks like:

cat transit_gateway.json

None
METRICS BytesDropCountBlackhole AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketsOut  AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesIn AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketDropCountBlackhole    AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketsIn   AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesOut    AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesDropCountBlackhole AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketDropCountNoRoute  AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesOut    AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesIn AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketDropCountNoRoute  AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketsOut  AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesOut    AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketsIn   AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesDropCountBlackhole AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesOut    AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesDropCountNoRoute   AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketsIn   AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesDropCountBlackhole AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketDropCountBlackhole    AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesIn AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketsIn   AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketsOut  AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketDropCountNoRoute  AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketsOut  AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesIn AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketDropCountNoRoute  AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesDropCountNoRoute   AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS PacketDropCountBlackhole    AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
METRICS BytesDropCountNoRoute   AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS BytesDropCountNoRoute   AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-09fcdd27877bd8abd
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a
METRICS PacketDropCountBlackhole    AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-0e0d912367b7a1c3a

I can confirm that running the aws cloudwatch list-metrics command generates the same output. I suspect the problem is with the way the main.py script is written because I tried manually changing one of the files to simply 

cat transit_gateway.json

some_variable: value

, then the ansible playbook passes for that particular file.

djloude commented 1 year ago

Hi @daniduhnev1, Are any other .json files formatted incorrectly?

daniduhnev1 commented 1 year ago

Hey @djloude, yes, all the .json files are in that format. Here is the full list of generated files:

❯ tree
.
├── auto_scaling.json
├── billing.json
├── dynamo_db.json
├── ebs.json
├── ec2.json
├── elasti_cache.json
├── events.json
├── lambda.json
├── logs.json
├── network_elb.json
├── rds.json
├── s3.json
├── transit_gateway.json
├── trusted_advisor.json
└── usage.json

1 directory, 15 files

Note that if I run the aws cloudwatch list-metrics command in my terminal I get the same output.

❯ aws cloudwatch list-metrics --namespace=AWS/TransitGateway

None
METRICS BytesIn AWS/TransitGateway
DIMENSIONS  TransitGatewayAttachment    tgw-attach-06569e5eae5c15d3e
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7
...
...
METRICS BytesDropCountNoRoute   AWS/TransitGateway
DIMENSIONS  TransitGateway  tgw-07ad63cf278f3f4e7

It seems that when running the main.py script, its not able to pick on these .json files and edit them accordingly.

djloude commented 1 year ago

Yes it looks like AWS might have changed how they return the data from the aws cloudwatch list-metrics command. It used to be a properly formatted JSON object that looked like:

{
    "Metrics": [
        {
            "Namespace": "AWS/TransitGateway",
            "MeticName": "BytesIn"
            "Dimensions": [
                 {
                      "Name": "TransitgatewayAttachment",
                      "Value": "tgw-attach-06569e5eae5c15d3e"
                 }
             ]
        },
...

and so on.

The format_metrics_yml_file function within main.py will need to be rewritten to reformat the aws cloudwatch list-metrics return object.

djloude commented 1 year ago

Try adding the global flag --output json on the cli command and see how the format is. Here are the cloudwatch list-metrics documentation

daniduhnev1 commented 1 year ago

Hi @djloude, this has now populated the metrics .json files in the right format and the ansible playbook is able to pass the pre-task of loading the config variables.

However, it trips up on the next task which is building the cloudwatch exporter metrics file with error:

ansible.errors.AnsibleUndefinedVariable: 'transit_gateway_metrics' is undefined. 'transit_gateway_metrics' is undefined
fatal: [localhost]: FAILED! => {
    "changed": false,
    "msg": "AnsibleUndefinedVariable: 'transit_gateway_metrics' is undefined. 'transit_gateway_metrics' is undefined"
}

This happens when I set the transit_gateway: true in the cloudwatch_config_builder/config.yml file. Otherwise if I don't set any of the variables to true, the playbooks executes successfully but as expected the cloudwatch_config_builder/cloudwatch.yml file is empty of any metrics and looks like this:

#
# Ansible managed
#
---
region: eu-west-1
metrics:

I suspect this is because the template in cloudwatch_config_builder/templates/cloudwatch.yml.j2 uses a for loop {% for metric in transit_gateway_metrics %} but I don't see the list of transit_gateway_metrics being specified or imported from anywhere.

djloude commented 1 year ago

The {% for metric in transit_gateway_metrics %} should be getting pulled from the transit_gateway.json. Can you send me a snippet of the JSON formatted transit_gateway.json file? Also are you trying to build other namespaces? if so is transit_gateway the only namespace giving you issues? (it should run top-to-bottom)