Inspection error resulting in ecs cluster not scaling down

[ghost]

In GitLab by @werwty on Mar 10, 2020, 16:18

Brad: Regarding the reaper, I just peeked at the CloudWatch Log Group for qa/test, and the last logs were at "2020-03-10 09:38 UTC-4" and appear to have normal activity in them. I wonder if something prevented the last run from even starting up successfully before it could log anything.

Brad: digging back up through the cloudwatch logs in test from earlier today, I found this curiosity… Provided drive(s) to inspect: (('ami-065d0be9883aa064a', '/dev/xvdba'), ('ami-065d0be9883aa064a', '/dev/xvdbb'), ('ami-065d0be9883aa064a', '/dev/xvdbc'), ('ami-065d0be9883aa064a', '/dev/xvdbd'), ('ami-065d0be9883aa064a', '/dev/xvdbe'), ('ami-065d0be9883aa064a', '/dev/xvdbf'), ('ami-065d0be9883aa064a', '/dev/xvdbg'), ('ami-065d0be9883aa064a', '/dev/xvdbh'), ('ami-065d0be9883aa064a', '/dev/xvdbi'), ('ami-065d0be9883aa064a', '/dev/xvdbj'), ('ami-065d0be9883aa064a', '/dev/xvdbk'), ('ami-065d0be9883aa064a', '/dev/xvdbl'), ('ami-065d0be9883aa064a', '/dev/xvdbm'), ('ami-065d0be9883aa064a', '/dev/xvdbn'), ('ami-065d0be9883aa064a', '/dev/xvdbo'), ('ami-065d0be9883aa064a', '/dev/xvdbp'), ('ami-065d0be9883aa064a', '/dev/xvdbq'), ('ami-065d0be9883aa064a', '/dev/xvdbr'), ('ami-065d0be9883aa064a', '/dev/xvdbs'), ('ami-065d0be9883aa064a', '/dev/xvdbt'), ('ami-065d0be9883aa064a', '/dev/xvdbu'), ('ami-065d0be9883aa064a', '/dev/xvdbv'), ('ami-065d0be9883aa064a', '/dev/xvdbw'), ('ami-065d0be9883aa064a', '/dev/xvdbx'), ('ami-065d0be9883aa064a', '/dev/xvdby'), ('ami-065d0be9883aa064a', '/dev/xvdbz'), ('ami-065d0be9883aa064a', '/dev/xvdca')) That's an awful lot of the same image which makes me think something buggy may be going on there.

[ghost]

In GitLab by @werwty on Mar 10, 2020, 16:19

assigned to @werwty

[ghost]

In GitLab by @infinitewarp on Mar 12, 2020, 09:49

unassigned @werwty

[ghost]

In GitLab by @werwty on Mar 13, 2020, 10:53

assigned to @werwty

[ghost]

In GitLab by @werwty on Mar 13, 2020, 14:16

I can't reproduce this problem. The AMI ami-065d0be9883aa064a is inspected correctly in a review environment.

{
            "cloud_type": "aws",
            "content_object": {
                "aws_image_id": 676,
                "created_at": "2020-03-12T15:10:55.886043Z",
                "ec2_ami_id": "ami-065d0be9883aa064a",
                "id": 676,
                "is_cloud_access": false,
                "is_marketplace": false,
                "owner_aws_account_id": "518028203513",
                "platform": "none",
                "region": "us-east-1",
                "updated_at": "2020-03-12T15:10:55.886059Z"
            },
            "created_at": "2020-03-12T15:10:55.891388Z",
            "image_id": 673,
            "inspection_json": "{\"/dev/xvdba\": {\"/dev/xvdba1\": {\"facts\": {\"os_version\": null, \"rhel_enabled_repos\": {\"rhel_enabled_repos\": [], \"rhel_found\": false}, \"rhel_product_certs\": {\"rhel_found\": false, \"rhel_pem_files\": []}, \"rhel_release_files\": {\"rhel_found\": false, \"status\": \"No release files found on /dev/xvdba1\"}, \"rhel_signed_packages\": {\"rhel_found\": false, \"rhel_signed_package_count\": 0, \"status\": \"RPM DB directory on /dev/xvdba1 has no data for ami-065d0be9883aa064a\"}, \"syspurpose_contents\": null}}, \"/dev/xvdba2\": {\"error\": \"mount: /mnt/inspect: unknown filesystem type 'swap'.\\n\", \"facts\": {\"rhel_enabled_repos\": {}, \"rhel_product_certs\": {}, \"rhel_release_files\": {}, \"rhel_signed_packages\": {}}}, \"/dev/xvdba3\": {\"facts\": {\"os_version\": \"7.4\", \"rhel_enabled_repos\": {\"rhel_enabled_repos\": [], \"rhel_found\": false}, \"rhel_product_certs\": {\"rhel_found\": true, \"rhel_pem_files\": [\"/mnt/inspect/etc/pki/product-default/69.pem\"]}, \"rhel_release_files\": {\"release_files\": [{\"rhel_found\": true, \"rhel_release_file\": \"/etc/os-release\", \"rhel_release_file_contents\": \"NAME=\\\"Red Hat Enterprise Linux Server\\\"\\nVERSION=\\\"7.4 (Maipo)\\\"\\nID=\\\"rhel\\\"\\nID_LIKE=\\\"fedora\\\"\\nVARIANT=\\\"Server\\\"\\nVARIANT_ID=\\\"server\\\"\\nVERSION_ID=\\\"7.4\\\"\\nPRETTY_NAME=\\\"Red Hat Enterprise Linux Server 7.4 (Maipo)\\\"\\nANSI_COLOR=\\\"0;31\\\"\\nCPE_NAME=\\\"cpe:/o:redhat:enterprise_linux:7.4:GA:server\\\"\\nHOME_URL=\\\"https://www.redhat.com/\\\"\\nBUG_REPORT_URL=\\\"https://bugzilla.redhat.com/\\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\\"Red Hat Enterprise Linux 7\\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=7.4\\nREDHAT_SUPPORT_PRODUCT=\\\"Red Hat Enterprise Linux\\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\\"7.4\\\"\\n\"}, {\"rhel_found\": true, \"rhel_release_file\": \"/etc/redhat-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux Server release 7.4 (Maipo)\\n\"}, {\"rhel_found\": true, \"rhel_release_file\": \"/etc/system-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux Server release 7.4 (Maipo)\\n\"}], \"rhel_found\": true}, \"rhel_signed_packages\": {\"rhel_found\": true, \"rhel_signed_package_count\": 323}, \"syspurpose_contents\": null}}}, \"/dev/xvdbb\": {\"/dev/xvdbb1\": {\"facts\": {\"os_version\": null, \"rhel_enabled_repos\": {\"rhel_enabled_repos\": [], \"rhel_found\": false}, \"rhel_product_certs\": {\"rhel_found\": false, \"rhel_pem_files\": []}, \"rhel_release_files\": {\"rhel_found\": false, \"status\": \"No release files found on /dev/xvdbb1\"}, \"rhel_signed_packages\": {\"rhel_found\": false, \"rhel_signed_package_count\": 0, \"status\": \"RPM DB directory on /dev/xvdbb1 has no data for ami-065d0be9883aa064a\"}, \"syspurpose_contents\": null}}, \"/dev/xvdbb2\": {\"error\": \"mount: /mnt/inspect: unknown filesystem type 'swap'.\\n\", \"facts\": {\"rhel_enabled_repos\": {}, \"rhel_product_certs\": {}, \"rhel_release_files\": {}, \"rhel_signed_packages\": {}}}, \"/dev/xvdbb3\": {\"facts\": {\"os_version\": \"7.4\", \"rhel_enabled_repos\": {\"rhel_enabled_repos\": [], \"rhel_found\": false}, \"rhel_product_certs\": {\"rhel_found\": true, \"rhel_pem_files\": [\"/mnt/inspect/etc/pki/product-default/69.pem\"]}, \"rhel_release_files\": {\"release_files\": [{\"rhel_found\": true, \"rhel_release_file\": \"/etc/os-release\", \"rhel_release_file_contents\": \"NAME=\\\"Red Hat Enterprise Linux Server\\\"\\nVERSION=\\\"7.4 (Maipo)\\\"\\nID=\\\"rhel\\\"\\nID_LIKE=\\\"fedora\\\"\\nVARIANT=\\\"Server\\\"\\nVARIANT_ID=\\\"server\\\"\\nVERSION_ID=\\\"7.4\\\"\\nPRETTY_NAME=\\\"Red Hat Enterprise Linux Server 7.4 (Maipo)\\\"\\nANSI_COLOR=\\\"0;31\\\"\\nCPE_NAME=\\\"cpe:/o:redhat:enterprise_linux:7.4:GA:server\\\"\\nHOME_URL=\\\"https://www.redhat.com/\\\"\\nBUG_REPORT_URL=\\\"https://bugzilla.redhat.com/\\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\\"Red Hat Enterprise Linux 7\\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=7.4\\nREDHAT_SUPPORT_PRODUCT=\\\"Red Hat Enterprise Linux\\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\\"7.4\\\"\\n\"}, {\"rhel_found\": true, \"rhel_release_file\": \"/etc/redhat-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux Server release 7.4 (Maipo)\\n\"}, {\"rhel_found\": true, \"rhel_release_file\": \"/etc/system-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux Server release 7.4 (Maipo)\\n\"}], \"rhel_found\": true}, \"rhel_signed_packages\": {\"rhel_found\": true, \"rhel_signed_package_count\": 323}, \"syspurpose_contents\": null}}}, \"rhel_enabled_repos_found\": false, \"rhel_found\": true, \"rhel_product_certs_found\": true, \"rhel_release_files_found\": true, \"rhel_signed_packages_found\": true, \"rhel_version\": \"7.4\", \"syspurpose\": null}",
            "is_encrypted": false,
            "name": "import-ami-05c88748fa241d139",
            "openshift": false,
            "openshift_detected": false,
            "rhel": true,
            "rhel_detected": true,
            "rhel_detected_by_tag": false,
            "rhel_enabled_repos_found": false,
            "rhel_product_certs_found": true,
            "rhel_release_files_found": true,
            "rhel_signed_packages_found": true,
            "rhel_version": "7.4",
            "status": "inspected",
            "syspurpose": null,
            "updated_at": "2020-03-13T04:38:58.208310Z"
        },

If this problem persists we'll open another issue with more details

[ghost]

In GitLab by @werwty on Mar 13, 2020, 14:16

closed