Inspection process returns different/conflicting results #111

Closed ghost closed 3 years ago

In GitLab by @Maribeth on Sep 10, 2020, 15:28

Summary

When Houndigrade's inspection process runs against images, it occasionally returns different results. Like sometimes it reports back RHEL=True and sometimes it reports back RHEL=False for instances of the same image.

Steps to Reproduce

In the qa env, using dev07 AWS account

Run a set of instances from images (see suggestions below)
Create a Cloud Meter account via Sources
Wait for CM inspection process to run and observe the results of the inspections.

Test automation fired up all these instances, created the Source and Cloud Meter accounts and waited for inspection. Once the inspection process finished, I httpied cloudigrade/v2/images/#####/ to see the results of the inspection.

Images used in my current test:

Image ID: ami-0d202a94482e3d42c 
Image ID: ami-0169e18bfd191f4e6 
Image ID: ami-06d8921b0a9cd7c9c 
Image ID: ami-054fcc5334b1abd1c 
Image ID: ami-07e90d06363f8ebd9 
Image ID: ami-078d7e9b3e890660a 
Image ID: ami-08d57a048f7c08e46

Expected Result

Houndigrade consistently identifies instances of the same image in the same way RHEL=True vs. RHEL=False

Actual Result

Houndigrade doesn't always return the same response.

Additional context

Responses from inspection process for image ami-0169e18bfd191f4e6 (RHEL=True)

http --verify=false --auth massopencloud:redhat https://qa.cloud.redhat.com/api/cloudigrade/v2/images/18431/
HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-Powered-By: Express
allow: GET, HEAD, OPTIONS
cache-control: private
connection: keep-alive
content-type: application/json
date: Thu, 10 Sep 2020 18:57:27 GMT
server: openresty/1.13.6.1
set-cookie: a549c90418df47840900855e57bbdfae=d12fee53c0ee7061f89cdf8133ebf995; path=/; HttpOnly; Secure
vary: Accept
x-cloudigrade-request-id: e852beb9-aaec-40ba-a814-7095e3414e3a
x-content-type-options: nosniff
x-fakamai-app-hint: API
x-fakamai-downstream-host: https://api-gateway-qa.5a9f.insights-dev.openshiftapps.com
x-fakamai-env: qa
x-frame-options: DENY
x-rh-insights-request-id: 8d1fce305df04efab0d00eb20a5a8215

{
    "architecture": "x86_64",
    "cloud_type": "aws",
    "content_object": {
        "aws_image_id": 18435,
        "created_at": "2020-09-10T18:18:23.225495Z",
        "ec2_ami_id": "ami-0169e18bfd191f4e6",
        "id": 18435,
        "is_cloud_access": false,
        "is_marketplace": false,
        "owner_aws_account_id": "439727791560",
        "platform": "none",
        "region": "us-east-1",
        "updated_at": "2020-09-10T18:18:23.225519Z"
    },
    "created_at": "2020-09-10T18:18:23.232922Z",
    "image_id": 18431,
    "inspection_json": "{\"rhel_found\": true, \"rhel_signed_packages_found\": true, \"rhel_product_certs_found\": true, \"rhel_release_files_found\": true, \"rhel_enabled_repos_found\": true, \"rhel_version\": \"8.1\", \"syspurpose\": {\"role\": \"Red Hat Enterprise Linux Server\", \"service_level_agreement\": \"Self-Support\", \"usage\": \"Disaster Recovery\"}, \"drives\": {\"/dev/xvdbb\": {\"/dev/xvdbb2\": {\"facts\": {\"rhel_release_files\": {\"rhel_found\": true, \"release_files\": [{\"rhel_release_file\": \"/etc/os-release\", \"rhel_release_file_contents\": \"NAME=\\\"Red Hat Enterprise Linux\\\"\\nVERSION=\\\"8.1 (Ootpa)\\\"\\nID=\\\"rhel\\\"\\nID_LIKE=\\\"fedora\\\"\\nVERSION_ID=\\\"8.1\\\"\\nPLATFORM_ID=\\\"platform:el8\\\"\\nPRETTY_NAME=\\\"Red Hat Enterprise Linux 8.1 (Ootpa)\\\"\\nANSI_COLOR=\\\"0;31\\\"\\nCPE_NAME=\\\"cpe:/o:redhat:enterprise_linux:8.1:GA\\\"\\nHOME_URL=\\\"https://www.redhat.com/\\\"\\nBUG_REPORT_URL=\\\"https://bugzilla.redhat.com/\\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\\"Red Hat Enterprise Linux 8\\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=8.1\\nREDHAT_SUPPORT_PRODUCT=\\\"Red Hat Enterprise Linux\\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\\"8.1\\\"\\n\", \"rhel_found\": true}, {\"rhel_release_file\": \"/etc/redhat-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux release 8.1 (Ootpa)\\n\", \"rhel_found\": true}, {\"rhel_release_file\": \"/etc/system-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux release 8.1 (Ootpa)\\n\", \"rhel_found\": true}]}, \"rhel_product_certs\": {\"rhel_pem_files\": [\"/mnt/inspect/etc/pki/product-default/479.pem\"], \"rhel_found\": true}, \"rhel_signed_packages\": {\"rhel_found\": true, \"rhel_signed_package_count\": 402}, \"rhel_enabled_repos\": {\"rhel_found\": true, \"rhel_enabled_repos\": [{\"repo\": \"rhui-client-config-server-8\", \"name\": \"Red Hat Update Infrastructure 3 Client Configuration Server 8\"}, {\"repo\": \"rhui-rhel-8-baseos-rhui-rpms\", \"name\": \"Red Hat Enterprise Linux 8 for $basearch - BaseOS from RHUI (RPMs)\"}, {\"repo\": \"rhui-rhel-8-appstream-rhui-rpms\", \"name\": \"Red Hat Enterprise Linux 8 for $basearch - AppStream from RHUI (RPMs)\"}]}, \"os_version\": \"8.1\", \"syspurpose_contents\": \"{\\n  \\\"role\\\": \\\"Red Hat Enterprise Linux Server\\\",\\n  \\\"service_level_agreement\\\": \\\"Self-Support\\\",\\n  \\\"usage\\\": \\\"Disaster Recovery\\\"\\n}\\n\"}}, \"/dev/xvdbb1\": {\"facts\": {\"rhel_release_files\": {}, \"rhel_product_certs\": {}, \"rhel_signed_packages\": {}, \"rhel_enabled_repos\": {}}, \"error\": {\"py/b64\": \"bW91bnQ6IC9tbnQvaW5zcGVjdDogd3JvbmcgZnMgdHlwZSwgYmFkIG9wdGlvbiwgYmFkIHN1cGVyYmxvY2sgb24gL2Rldi94dmRiYjEsIG1pc3NpbmcgY29kZXBhZ2Ugb3IgaGVscGVyIHByb2dyYW0sIG9yIG90aGVyIGVycm9yLgo=\"}}}}, \"errors\": [\"Mount of /dev/xvdbb1 on image ami-0169e18bfd191f4e6 failed with error: b'mount: /mnt/inspect: wrong fs type, bad option, bad superblock on /dev/xvdbb1, missing codepage or helper program, or other error.\\\\n' full_command: /usr/bin/mount -t auto /dev/xvdbb1 /mnt/inspect stdout: b''\"]}",
    "is_encrypted": false,
    "name": "RHEL-x86_64-Server-self-support",
    "openshift": false,
    "openshift_detected": false,
    "rhel": true,
    "rhel_detected": true,
    "rhel_detected_by_tag": false,
    "rhel_enabled_repos_found": true,
    "rhel_product_certs_found": true,
    "rhel_release_files_found": true,
    "rhel_signed_packages_found": true,
    "rhel_version": "8.1",
    "status": "inspected",
    "syspurpose": {
        "role": "Red Hat Enterprise Linux Server",
        "service_level_agreement": "Self-Support",
        "usage": "Disaster Recovery"
    },
    "updated_at": "2020-09-10T18:36:02.432018Z"
}

Same image, RHEL=False

✗ http --verify=false --auth massopencloud:redhat https://qa.cloud.redhat.com/api/cloudigrade/v2/images/18410/
HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-Powered-By: Express
allow: GET, HEAD, OPTIONS
cache-control: private
connection: keep-alive
content-type: application/json
date: Thu, 10 Sep 2020 17:22:18 GMT
server: openresty/1.13.6.1
set-cookie: a549c90418df47840900855e57bbdfae=bf36ce8382f24bdd5b70b258298f5092; path=/; HttpOnly; Secure
vary: Accept
x-cloudigrade-request-id: 14777d57-3733-420a-aea5-b29da20653c4
x-content-type-options: nosniff
x-fakamai-app-hint: API
x-fakamai-downstream-host: https://api-gateway-qa.5a9f.insights-dev.openshiftapps.com
x-fakamai-env: qa
x-frame-options: DENY
x-rh-insights-request-id: 062a64ecfc6e424fa2da91d3f0fca738

{
    "architecture": "x86_64",
    "cloud_type": "aws",
    "content_object": {
        "aws_image_id": 18414,
        "created_at": "2020-09-10T15:10:05.538916Z",
        "ec2_ami_id": "ami-0169e18bfd191f4e6",
        "id": 18414,
        "is_cloud_access": false,
        "is_marketplace": false,
        "owner_aws_account_id": "439727791560",
        "platform": "none",
        "region": "us-east-1",
        "updated_at": "2020-09-10T15:10:05.538940Z"
    },
    "created_at": "2020-09-10T15:10:05.545851Z",
    "image_id": 18410,
    "inspection_json": "{\"rhel_found\": false, \"rhel_signed_packages_found\": false, \"rhel_product_certs_found\": false, \"rhel_release_files_found\": false, \"rhel_enabled_repos_found\": false, \"rhel_version\": null, \"syspurpose\": null, \"drives\": {\"/dev/xvdbd\": {}}, \"errors\": [\"Nothing found at path /dev/xvdbd for ami-0169e18bfd191f4e6\"]}",
    "is_encrypted": false,
    "name": "RHEL-x86_64-Server-self-support",
    "openshift": false,
    "openshift_detected": false,
    "rhel": false,
    "rhel_detected": false,
    "rhel_detected_by_tag": false,
    "rhel_enabled_repos_found": false,
    "rhel_product_certs_found": false,
    "rhel_release_files_found": false,
    "rhel_signed_packages_found": false,
    "rhel_version": null,
    "status": "inspected",
    "syspurpose": null,
    "updated_at": "2020-09-10T15:16:01.766897Z"
}

Inspection results returned RHEL=False for different image ami-07e90d06363f8ebd9

✗ http --verify=false --auth massopencloud:redhat https://qa.cloud.redhat.com/api/cloudigrade/v2/images/18436/
HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-Powered-By: Express
allow: GET, HEAD, OPTIONS
cache-control: private
connection: keep-alive
content-type: application/json
date: Thu, 10 Sep 2020 18:58:53 GMT
server: openresty/1.13.6.1
set-cookie: a549c90418df47840900855e57bbdfae=d12fee53c0ee7061f89cdf8133ebf995; path=/; HttpOnly; Secure
vary: Accept
x-cloudigrade-request-id: 9a13c7b7-8a85-4ab1-80be-ef8727afae97
x-content-type-options: nosniff
x-fakamai-app-hint: API
x-fakamai-downstream-host: https://api-gateway-qa.5a9f.insights-dev.openshiftapps.com
x-fakamai-env: qa
x-frame-options: DENY
x-rh-insights-request-id: 468cae6113474fd9b7d05edd9c1d7f6f

{
    "architecture": "arm64",
    "cloud_type": "aws",
    "content_object": {
        "aws_image_id": 18440,
        "created_at": "2020-09-10T18:18:23.327385Z",
        "ec2_ami_id": "ami-07e90d06363f8ebd9",
        "id": 18440,
        "is_cloud_access": false,
        "is_marketplace": false,
        "owner_aws_account_id": "439727791560",
        "platform": "none",
        "region": "us-east-1",
        "updated_at": "2020-09-10T18:18:23.327406Z"
    },
    "created_at": "2020-09-10T18:18:23.332979Z",
    "image_id": 18436,
    "inspection_json": "{\"rhel_found\": false, \"rhel_signed_packages_found\": false, \"rhel_product_certs_found\": false, \"rhel_release_files_found\": false, \"rhel_enabled_repos_found\": false, \"rhel_version\": null, \"syspurpose\": null, \"drives\": {\"/dev/xvdbf\": {}}, \"errors\": [\"Nothing found at path /dev/xvdbf for ami-07e90d06363f8ebd9\"]}",
    "is_encrypted": false,
    "name": "RHEL x86_64 Server Standard for inspections",
    "openshift": true,
    "openshift_detected": true,
    "rhel": false,
    "rhel_detected": false,
    "rhel_detected_by_tag": false,
    "rhel_enabled_repos_found": false,
    "rhel_product_certs_found": false,
    "rhel_release_files_found": false,
    "rhel_signed_packages_found": false,
    "rhel_version": null,
    "status": "inspected",
    "syspurpose": null,
    "updated_at": "2020-09-10T18:36:02.519877Z"
}

and RHEL=True from a different inspection of the same image:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-Powered-By: Express
allow: GET, HEAD, OPTIONS
cache-control: private
connection: keep-alive
content-type: application/json
date: Thu, 10 Sep 2020 17:25:01 GMT
server: openresty/1.13.6.1
set-cookie: a549c90418df47840900855e57bbdfae=963a6c10b9914ab1921ff0d39c17b629; path=/; HttpOnly; Secure
vary: Accept
x-cloudigrade-request-id: fb5aeb8c-b76b-43cb-9ed5-95e5c7cd5a06
x-content-type-options: nosniff
x-fakamai-app-hint: API
x-fakamai-downstream-host: https://api-gateway-qa.5a9f.insights-dev.openshiftapps.com
x-fakamai-env: qa
x-frame-options: DENY
x-rh-insights-request-id: bb645794e1d742e68bcd0c02bed21355

{
    "architecture": "arm64",
    "cloud_type": "aws",
    "content_object": {
        "aws_image_id": 18418,
        "created_at": "2020-09-10T15:10:05.641083Z",
        "ec2_ami_id": "ami-07e90d06363f8ebd9",
        "id": 18418,
        "is_cloud_access": false,
        "is_marketplace": false,
        "owner_aws_account_id": "439727791560",
        "platform": "none",
        "region": "us-east-1",
        "updated_at": "2020-09-10T15:10:05.641098Z"
    },
    "created_at": "2020-09-10T15:10:05.646650Z",
    "image_id": 18414,
    "inspection_json": "{\"rhel_found\": true, \"rhel_signed_packages_found\": true, \"rhel_product_certs_found\": false, \"rhel_release_files_found\": true, \"rhel_enabled_repos_found\": true, \"rhel_version\": \"8.0\", \"syspurpose\": {\"role\": \"Red Hat Enterprise Linux Server\", \"service_level_agreement\": \"premium\", \"usage\": \"test\"}, \"drives\": {\"/dev/xvdbb\": {\"/dev/xvdbb3\": {\"facts\": {\"rhel_release_files\": {\"rhel_found\": true, \"release_files\": [{\"rhel_release_file\": \"/etc/os-release\", \"rhel_release_file_contents\": \"NAME=\\\"Red Hat Enterprise Linux\\\"\\nVERSION=\\\"8.0 (Ootpa)\\\"\\nID=\\\"rhel\\\"\\nID_LIKE=\\\"fedora\\\"\\nVERSION_ID=\\\"8.0\\\"\\nPLATFORM_ID=\\\"platform:el8\\\"\\nPRETTY_NAME=\\\"Red Hat Enterprise Linux 8.0 (Ootpa)\\\"\\nANSI_COLOR=\\\"0;31\\\"\\nCPE_NAME=\\\"cpe:/o:redhat:enterprise_linux:8.0:GA\\\"\\nHOME_URL=\\\"https://www.redhat.com/\\\"\\nBUG_REPORT_URL=\\\"https://bugzilla.redhat.com/\\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\\"Red Hat Enterprise Linux 8\\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=8.0\\nREDHAT_SUPPORT_PRODUCT=\\\"Red Hat Enterprise Linux\\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\\"8.0\\\"\\n\", \"rhel_found\": true}, {\"rhel_release_file\": \"/etc/redhat-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux release 8.0 (Ootpa)\\n\", \"rhel_found\": true}, {\"rhel_release_file\": \"/etc/system-release\", \"rhel_release_file_contents\": \"Red Hat Enterprise Linux release 8.0 (Ootpa)\\n\", \"rhel_found\": true}]}, \"rhel_product_certs\": {\"rhel_pem_files\": [], \"rhel_found\": false}, \"rhel_signed_packages\": {\"rhel_found\": true, \"rhel_signed_package_count\": 404}, \"rhel_enabled_repos\": {\"rhel_found\": true, \"rhel_enabled_repos\": [{\"repo\": \"rhui-client-config-server-8\", \"name\": \"Red Hat Update Infrastructure 3 Client Configuration Server 8\"}, {\"repo\": \"rhui-rhel-8-baseos-rhui-rpms\", \"name\": \"Red Hat Enterprise Linux 8 for $basearch - BaseOS from RHUI (RPMs)\"}, {\"repo\": \"rhui-rhel-8-appstream-rhui-rpms\", \"name\": \"Red Hat Enterprise Linux 8 for $basearch - AppStream from RHUI (RPMs)\"}]}, \"os_version\": \"8.0\", \"syspurpose_contents\": \"{\\n  \\\"role\\\": \\\"Red Hat Enterprise Linux Server\\\",\\n  \\\"service_level_agreement\\\": \\\"premium\\\",\\n  \\\"usage\\\": \\\"test\\\"\\n}\\n\"}}, \"/dev/xvdbb2\": {\"facts\": {\"rhel_release_files\": {\"rhel_found\": false, \"status\": \"No release files found on /dev/xvdbb2\"}, \"rhel_product_certs\": {\"rhel_pem_files\": [], \"rhel_found\": false}, \"rhel_signed_packages\": {\"rhel_found\": false, \"rhel_signed_package_count\": 0, \"status\": \"RPM DB directory on /dev/xvdbb2 has no data for ami-07e90d06363f8ebd9\"}, \"rhel_enabled_repos\": {\"rhel_found\": false, \"rhel_enabled_repos\": []}, \"os_version\": null, \"syspurpose_contents\": null}}, \"/dev/xvdbb1\": {\"facts\": {\"rhel_release_files\": {\"rhel_found\": false, \"status\": \"No release files found on /dev/xvdbb1\"}, \"rhel_product_certs\": {\"rhel_pem_files\": [], \"rhel_found\": false}, \"rhel_signed_packages\": {\"rhel_found\": false, \"rhel_signed_package_count\": 0, \"status\": \"RPM DB directory on /dev/xvdbb1 has no data for ami-07e90d06363f8ebd9\"}, \"rhel_enabled_repos\": {\"rhel_found\": false, \"rhel_enabled_repos\": []}, \"os_version\": null, \"syspurpose_contents\": null}}}}, \"errors\": []}",
    "is_encrypted": false,
    "name": "RHEL x86_64 Server Standard for inspections",
    "openshift": true,
    "openshift_detected": true,
    "rhel": true,
    "rhel_detected": true,
    "rhel_detected_by_tag": false,
    "rhel_enabled_repos_found": true,
    "rhel_product_certs_found": false,
    "rhel_release_files_found": true,
    "rhel_signed_packages_found": true,
    "rhel_version": "8.0",
    "status": "inspected",
    "syspurpose": {
        "role": "Red Hat Enterprise Linux Server",
        "service_level_agreement": "premium",
        "usage": "test"
    },
    "updated_at": "2020-09-10T15:16:01.706850Z"
}

In GitLab by @infinitewarp on Sep 10, 2020, 15:51

marked this issue as related to cloudigrade#728

In GitLab by @infinitewarp on Sep 10, 2020, 15:51

I'm marking this as related to https://gitlab.com/cloudigrade/cloudigrade/-/issues/728 because I think that may be the root cause of this problem.

In GitLab by @infinitewarp on Sep 17, 2020, 10:49

removed the relation with cloudigrade#728

In GitLab by @infinitewarp on Sep 17, 2020, 10:49

marked this issue as related to cloudigrade#728

In GitLab by @mpierce on Oct 6, 2020, 14:45

There are additional images that return conflicting results from inspections. Still in DEV07 account. All details about recreating behavior are the same as above.

(The details below are formatted differently in each run because I didn't realize I needed to collect it until after-the-fact and that's what I had available. I'll gather more details in another run if needed.)

ami-01eb13851a0ed6486

and

ami-0d1b0d4e138bf4cdf

More details about each image is below:

'ami-01eb13851a0ed6486': image(
    ami='ami-01eb13851a0ed6486', 
    arch='x86_64', 
    role='Workstation', 
    type='t2.micro', 
    sla='Premium', 
    usage='Dev/Test', 
    serv_type='L3', 
    name='swatch-conc-test-1', 
    RHEL_expected='False'
),

test_run_2:
Arch: x86_64
AMI: ami-01eb13851a0ed6486
RHEL: True
rhel_detected: True


test_run_1:
'ami-0d1b0d4e138bf4cdf': image(
    ami='ami-0d1b0d4e138bf4cdf', 
    arch='arm64', 
    role='Console', 
    type='a1.medium', 
    sla='self-support', 
    usage='prod', 
    serv_type='L3', 
    name='swatch-concurrent-test-4', 
    RHEL_expected='True'
),

test_run_2:
Arch: arm64
AMI: ami-0d1b0d4e138bf4cdf
RHEL: False
rhel_detected: False```

In GitLab by @infinitewarp on Oct 8, 2020, 09:52

assigned to @infinitewarp

In GitLab by @infinitewarp on Oct 12, 2020, 10:55

mentioned in merge request cloudigrade!806

In GitLab by @infinitewarp on Oct 12, 2020, 10:58

mentioned in merge request shiftigrade!136

In GitLab by @infinitewarp on Oct 12, 2020, 14:04

With the aforementioned cloudigrade!806, I introduced changes to cloudigrade's inspection process so that it should verify that all expected volumes are actually attached to the cluster instance before running the houndigrade task, and it should back off and retry until volumes are ready.

In GitLab by @infinitewarp on Oct 13, 2020, 09:49

assigned to @pakamble

In GitLab by @pakamble on Oct 15, 2020, 06:28

Took several round of image inspection test runs and no image inspection is showing conflicting results. find the log file here. images

cloudigrade / houndigrade