Inspection process hanging in `preparing` or `inspecting` #106

Closed ghost closed 3 years ago

In GitLab by @mpierce on Jul 30, 2020, 14:53

Summary

When I

run one instance each of two different x86_64 and/or arm64 images that should be inspected and then recognized as RHEL,
then create a source and thereby a Cloud Meter account,

Houndigrade's inspection process hangs up and doesn't finish, leaving instances in either status:preparing or status:inspecting

Steps to Reproduce

Launch an instance each of the below two images
Create a Source, Cloud Meter account
Wait for inspection process (should not take more than 30 minutes (really 15, but just to be sure)).
Check facts returned by Cloud Meter about each instance

    Image(
        'ami-0d202a94482e3d42c',
        'x86_64',
        't2.micro',
        'Premium',
        'Development/Test',
        'RHEL-x86_64-Workstation-premium',
    ),
    Image(
        'ami-0169e18bfd191f4e6',
        'x86_64',
        't2.micro',
        'Self_support',
        'Disaster Recovery',
        'RHEL-x86_64-Server-self-support',
    ),
    Image(
        'ami-06d8921b0a9cd7c9c',
        'arm64',
        'a1.medium',
        '',
        'Red Hat Enterprise Linux Server',
        'RHEL x86_64 Server no-sla for inspections', # x86_64 is a lie here, but I can't change the name
        ),
    Image( 
        'ami-054fcc5334b1abd1c',
        'arm64',
        'a1.medium',
        'self-support',
        'Red Hat Enterprise Linux Server',
        'RHEL x86_64 Server Self-support for inspections', # x86_64 is a lie here, but I can't change the name
        ),

These images are in the DEV07 account. Much greater detail about each of the instances is available in the attached files.

Expected Result

Within a reasonable amount of time (< 20 min?), images are:
- status:inspected
- identified as RHEL
- Cloud Meter reports accurate syspurpose.json details

Actual Result

Even after 60 minutes, instances have not made it through the inspection process. Process seems to have frozen.

Additional context

All of these images are originally copies of Red Hat's Cloud Accounts (Access2) images. Based on our present understanding of our code and the way AWS behaves, that should be OKAY and largely irrelevant since the new copies both A) have a different owner account ID and B) have new names that DO NOT match the Cloud Access naming conventions. However, it's a possible connection to weird inspection behaviors!!

In GitLab by @infinitewarp on Jul 31, 2020, 17:06

assigned to @infinitewarp

In GitLab by @infinitewarp on Aug 3, 2020, 10:36

changed the description

In GitLab by @infinitewarp on Aug 6, 2020, 09:50

unassigned @infinitewarp

In GitLab by @infinitewarp on Aug 6, 2020, 09:50

assigned to @katherine-black

In GitLab by @infinitewarp on Aug 11, 2020, 09:52

marked this issue as related to cloudigrade#682

In GitLab by @katherine-black on Aug 24, 2020, 10:46

mentioned in merge request !85

In GitLab by @pakamble on Aug 26, 2020, 05:44

Following image got stuck in 'inspecting' phase (since more than 2 hr).

AMI-ID : ami-065d0be9883aa064a

Image source : Private image.

AWS Account: DEV07

 % http --verify=no --auth $AUTH https://qa.cloud.redhat.com/api/cloudigrade/v2/images/
HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-Powered-By: Express
allow: GET, HEAD, OPTIONS
cache-control: private
connection: keep-alive
content-type: application/json
date: Wed, 26 Aug 2020 08:50:50 GMT
server: openresty/1.13.6.1
set-cookie: a549c90418df47840900855e57bbdfae=3fe0c683344f8e56cb4d585aba55b0ec; path=/; HttpOnly; Secure
vary: Accept
x-cloudigrade-request-id: 2168aee1-c260-4562-852b-08f7e6547b24
x-content-type-options: nosniff
x-fakamai-app-hint: API
x-fakamai-downstream-host: https://api-gateway-qa.5a9f.insights-dev.openshiftapps.com
x-fakamai-env: qa
x-frame-options: DENY
x-rh-insights-request-id: 0efdaf0fbce740939b2c4e09b2bc1283

{
    "data": [
        {
            "architecture": "x86_64",
            "cloud_type": "aws",
            "content_object": {
                "aws_image_id": 17549,
                "created_at": "2020-08-26T07:27:18.921700Z",
                "ec2_ami_id": "ami-065d0be9883aa064a",
                "id": 17549,
                "is_cloud_access": false,
                "is_marketplace": false,
                "owner_aws_account_id": "518028203513",
                "platform": "none",
                "region": "us-east-1",
                "updated_at": "2020-08-26T07:27:18.921734Z"
            },
            "created_at": "2020-08-26T07:27:18.928557Z",
            "image_id": 17545,
            "inspection_json": null,
            "is_encrypted": false,
            "name": "import-ami-05c88748fa241d139",
            "openshift": false,
            "openshift_detected": false,
            "rhel": false,
            "rhel_detected": false,
            "rhel_detected_by_tag": false,
            "rhel_enabled_repos_found": false,
            "rhel_product_certs_found": false,
            "rhel_release_files_found": false,
            "rhel_signed_packages_found": false,
            "rhel_version": null,
            "status": "inspecting",
            "syspurpose": null,
            "updated_at": "2020-08-26T08:11:09.968768Z"
        }
    ],
    "links": {
        "first": "/api/cloudigrade/api/cloudigrade/v2/images/?limit=10&offset=0",
        "last": "/api/cloudigrade/api/cloudigrade/v2/images/?limit=10&offset=0",
        "next": null,
        "previous": null
    },
    "meta": {
        "count": 1
    }
}

Image Description

 % aws ec2 describe-images --image-ids ami-065d0be9883aa064a

{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2018-03-02T17:13:01.000Z",
            "ImageId": "ami-065d0be9883aa064a",
            "ImageLocation": "518028203513/import-ami-05c88748fa241d139",
            "ImageType": "machine",
            "Public": false,
            "OwnerId": "518028203513",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": false,
                        "SnapshotId": "snap-0dcb82dfaeb5b349a",
                        "VolumeSize": 8,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                }
            ],
            "Description": "RHEL7 XFS",
            "Hypervisor": "xen",
            "Name": "import-ami-05c88748fa241d139",
            "RootDeviceName": "/dev/sda1",
            "RootDeviceType": "ebs",
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "RHEL7_XFS"
                }
            ],
            "VirtualizationType": "hvm"
        }
    ]
}

In GitLab by @pakamble on Sep 4, 2020, 09:33

assigned to @pakamble

In GitLab by @pakamble on Sep 8, 2020, 05:31

I have verified image inspection process by taking images from different sources like marketplace, community and owned and all are successfully got inspected. Find the attached logs for more details. images.json

cloudigrade / houndigrade