search

Seagate / cortx-monitor

CORTX Monitor tracks platform health and raises alerts on sensing any unintended state. It can detect hardware faults/ removal/ replacement by continuously sensing sub-systems like Storage Enclosure, Node Servers and components, network interfaces.
https://github.com/Seagate/cortx
GNU Affero General Public License v3.0
4 stars 38 forks source link

Sanity Test doesn't give the expected output #500

Closed daniarherikurniawan closed 3 years ago

daniarherikurniawan commented 3 years ago

I am following the most updated Readme here https://github.com/Seagate/cortx-monitor/blob/thavanathan-patch-1/cortx-monitorQuickstartGuide.md. I am using VM running on VMware Workstation installed in Seagate's laptop (has access to Seagate network).

When following the last step (running sanity test: /opt/seagate/cortx/sspl/bin/sspl_setup test --config yaml:///etc/sspl_global_config_copy.yaml --plan dev_sanity). I got this following output:

[root@cmu cortx-monitor]#  /opt/seagate/cortx/sspl/bin/sspl_setup test --config yaml:///etc/sspl_global_config_copy.yaml --plan dev_sanity
Aug 12 15:22:35 sspl-setup[40334]: INFO test - Init done (sspl_setup:296)
Aug 12 15:22:36 sspl-setup[40334]: INFO test - Validation done (sspl_setup:340)
Aug 12 15:22:36 sspl-setup[40334]: INFO Ensured Package Dependency: Flask(v1.1.1). (sspl_test.py:94)
Starting mock server on 127.0.0.1:28200
 * Serving Flask app "mock_server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
Stoping the SSPL service
Code Coverage enabled : False
Starting the SSPL service
Waiting for SSPL to complete initialization of all the plugins..
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "16287817719cb64376cd4f403ca52baf37b9777c5d", "severity": "critical", "info": {"resource_type": "enclosure:hw:psu", "fru": "true:cold_swappable", "event_time": "1628781771", "resource_id": "psu_0.0", "description": "Unable to obtain FRU ID data for this FRU.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"enclosure_id": 0, "serial_number": "NA", "description": "NA", "revision": "NA", "model": "NA", "vendor": "NA", "location": "Enclosure 0 - Left", "part_number": "NA", "fru_shortname": "NA", "mfg_date": "NA", "mfg_vendor_id": "NA", "dc12v": 0, "dc5v": 0, "dc33v": 0, "dc12i": 0, "dc5i": 0, "dctemp": 0, "health": "Degraded", "health_reason": "Unable to obtain FRU ID data for this FRU.", "health_recommendation": "- The FRU ID content was not read. This could be temporary or it may not programmed correctly.     - If this persists, contact technical support.", "status": "Up", "durable_id": "psu_0.0", "position": "Left"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1628781771c5f440abb4e74569b38ee1d046f5cb01", "severity": "critical", "info": {"resource_type": "enclosure:hw:fan", "fru": "true:cold_swappable", "event_time": "1628781771", "resource_id": "Fan Module 4", "description": "A subcomponent of this component is unhealthy.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"durable_id": "fan_module_0.4", "enclosure_id": 0, "health": "Degraded", "health_reason": "A subcomponent of this component is unhealthy.", "health_recommendation": "- See the health information for the unhealthy subcomponents of this component.", "location": "Enclosure 0 - Right", "name": "Fan Module 4", "position": "Indexed", "status": "OK", "fans": [{"durable_id": "fan_0.fm4.1", "fw_revision": "", "health": "OK", "health_reason": "", "health_recommendation": "", "hw_revision": "", "location": "Enclosure 0, Fan Module 4", "locator_led": "Off", "name": "Fan 1", "part_number": "NA", "position": "NA", "serial_number": "NA", "speed": 6480, "status": "Up"}, {"durable_id": "fan_0.fm4.1", "fw_revision": "", "health": "OK", "health_reason": "", "health_recommendation": "", "hw_revision": "", "location": "Enclosure 0, Fan Module 4", "locator_led": "Off", "name": "Fan 1", "part_number": "NA", "position": "NA", "serial_number": "NA", "speed": 6480, "status": "Up"}]}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "16287817711ad9847645a14983bbd9750c56f4f19f", "severity": "critical", "info": {"resource_type": "enclosure:hw:controller", "fru": "true:hot_swappable", "event_time": "1628781771", "resource_id": "controller_b", "description": "A subcomponent of this component is unhealthy.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"active_version": 4, "cache_lock": "Yes", "cache_lock_numeric": 1, "cache_memory_size": 4096, "compact_flash": [{"cache_flush": "Enabled", "cache_flush_numeric": 1, "controller_id": "B", "controller_id_numeric": 0, "durable_id": "Ctlr B CF", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Controller B CompactFlash", "object_name": "controller-b-compact-flash", "status": "Installed", "status_numeric": 1}], "controller_id": "B", "controller_id_numeric": 0, "cpld_version": "3.9", "dash_level": "", "description": "NA", "disks": 84, "drive_bus_type": "SAS", "drive_bus_type_numeric": 8, "drive_channels": 2, "durable_id": "controller_b", "expander_ports": [{"controller": "B", "controller_numeric": 0, "durable_id": "expport_universal_0.B0", "enclosure_id": 0, "health": "NA", "health_numeric": 4, "health_reason": "No drive enclosure is connected to this expansion port. This is normal if this is the last (or only) enclosure in the cabling sequence of the system.", "health_recommendation": "- No action is required.", "name": "Expansion Port", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Expansion Port Universal", "sas_port_type_numeric": 5, "status": "Disconnected", "status_numeric": 6}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B0", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B0", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B1", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B1", "object_name": "expander-port", "sas_port_index": 1, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B2", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B2", "object_name": "expander-port", "sas_port_index": 2, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B0", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B0", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B1", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B1", "object_name": "expander-port", "sas_port_index": 1, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B2", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B2", "object_name": "expander-port", "sas_port_index": 2, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}], "expanders": [{"dom_id": 0, "drawer_id": 255, "durable_id": "expander_0.B", "enclosure_id": 0, "extended_status": "00000000", "fw_revision": "5140", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "location": "Enclosure 0, Iom B", "name": "Root Expander-B", "object_name": "expanders", "path_id": "B", "path_id_numeric": 0, "status": "OK", "status_numeric": 1}], "fail_over_reason": "Not applicable", "fail_over_reason_numeric": 0, "failed_over": "No", "failed_over_numeric": 0, "fru_shortname": "", "hardware_version": "1.1", "health": "Degraded", "health_numeric": 1, "health_reason": "A subcomponent of this component is unhealthy.", "health_recommendation": "- See the health information for the unhealthy subcomponents of this component.", "host_ports": 2, "internal_serial_number": "00C0FF3C63BA", "ip_address": "10.230.162.148", "ip_gateway": "10.230.160.1", "ip_subnet_mask": "255.255.248.0", "locator_led": "Off", "locator_led_numeric": 0, "mac_address": "00:C0:FF:3C:63:BA", "mfg_date": "2007-01-03 16:24:00", "mfg_date_numeric": 1167841440, "mfg_location": "", "mfg_vendor_id": "", "model": "3865", "multicore": "Enabled", "multicore_numeric": 1, "network_parameters": [{"active_version": 4, "addressing_mode": "DHCP", "addressing_mode_numeric": 2, "auto_negotiation": "Enabled", "auto_negotiation_numeric": 1, "duplex_mode": "full", "duplex_mode_numeric": 0, "durable_id": "mgmtport_b", "gateway": "10.230.160.1", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "ip_address": "10.230.162.148", "link_speed": "1000mbps", "link_speed_numeric": 2, "mac_address": "00:c0:ff:3c:63:ba", "object_name": "controller-b", "ping_broadcast": "Disabled", "ping_broadcast_numeric": 0, "subnet_mask": "255.255.248.0"}], "node_wwn": "208000C0FFA7117A", "number_of_storage_pools": 0, "object_name": "controllers", "part_number": "81-00000117-00-15", "phy_isolation": "Enabled", "phy_isolation_numeric": 0, "platform_type": "Gallium3 EX", "platform_type_numeric": 7, "port": [{"actual_speed": "8Gb", "actual_speed_numeric": 7, "configured_speed": "Auto", "configured_speed_numeric": 3, "controller": "B", "controller_numeric": 0, "durable_id": "hostport_B0", "fan_out": 1, "fc_port": [{"configured_topology": "PTP", "configured_topology_numeric": 1, "object_name": "port-details", "primary_loop_id": "NA", "sfp_part_number": "FTLF8529P3BCV-1D", "sfp_present": "Present", "sfp_present_numeric": 1, "sfp_revision": "A", "sfp_status": "OK", "sfp_status_numeric": 3, "sfp_supported_speeds": "4G,8G,16G", "sfp_supported_speeds_numeric": 112, "sfp_vendor": "DHS"}], "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "media": "FC(P)", "object_name": "ports", "port": "B0", "port_type": "FC", "port_type_numeric": 6, "status": "Up", "status_numeric": 0, "target_id": "247000c0ffa7117a"}, {"actual_speed": "8Gb", "actual_speed_numeric": 7, "configured_speed": "Auto", "configured_speed_numeric": 3, "controller": "B", "controller_numeric": 0, "durable_id": "hostport_B1", "fan_out": 1, "fc_port": [{"configured_topology": "PTP", "configured_topology_numeric": 1, "object_name": "port-details", "primary_loop_id": "NA", "sfp_part_number": "FTLF8529P3BCV-1D", "sfp_present": "Present", "sfp_present_numeric": 1, "sfp_revision": "A", "sfp_status": "OK", "sfp_status_numeric": 3, "sfp_supported_speeds": "4G,8G,16G", "sfp_supported_speeds_numeric": 112, "sfp_vendor": "DHS"}], "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "media": "FC(P)", "object_name": "ports", "port": "B1", "port_type": "FC", "port_type_numeric": 6, "status": "Up", "status_numeric": 0, "target_id": "257000c0ffa7117a"}], "position": "Right", "position_numeric": 1, "redundancy_mode": "Active-Active ULP", "redundancy_mode_numeric": 8, "redundancy_status": "Redundant", "redundancy_status_numeric": 2, "revision": "A", "rotation": "0 Degrees", "rotation_numeric": 0, "sc_cpu_speed": 1200, "sc_cpu_type": "Broadwell", "sc_fw": "GTS265R18-01", "serial_number": "DHSIFTJ-18253C63BA", "ssd_alt_path_io_count": 2, "status": "Operational", "status_numeric": 0, "system_memory_size": 8192, "vendor": "", "virtual_disks": 2, "write_policy": "write-back", "write_policy_numeric": 0}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "missing", "alert_id": "1628781771cefe655e77774f9c8e42db7496810689", "host_id": "cmu", "severity": "critical", "info": {"resource_type": "enclosure:hw:sideplane", "fru": "true:cold_swappable", "event_time": "1628781771", "resource_id": "drawer 0 Left Sideplane", "description": "", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"drawer_id": 0, "durable_id": "sideplane_0.D0.B", "enclosure_id": 0, "health": "Fault", "health_reason": "", "health_recommendation": "Replace the enclosure. Contact technical support.", "location": "enclosure 0, drawer 0", "name": "Left Sideplane", "position": "Top", "status": "Critical", "unhealthy_components": [{"basetype": "sensor", "component_id": "Sideplane 0.D0.B, 24-port Expander, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.0"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.B, Ambient Temp, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.2"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.B, 36-port Expander, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.1"}]}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1628781771e20de526132642d1bc220cd6aa50e2f9", "severity": "critical", "info": {"resource_type": "enclosure:hw:disk", "fru": "true:hot_swappable", "event_time": "1628781771", "resource_id": "disk_00.65", "description": "Disk failure is imminent.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"enclosure_id": 0, "enclosure_wwn": "500C0FF0CFE4933C", "slot": 65, "description": "SAS MDL", "architecture": "HDD", "interface": "SAS", "serial_number": "Z290KXYM000091318WQQ", "size": "3000.5GB", "vendor": "SEAGATE", "model": "ST33000650SS", "revision": "XRR6", "temperature": "29 C", "status": "Up", "LED_status": "NA", "locator_LED": "NA", "blink": 0, "smart": "Enabled", "health": "Fault", "health_reason": "Disk failure is imminent.", "health_recommendation": "- Replace the disk with one of the same type (SSD, enterprise SAS, or midline SAS) and the same or greater capacity. For continued optimum I/O performance, the replacement disk should have performance that is the same as or better than the one it is replacing.", "enclosure_family": "Realstor Storage_enclosure", "attributes": " ", "avg_rsp_time": 0, "blocks": 5860533168, "blocksize": 512, "container_index": 0, "copyback_state": "NA", "current_job_completion": "NA", "disk_dsd_count": 4, "disk_group": "NA", "drawer_id": 1, "drive_down_code": 2, "dual_port": 1, "durable_id": "disk_00.65", "enclosure_url": "/enclosures/0", "error": 1, "extended_status": 512, "fc_p1_channel": 1, "fc_p1_device_id": 65, "fc_p1_node_wwn": "5000C500347D7F9F", "fc_p1_port_wwn": "0000000000000000", "fc_p1_unit_number": 0, "fc_p2_channel": 0, "fc_p2_device_id": 0, "fc_p2_node_wwn": "NA", "fc_p2_port_wwn": "NA", "fc_p2_unit_number": 0, "fde_config_time": "NA", "fde_state": "Not FDE Capable", "import_lock_key_id": "00000000", "index": 30, "job_running": "NA", "led_status": " Online", "location": "0.65", "locator_led": "Off", "lock_key_id": "00000000", "member_index": 0, "meta": "/meta/drives", "number_of_ios": 0, "object_name": "drive", "owner": "A", "pi_formatted": "No", "port": 1, "power_on_hours": 0, "recon_state": "NA", "rpm": 7, "scsi_id": 65, "secondary_channel": 0, "sector_format": "512n", "single_ported": "Disabled", "speed": 0, "spun_down": 0, "ssd_life_left": "NA", "state": "FAILED", "storage_pool_name": "NA", "storage_tier": "Archive", "temperature_status": "Unavailable", "total_data_transferred": "0B", "transfer_rate": "6.0", "type": "SAS MDL", "url": "/drives/0.65", "usage": "FAILED", "virtual_disk_serial": "NA"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"disk_status_drivemanager": {"enclosureSN": "HPI_Data_Not_Available", "diskNum": -1, "diskStatus": "OK", "diskReason": "None", "serialNumber": "00000000000000000001", "pathID": "/dev/disk/by-id/wwn-0x5000c29fd27c73d3-part1"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "insertion", "alert_id": "16287817726f1c7b9e01f04ae28eaf6b832248d0e0", "severity": "informational", "info": {"resource_type": "node:os:disk", "resource_id": "VMware-Virtual-IDE-Hard-Drive-00000000000000000001", "event_time": "1628781772", "description": "Server drive is inserted.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"vendor": "", "model": "VMware Virtual IDE Hard Drive", "revision": "00000001", "serial": "00000000000000000001", "wwn": "0x5000c29fd27c73d3", "id": "VMware-Virtual-IDE-Hard-Drive-00000000000000000001", "configuration": {}, "media": "", "mediacompatibility": [], "mediaremovable": 0, "mediaavailable": 1, "mediachangedetected": 1, "size": 42949672960, "timedetected": 1628706285476214, "timemediadetected": 1628706285476214, "optical": 0, "opticalblank": 0, "opticalnumtracks": 0, "opticalnumaudiotracks": 0, "opticalnumdatatracks": 0, "opticalnumsessions": 0, "rotationrate": -1, "connectionbus": "", "seat": "seat0", "removable": 0, "ejectable": 0, "sortkey": "00coldplug/00fixed/sd____a", "canpoweroff": 0, "siblingid": ""}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "16287817720b57a57ae0ae49dca43a7ae50bb5ca1e", "severity": "critical", "info": {"resource_type": "node:os:disk", "resource_id": "VMware-Virtual-IDE-Hard-Drive-00000000000000000001", "event_time": "1628781772", "description": "Fault detected for server drive.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"health_status": "smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-1160.11.1.el7.x86_64] (local build)\\nCopyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org\\n\\nSMART support is: Unavailable - device lacks SMART capability.\\nA mandatory SMART command failed: exiting. To continue, add one or more \'-T permissive\' options."}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault_resolved", "alert_id": "1582898505e4e1baf21e574cd39595e9390bcc24b2", "severity": "informational", "info": {"resource_type": "node:hw:fan", "fru": "false", "resource_id": "Fan Redundancy", "event_time": "1582898505", "description": "\'Fan Redundancy Sensor\' has recovered from a failure. [Fan #0x0c]", "impact": "None", "recommendation": "None", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"fru_id": "Fan #0x0c", "event": "Asserted - Fully Redundant", "States Asserted": "Fan\\n                         [Fully Redundant]"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "missing", "alert_id": "16287817762ee35c15b25a4740989bc9b6176ca01e", "host_id": "cmu", "severity": "critical", "info": {"resource_type": "enclosure:hw:sideplane", "fru": "true:cold_swappable", "event_time": "1628781776", "resource_id": "drawer 0 Right Sideplane", "description": "", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"drawer_id": 0, "durable_id": "sideplane_0.D0.A", "enclosure_id": 0, "health": "Fault", "health_reason": "", "health_recommendation": "Replace the enclosure. Contact technical support.", "location": "enclosure 0, drawer 0", "name": "Right Sideplane", "position": "Top", "status": "Critical", "unhealthy_components": [{"basetype": "sensor", "component_id": "Sideplane 0.D0.A, 36-port Expander, Right Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.A.1"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.A, 24-port Expander, Right Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.A.0"}]}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "15829038558e4dfe6ae8c74234b02a2ed83231b5a0", "severity": "critical", "info": {"resource_type": "node:hw:psu", "fru": "true:hot_swappable", "resource_id": "Power Supply #0x51", "event_time": "1582903855", "description": "\'Power Supply 2\' lost the AC supply. [PS2 (0x51)]", "impact": "Power Supply Redundancy is affected. If more power supplies go down or missing, server may go offline.", "recommendation": "Check power supply cable connectivity. If power cable is connected, cable might be faulty, please check & replace.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"fru_id": "Power Supply #0x51", "Sensor ID": "PS2 Status (0x51)", "Entity ID": "10.2 (Power Supply)", "Sensor Type (Discrete)": "Power Supply (0x08)", "Event Message Control": "Per-threshold", "Assertion Events": "Power Supply, Presence detected, Power Supply AC lost", "Assertions Enabled": "Power Supply, Presence detected, Failure detected, Predictive failure, Power Supply AC lost, Config Error, Power Supply Inactive", "Deassertions Enabled": "Power Supply, Presence detected, Failure detected, Predictive failure, Power Supply AC lost, Config Error, Power Supply Inactive", "OEM": "0", "States Asserted": "Power Supply\\n                         [Presence detected]\\n                         [Power Supply AC lost]", "Sensor Reading": "0h"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1628781776d855b248792f42c9a2f939f924be0a9e", "severity": "critical", "info": {"resource_type": "enclosure:hw:disk", "fru": "true:hot_swappable", "event_time": "1628781776", "resource_id": "disk_00.81", "description": "A disk that was previously a member of a disk group has been detected.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"enclosure_id": 0, "enclosure_wwn": "500C0FF0CFE4933C", "slot": 81, "description": "SAS MDL", "architecture": "HDD", "interface": "SAS", "serial_number": "ZAD01M280000R634H1KV", "size": "6001.1GB", "vendor": "SEAGATE", "model": "ST6000NM0095", "revision": "E002", "temperature": "0 C", "status": "Up", "LED_status": "NA", "locator_LED": "NA", "blink": 0, "smart": "Enabled", "health": "Degraded", "health_reason": "A disk that was previously a member of a disk group has been detected.", "health_recommendation": "- If the associated disk group is offline or quarantined, contact technical support. Otherwise, clear the disk\'s metadata to reuse the disk.", "enclosure_family": "Realstor Storage_enclosure", "attributes": " ", "avg_rsp_time": 0, "blocks": 11721045168, "blocksize": 512, "container_index": 0, "copyback_state": "NA", "current_job_completion": "NA", "disk_dsd_count": 4, "disk_group": "NA", "drawer_id": 1, "drive_down_code": 7, "dual_port": 1, "durable_id": "disk_00.81", "enclosure_url": "/enclosures/0", "error": 0, "extended_status": 128, "fc_p1_channel": 1, "fc_p1_device_id": 81, "fc_p1_node_wwn": "5000C500856DCF6B", "fc_p1_port_wwn": "0000000000000000", "fc_p1_unit_number": 0, "fc_p2_channel": 0, "fc_p2_device_id": 0, "fc_p2_node_wwn": "NA", "fc_p2_port_wwn": "NA", "fc_p2_unit_number": 0, "fde_config_time": "NA", "fde_state": "Not FDE Capable", "import_lock_key_id": "00000000", "index": 43, "job_running": "NA", "led_status": " Online", "location": "0.81", "locator_led": "Off", "lock_key_id": "00000000", "member_index": 0, "meta": "/meta/drives", "number_of_ios": 0, "object_name": "drive", "owner": "A", "pi_formatted": "No", "port": 1, "power_on_hours": 0, "recon_state": "NA", "rpm": 7, "scsi_id": 81, "secondary_channel": 0, "sector_format": "512e", "single_ported": "Disabled", "speed": 0, "spun_down": 0, "ssd_life_left": "NA", "state": "LEFTOVR", "storage_pool_name": "NA", "storage_tier": "Archive", "temperature_status": "Unavailable", "total_data_transferred": "0B", "transfer_rate": "12.0", "type": "SAS MDL", "url": "/drives/0.81", "usage": "LEFTOVR", "virtual_disk_serial": "NA"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "insertion", "alert_id": "1582907502d96fed52fc584da381405a1ba263ee8d", "severity": "informational", "info": {"resource_type": "node:hw:disk", "fru": "true:hot_swappable", "resource_id": "HDD 0 (0xf0)", "event_time": "1582907502", "description": "Disk is inserted in slot \'0\'. [0]", "impact": "None", "recommendation": "None", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"Sensor Type (Discrete)": "Drive Slot / Bay", "fru_id": "HDD 0 (0xf0)", "States Asserted": "Drive Slot / Bay, Drive Present"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1582909521ab7e466600b24f95a9b93f0f359ff119", "severity": "critical", "info": {"resource_type": "node:hw:psu", "fru": "true:hot_swappable", "resource_id": "Power Unit #0x02", "event_time": "1582909521", "description": "Power off/down", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"fru_id": "Power Unit #0x02", "Sensor ID": "Pwr Unit Redund (0x2)", "Entity ID": "21.1 (Power Management)", "Sensor Type (Discrete)": "Power Unit (0x09)", "Event Message Control": "Per-threshold", "Assertion Events": "Power Unit, Redundancy Lost", "Assertions Enabled": "Power Unit, Fully Redundant, Redundancy Lost, Redundancy Degraded, Non-Redundant: Sufficient from Redundant, Non-Redundant: Sufficient from Insufficient, Non-Redundant: Insufficient Resources, Redundancy Degraded from Fully Redundant, Redundancy Degraded from Non-Redundant", "Deassertions Enabled": "Power Unit, Fully Redundant, Redundancy Lost, Redundancy Degraded, Non-Redundant: Sufficient from Redundant, Non-Redundant: Sufficient from Insufficient, Non-Redundant: Insufficient Resources, Redundancy Degraded from Fully Redundant, Redundancy Degraded from Non-Redundant", "OEM": "0", "States Asserted": "Power Unit\\n                         [Redundancy Lost]", "Sensor Reading": "0h"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1628781781fe6548602a9f42deb0a3a6672e9e90d6", "severity": "critical", "info": {"resource_type": "enclosure:hw:disk", "fru": "true:hot_swappable", "event_time": "1628781781", "resource_id": "disk_00.82", "description": "The disk may contain invalid metadata.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"enclosure_id": 0, "enclosure_wwn": "500C0FF0CFE4933C", "slot": 82, "description": "SAS MDL", "architecture": "HDD", "interface": "SAS", "serial_number": "ZAD01MGM0000R6348P9L", "size": "6001.1GB", "vendor": "SEAGATE", "model": "ST6000NM0095", "revision": "E002", "temperature": "0 C", "status": "Up", "LED_status": "NA", "locator_LED": "NA", "blink": 0, "smart": "Enabled", "health": "Degraded", "health_reason": "The disk may contain invalid metadata.", "health_recommendation": "- If the associated disk group is offline or quarantined, contact technical support. Otherwise, clear the disk\'s metadata to reuse the disk.", "enclosure_family": "Realstor Storage_enclosure", "attributes": " ", "avg_rsp_time": 0, "blocks": 11721045168, "blocksize": 512, "container_index": 0, "copyback_state": "NA", "current_job_completion": "NA", "disk_dsd_count": 4, "disk_group": "NA", "drawer_id": 1, "drive_down_code": 0, "dual_port": 1, "durable_id": "disk_00.82", "enclosure_url": "/enclosures/0", "error": 0, "extended_status": 524288, "fc_p1_channel": 1, "fc_p1_device_id": 82, "fc_p1_node_wwn": "5000C500856DC1A7", "fc_p1_port_wwn": "0000000000000000", "fc_p1_unit_number": 0, "fc_p2_channel": 0, "fc_p2_device_id": 0, "fc_p2_node_wwn": "NA", "fc_p2_port_wwn": "NA", "fc_p2_unit_number": 0, "fde_config_time": "NA", "fde_state": "Not FDE Capable", "import_lock_key_id": "00000000", "index": 44, "job_running": "NA", "led_status": " Online", "location": "0.82", "locator_led": "Off", "lock_key_id": "00000000", "member_index": 0, "meta": "/meta/drives", "number_of_ios": 0, "object_name": "drive", "owner": "A", "pi_formatted": "No", "port": 1, "power_on_hours": 0, "recon_state": "NA", "rpm": 7, "scsi_id": 82, "secondary_channel": 0, "sector_format": "512e", "single_ported": "Disabled", "speed": 0, "spun_down": 0, "ssd_life_left": "NA", "state": "LEFTOVR", "storage_pool_name": "NA", "storage_tier": "Archive", "temperature_status": "Unavailable", "total_data_transferred": "0B", "transfer_rate": "12.0", "type": "SAS MDL", "url": "/drives/0.82", "usage": "LEFTOVR", "virtual_disk_serial": "NA"}}}}'
'actuator_response_type'
b'{"username": "sspl-ll", "description": "Seagate Storage Platform Library - Sensor Response", "title": "SSPL Sensor Response", "expires": 3600, "signature": "None", "time": "1628781784", "message": {"sspl_ll_msg_header": {"msg_version": "1.0.0", "schema_version": "1.0.0", "sspl_version": "2.0.0"}, "sensor_response_type": {"alert_type": "fault", "severity": "warning", "alert_id": "1628781781936707e91ec64716a55aae43d94ded5f", "host_id": "cmu", "info": {"resource_type": "node:os:disk_space", "resource_id": "0", "event_time": "1628781781", "description": "Disk usage has increased to 18.4%, beyond the configured threshold of 16%.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"freeSpace": {"value": 42927, "units": "MB"}, "totalSpace": {"value": 35033, "units": "MB"}, "diskUsedPercentage": 18.4}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781784", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault_resolved", "alert_id": "162878177662efc33cbb764387ad9ffd2e8320d307", "severity": "informational", "info": {"resource_type": "node:hw:psu", "fru": "true:hot_swappable", "resource_id": "Power Supply #0x51", "event_time": "1628781776", "description": "\'Power Supply 2\' regained the AC Supply. [PS2 (0x51)]", "impact": "Power Supply regained the healthy state.", "recommendation": "None", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"fru_id": "Power Supply #0x51", "Sensor ID": "PS2 Status (0x51)", "Entity ID": "10.2 (Power Supply)", "Sensor Type (Discrete)": "Power Supply (0x08)", "Event Message Control": "Per-threshold", "Assertion Events": "Power Supply, Presence detected, Power Supply AC lost", "Assertions Enabled": "Power Supply, Presence detected, Failure detected, Predictive failure, Power Supply AC lost, Config Error, Power Supply Inactive", "Deassertions Enabled": "Power Supply, Presence detected, Failure detected, Predictive failure, Power Supply AC lost, Config Error, Power Supply Inactive", "OEM": "0", "States Asserted": "Power Supply\\n                         [Presence detected]\\n                         [Power Supply AC lost]", "Sensor Reading": "0h"}}}}'
'actuator_response_type'
b'{"username": "sspl-ll", "description": "Seagate Storage Platform Library - Sensor Response", "title": "SSPL Sensor Response", "expires": 3600, "signature": "None", "time": "1628781794", "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "fault", "severity": "warning", "alert_id": "1628781793db7a103da9c246bfa91be7aacab83ef4", "host_id": "cmu", "info": {"resource_type": "node:os:cpu_usage", "resource_id": "0", "event_time": "1628781793", "description": "CPU usage has increased to 2.0%, beyond the configured threshold of 0% for more than 10 seconds.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"localtime": "1628781769", "csps": 0, "idleTime": 95, "interruptTime": 0, "iowaitTime": 0, "niceTime": 0, "softirqTime": 0, "stealTime": 0, "systemTime": 2, "userTime": 2, "coreData": [{"coreId": 0, "load1MinAvg": 3, "load5MinAvg": 8, "load15MinAvg": 3, "ips": 0}, {"coreId": 1, "load1MinAvg": 2, "load5MinAvg": 7, "load15MinAvg": 3, "ips": 0}, {"coreId": 2, "load1MinAvg": 2, "load5MinAvg": 7, "load15MinAvg": 3, "ips": 0}, {"coreId": 3, "load1MinAvg": 2, "load5MinAvg": 8, "load15MinAvg": 4, "ips": 0}], "cpu_usage": 2.0}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781802", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "severity": "critical", "alert_id": "1628781801bae0ff95e7e0478f96613061c1e4e16d", "alert_type": "fault", "info": {"resource_type": "node:sw:os:service", "resource_id": "multipathd.service", "event_time": "1628781801", "description": "multipathd.service in inactive state for more than 30 seconds.", "impact": "multipathd.service service is unavailable.", "recommendation": "Try to restart the service", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"service_name": "multipathd.service", "previous_state": "NA", "state": "inactive", "previous_substate": "NA", "substate": "dead", "previous_pid": "NA", "pid": "0"}}}}'
'actuator_response_type'
b'{"username": "sspl-ll", "description": "Seagate Storage Platform Library - Sensor Response", "title": "SSPL Sensor Response", "expires": 3600, "signature": "None", "time": "1628781805", "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "fault", "severity": "warning", "alert_id": "16287818041a9038abe008410384c1f0b294dbc472", "host_id": "cmu", "info": {"resource_type": "node:os:memory_usage", "resource_id": "0", "event_time": "1628781804", "description": "Host memory usage has increased to 27.2%,beyond the configured threshold of 0% for more than 20 seconds.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01", "fru": "false"}, "specific_info": {"localtime": "1628781769", "bootTime": "1628781770", "upTime": 1628706285, "uname": {"sysname": "Linux", "nodename": "cmu", "version": "3.10.0-1160.11.1.el7.x86_64", "release": "#1 SMP Fri Dec 18 16:34:56 UTC 2020", "machine": "x86_64"}, "totalMemory": {"total": 5919002624, "available": 4311072768, "percent": 27.2, "used": 1299886080, "free": 1409916928, "active": 1500254208, "inactive": 2230972416, "buffers": 1060864, "cached": 3208138752, "shared": 3792896, "slab": 426532864}, "loggedInUsers": [{"name": "root", "terminal": "tty1", "host": "", "started": 1628706304.0, "pid": 1260}, {"name": "root", "terminal": "pts/0", "host": "192.168.187.1", "started": 1628707328.0, "pid": 2147}], "processCount": 145, "runningProcessCount": 0}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781806", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "16287818065dbcacc9558f48c2b821ef0d4f2acbe7", "severity": "critical", "info": {"resource_type": "enclosure:hw:controller", "fru": "true:hot_swappable", "event_time": "1628781806", "resource_id": "controller_b", "description": "A subcomponent of this component is unhealthy.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"active_version": 4, "cache_lock": "Yes", "cache_lock_numeric": 1, "cache_memory_size": 4096, "compact_flash": [{"cache_flush": "Enabled", "cache_flush_numeric": 1, "controller_id": "B", "controller_id_numeric": 0, "durable_id": "Ctlr B CF", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Controller B CompactFlash", "object_name": "controller-b-compact-flash", "status": "Installed", "status_numeric": 1}], "controller_id": "B", "controller_id_numeric": 0, "cpld_version": "3.9", "dash_level": "", "description": "NA", "disks": 84, "drive_bus_type": "SAS", "drive_bus_type_numeric": 8, "drive_channels": 2, "durable_id": "controller_b", "expander_ports": [{"controller": "B", "controller_numeric": 0, "durable_id": "expport_universal_0.B0", "enclosure_id": 0, "health": "NA", "health_numeric": 4, "health_reason": "No drive enclosure is connected to this expansion port. This is normal if this is the last (or only) enclosure in the cabling sequence of the system.", "health_recommendation": "- No action is required.", "name": "Expansion Port", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Expansion Port Universal", "sas_port_type_numeric": 5, "status": "Disconnected", "status_numeric": 6}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B0", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B0", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B1", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B1", "object_name": "expander-port", "sas_port_index": 1, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D0.B2", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer0-egress B2", "object_name": "expander-port", "sas_port_index": 2, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B0", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B0", "object_name": "expander-port", "sas_port_index": 0, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B1", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B1", "object_name": "expander-port", "sas_port_index": 1, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}, {"controller": "B", "controller_numeric": 0, "durable_id": "drawer_egress_0.D1.B2", "enclosure_id": 0, "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "name": "Drawer1-egress B2", "object_name": "expander-port", "sas_port_index": 2, "sas_port_type": "Drawer Port Egress", "sas_port_type_numeric": 1, "status": "Up", "status_numeric": 0}], "expanders": [{"dom_id": 0, "drawer_id": 255, "durable_id": "expander_0.B", "enclosure_id": 0, "extended_status": "00000000", "fw_revision": "5140", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "location": "Enclosure 0, Iom B", "name": "Root Expander-B", "object_name": "expanders", "path_id": "B", "path_id_numeric": 0, "status": "OK", "status_numeric": 1}], "fail_over_reason": "Not applicable", "fail_over_reason_numeric": 0, "failed_over": "No", "failed_over_numeric": 0, "fru_shortname": "", "hardware_version": "1.1", "health": "Degraded", "health_numeric": 1, "health_reason": "A subcomponent of this component is unhealthy.", "health_recommendation": "- See the health information for the unhealthy subcomponents of this component.", "host_ports": 2, "internal_serial_number": "00C0FF3C63BA", "ip_address": "10.230.162.148", "ip_gateway": "10.230.160.1", "ip_subnet_mask": "255.255.248.0", "locator_led": "Off", "locator_led_numeric": 0, "mac_address": "00:C0:FF:3C:63:BA", "mfg_date": "2007-01-03 16:24:00", "mfg_date_numeric": 1167841440, "mfg_location": "", "mfg_vendor_id": "", "model": "3865", "multicore": "Enabled", "multicore_numeric": 1, "network_parameters": [{"active_version": 4, "addressing_mode": "DHCP", "addressing_mode_numeric": 2, "auto_negotiation": "Enabled", "auto_negotiation_numeric": 1, "duplex_mode": "full", "duplex_mode_numeric": 0, "durable_id": "mgmtport_b", "gateway": "10.230.160.1", "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "ip_address": "10.230.162.148", "link_speed": "1000mbps", "link_speed_numeric": 2, "mac_address": "00:c0:ff:3c:63:ba", "object_name": "controller-b", "ping_broadcast": "Disabled", "ping_broadcast_numeric": 0, "subnet_mask": "255.255.248.0"}], "node_wwn": "208000C0FFA7117A", "number_of_storage_pools": 0, "object_name": "controllers", "part_number": "81-00000117-00-15", "phy_isolation": "Enabled", "phy_isolation_numeric": 0, "platform_type": "Gallium3 EX", "platform_type_numeric": 7, "port": [{"actual_speed": "8Gb", "actual_speed_numeric": 7, "configured_speed": "Auto", "configured_speed_numeric": 3, "controller": "B", "controller_numeric": 0, "durable_id": "hostport_B0", "fan_out": 1, "fc_port": [{"configured_topology": "PTP", "configured_topology_numeric": 1, "object_name": "port-details", "primary_loop_id": "NA", "sfp_part_number": "FTLF8529P3BCV-1D", "sfp_present": "Present", "sfp_present_numeric": 1, "sfp_revision": "A", "sfp_status": "OK", "sfp_status_numeric": 3, "sfp_supported_speeds": "4G,8G,16G", "sfp_supported_speeds_numeric": 112, "sfp_vendor": "DHS"}], "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "media": "FC(P)", "object_name": "ports", "port": "B0", "port_type": "FC", "port_type_numeric": 6, "status": "Up", "status_numeric": 0, "target_id": "247000c0ffa7117a"}, {"actual_speed": "8Gb", "actual_speed_numeric": 7, "configured_speed": "Auto", "configured_speed_numeric": 3, "controller": "B", "controller_numeric": 0, "durable_id": "hostport_B1", "fan_out": 1, "fc_port": [{"configured_topology": "PTP", "configured_topology_numeric": 1, "object_name": "port-details", "primary_loop_id": "NA", "sfp_part_number": "FTLF8529P3BCV-1D", "sfp_present": "Present", "sfp_present_numeric": 1, "sfp_revision": "A", "sfp_status": "OK", "sfp_status_numeric": 3, "sfp_supported_speeds": "4G,8G,16G", "sfp_supported_speeds_numeric": 112, "sfp_vendor": "DHS"}], "health": "OK", "health_numeric": 0, "health_reason": "", "health_recommendation": "", "media": "FC(P)", "object_name": "ports", "port": "B1", "port_type": "FC", "port_type_numeric": 6, "status": "Up", "status_numeric": 0, "target_id": "257000c0ffa7117a"}], "position": "Right", "position_numeric": 1, "redundancy_mode": "Active-Active ULP", "redundancy_mode_numeric": 8, "redundancy_status": "Redundant", "redundancy_status_numeric": 2, "revision": "A", "rotation": "0 Degrees", "rotation_numeric": 0, "sc_cpu_speed": 1200, "sc_cpu_type": "Broadwell", "sc_fw": "GTS265R18-01", "serial_number": "DHSIFTJ-18253C63BA", "ssd_alt_path_io_count": 2, "status": "Operational", "status_numeric": 0, "system_memory_size": 8192, "vendor": "", "virtual_disks": 2, "write_policy": "write-back", "write_policy_numeric": 0}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781806", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "1628781806f4748d423055487cb585d5af0ba797b0", "severity": "critical", "info": {"resource_type": "enclosure:hw:psu", "fru": "true:cold_swappable", "event_time": "1628781806", "resource_id": "psu_0.0", "description": "Unable to obtain FRU ID data for this FRU.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"enclosure_id": 0, "serial_number": "NA", "description": "NA", "revision": "NA", "model": "NA", "vendor": "NA", "location": "Enclosure 0 - Left", "part_number": "NA", "fru_shortname": "NA", "mfg_date": "NA", "mfg_vendor_id": "NA", "dc12v": 0, "dc5v": 0, "dc33v": 0, "dc12i": 0, "dc5i": 0, "dctemp": 0, "health": "Degraded", "health_reason": "Unable to obtain FRU ID data for this FRU.", "health_recommendation": "- The FRU ID content was not read. This could be temporary or it may not programmed correctly.     - If this persists, contact technical support.", "status": "Up", "durable_id": "psu_0.0", "position": "Left"}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781807", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"host_id": "cmu", "alert_type": "fault", "alert_id": "162878180655abed0f8b3e426984c552b07491c8fd", "severity": "critical", "info": {"resource_type": "enclosure:hw:fan", "fru": "true:cold_swappable", "event_time": "1628781806", "resource_id": "Fan Module 4", "description": "A subcomponent of this component is unhealthy.", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"durable_id": "fan_module_0.4", "enclosure_id": 0, "health": "Degraded", "health_reason": "A subcomponent of this component is unhealthy.", "health_recommendation": "- See the health information for the unhealthy subcomponents of this component.", "location": "Enclosure 0 - Right", "name": "Fan Module 4", "position": "Indexed", "status": "OK", "fans": [{"durable_id": "fan_0.fm4.1", "fw_revision": "", "health": "OK", "health_reason": "", "health_recommendation": "", "hw_revision": "", "location": "Enclosure 0, Fan Module 4", "locator_led": "Off", "name": "Fan 1", "part_number": "NA", "position": "NA", "serial_number": "NA", "speed": 6480, "status": "Up"}, {"durable_id": "fan_0.fm4.1", "fw_revision": "", "health": "OK", "health_reason": "", "health_recommendation": "", "hw_revision": "", "location": "Enclosure 0, Fan Module 4", "locator_led": "Off", "name": "Fan 1", "part_number": "NA", "position": "NA", "serial_number": "NA", "speed": 6480, "status": "Up"}]}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781811", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "missing", "alert_id": "1628781811e2fd3771903c43cfacc7025a220de60a", "host_id": "cmu", "severity": "critical", "info": {"resource_type": "enclosure:hw:sideplane", "fru": "true:cold_swappable", "event_time": "1628781811", "resource_id": "drawer 0 Left Sideplane", "description": "", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"drawer_id": 0, "durable_id": "sideplane_0.D0.B", "enclosure_id": 0, "health": "Fault", "health_reason": "", "health_recommendation": "Replace the enclosure. Contact technical support.", "location": "enclosure 0, drawer 0", "name": "Left Sideplane", "position": "Top", "status": "Critical", "unhealthy_components": [{"basetype": "sensor", "component_id": "Sideplane 0.D0.B, 24-port Expander, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.0"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.B, Ambient Temp, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.2"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.B, 36-port Expander, Left Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.B.1"}]}}}}'
'actuator_response_type'
b'{"title": "SSPL Sensor Response", "description": "Seagate Storage Platform Library - Sensor Response", "username": "sspl-ll", "signature": "None", "time": "1628781812", "expires": 3600, "message": {"sspl_ll_msg_header": {"schema_version": "1.0.0", "sspl_version": "2.0.0", "msg_version": "1.0.0"}, "sensor_response_type": {"alert_type": "missing", "alert_id": "16287818115ccaa314004b47b4a89f39b693aa2909", "host_id": "cmu", "severity": "critical", "info": {"resource_type": "enclosure:hw:sideplane", "fru": "true:cold_swappable", "event_time": "1628781811", "resource_id": "drawer 0 Right Sideplane", "description": "", "site_id": "DC01", "node_id": "SN01", "rack_id": "RC01", "cluster_id": "CC01"}, "specific_info": {"drawer_id": 0, "durable_id": "sideplane_0.D0.A", "enclosure_id": 0, "health": "Fault", "health_reason": "", "health_recommendation": "Replace the enclosure. Contact technical support.", "location": "enclosure 0, drawer 0", "name": "Right Sideplane", "position": "Top", "status": "Critical", "unhealthy_components": [{"basetype": "sensor", "component_id": "Sideplane 0.D0.A, 36-port Expander, Right Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.A.1"}, {"basetype": "sensor", "component_id": "Sideplane 0.D0.A, 24-port Expander, Right Sideplane", "component_type": "sensor", "component_type_numeric": 11, "health": "Fault", "health_numeric": 2, "health_reason": "Unable to communicate with the sensor.", "health_recommendation": "- Check that all I/O modules and power supplies in the enclosure are fully seated in their slots and that their latches are locked.  - If this does not resolve the problem and the fault is still indicated, perform the following actions:      - If all of the indicated sensors are in one power supply, replace that power supply.      - If the indicated sensors are in both power supplies in an enclosure, the problem is probably with an I/O module.          - If the problem is in the controller enclosure:              - Shut down and reseat the controller modules one at a time until the problem is resolved.              - If this does not correct the problem, shut down and replace the controller modules one at a time until the problem is resolved.          - If the problem is in a disk enclosure:              - Reseat the expansion modules in that enclosure one at a time until the problem is resolved.              - If this does not correct the problem, replace the expansion modules in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the power supplies in that enclosure one at a time until the problem is resolved.          - If this does not correct the problem, replace the chassis FRU.", "meta": "/meta/unhealthy-component", "object_name": "unhealthy-component", "primary_key": "sensor_temp_sp_0.D0.A.0"}]}}}}'
'actuator_response_type'

Is this test can be run on a VM? Seems like the test is expecting sensors reading, which I don't have on my systems. Thanks! cc: @thavanathan @cdeshmukh

hessio commented 3 years ago

I am not sure, @thavanathan can you take a look when you get a chance?

daniarherikurniawan commented 3 years ago

Hi @hessio @thavanathan ! Thanks, this problem is solved by completely reinitiating the VM (using a fresh OVA).

I got this output:


 ******************************************************************************************
TestSuite                                                    Status     Duration(secs)
******************************************************************************************
alerts.node.test_node_disk_actuator                          Fail                0s
alerts.realstor.test_real_stor_controller_sensor             Pass                4s
alerts.realstor.test_real_stor_disk_sensor                   Pass                4s
alerts.realstor.test_real_stor_fan_sensor                    Pass                4s
alerts.realstor.test_real_stor_fan_actuator                  Pass                2s
alerts.realstor.test_real_stor_psu_sensor                    Pass                4s
alerts.realstor.test_real_stor_for_platform_sensor           Pass               16s
alerts.realstor.test_real_stor_sideplane_expander_sensor     Pass                4s
alerts.realstor.test_real_stor_disk_actuator                 Pass                2s
alerts.realstor.test_real_stor_psu_actuator                  Pass                2s
alerts.realstor.test_real_stor_controller_actuator           Pass                4s
alerts.realstor.test_real_stor_sideplane_actuator            Pass                2s
alerts.node.test_node_psu_actuator                           Pass                4s
alerts.node.test_node_fan_actuator                           Pass                2s
alerts.os.test_systemd_service_actuator                      Pass               12s
alerts.node.test_node_bmc_interface                          Pass                8s

****************************************************
TestSuite:16 Tests:19 Passed:18 Failed:1 Skipped:0 TimeTaken:136s
******************************************************
SSPL Automated Test Process ended successfully
Tests completed, restored configs and services ..
Stopping mock server
Cleaned Up ..
Restarting the SSPL service..
Aug 16 15:57:18 sspl-setup[14506]: INFO test - Process done (sspl_setup:348)