Problem with the Netapp E-Series Check

[Cologran]

Hello!

We use the Netapp E-Series Check with the CheckMK RAW Edition version 2.2.0p6.

Since updating the NetApp to firmware version 11.80GA we get an error in CheckMK. SAS A-1 "Port status" is always set to "failed: WARN".

Exactly as described here: https://kb.netapp.com/onprem/E-Series/Hardware/E-Series%3A_Warning_SAS_Port_failed_on_monitoring_system_-_CheckMK

1. We have not found any newer firmware for the NetApp.
2. The NetApp has also been rebooted, but without success.

[chexma]

Hi Cologran,

unfortunately I dont have systems on 11. 80 yet, so i can't test it. I assume that the API returns different states for the SAS Ports now. If you can run the special agent on the cli and send me the output, i can debug it. Do you have external SAS connections / disk shelfes or is it internal storage only?

[chexma]

Cologran,

i tried four different Configs in the Netapp E-Series Simulator but was not able to reproduce the error. If you want, you can provide the agent output of your E-Series, otherwise I will have to wait until i have a system running on 11.80.

[Cologran]

Hi!

"Do you have external SAS connections / disk shelfes or is it internal storage only?" -> We have internal storage only.

Do you mean this output?

`OMD[monitoring]:~$ cmk -vvv --debug --checks netapp_eseries_interfaces NETAPP02 WARNING: '--checks' is deprecated in favour of option 'detect-plugins' Checkmk version 2.2.0p6

• FETCHING DATA Source: SourceInfo(hostname='NETAPP02', ipaddress='10.95.xxx.xxx', ident='special_netappeseries', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>) [cpu_tracking] Start [7fab2c68fe50] Read from cache: AgentFileCache(NETAPP02, path_template=/omd/sites/monitoring/tmp/check_mk/data_source_cache/special_netappeseries/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6) Not using cache (Mode Mode.FORCE_SECTIONS) [ProgramFetcher] Execute data source Calling: /omd/sites/monitoring/local/share/check_mk/agents/special/agent_netappeseries -u monitor -s PASSWORD 10.95.xxx.xxx Not using cache (Mode Mode.FORCE_SECTIONS) [cpu_tracking] Stop [7fab2c68fe50 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.23, children_system=0.04, elapsed=0.3399999998509884))] Source: SourceInfo(hostname='NETAPP02', ipaddress='10.95.xxx.xxx', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>) [cpu_tracking] Start [7fab2c68e650] Read from cache: NoCache(NETAPP02, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1) [PiggybackFetcher] Execute data source No piggyback files for 'NETAPP02'. Skip processing. No piggyback files for '10.95.xxx.xxx'. Skip processing. [cpu_tracking] Stop [7fab2c68e650 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
• PARSE FETCHER RESULTS <<<netapp_eseries_batteries:sep(0)>>> / Transition NOOPParser -> HostSectionParser <<<netapp_eseries_controllers:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_drawers:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_drives:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_esms:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_fans:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_interfaces:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_pools:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_powersupplies:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_system:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_trays:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_volumes:sep(0)>>> / Transition HostSectionParser -> HostSectionParser <<<netapp_eseries_thermalsensors:sep(0)>>> / Transition HostSectionParser -> HostSectionParser No persisted sections No persisted sections HostKey(hostname='NETAPP02', source_type=<SourceType.HOST: 1>) -> Add sections: ['netapp_eseries_interfaces'] HostKey(hostname='NETAPP02', source_type=<SourceType.HOST: 1>) -> Add sections: [] Received no piggyback data [cpu_tracking] Start [7fab2c6b12d0] value store: synchronizing Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 Got lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 value store: loading from disk Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 Released lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 value store: synchronizing Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 Got lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 value store: already loaded Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 Released lock on /omd/sites/monitoring/tmp/check_mk/counters/NETAPP02 Interface ISCSI A-1 Port status: up, Read: 1.05 kB/s, Write: 83.9 kB/s, Read operations: 0.293/s, Write operations: 15.313/s Interface ISCSI A-2 Port status: up, Read: 1.05 kB/s, Write: 79.7 kB/s, Read operations: 0.28/s, Write operations: 15.463/s Interface SAS A-1 Port status: failed(!), Read: 12.2 MB/s, Write: 0.00 B/s, Read operations: 93.487/s, Write operations: 0.0/s Interface SAS A-2 Port status: optimal, Read: 0.00 B/s, Write: 0.00 B/s, Read operations: 0.0/s, Write operations: 0.0/s No piggyback files for 'NETAPP02'. Skip processing. No piggyback files for '10.95.xxx.xxx'. Skip processing. [cpu_tracking] Stop [7fab2c6b12d0 - Snapshot(process=posix.times_result(user=0.08000000000000007, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.08000000193715096))] [special_netappeseries] Success, [piggyback] Success (but no data found for this host), execution time 0.4 sec | execution_time=0.420 user_time=0.080 system_time=0.000 children_user_time=0.230 children_system_time=0.040 cmk_time_ds=0.070 cmk_time_agent=0.000`

Or this one?

`OMD[monitoring]:~$ cmk -nv --debug NETAPP02

• FETCHING DATA [ProgramFetcher] Execute data source [PiggybackFetcher] Execute data source No piggyback files for 'NETAPP02'. Skip processing. No piggyback files for '10.95.xxx.xxx'. Skip processing. Battery Tray 99-1 Battery Tray 99-1, status: optimal, serial nr.: SN SU92081476SO Controller A Controller A Model 2806, status: optimal, serial nr.: 021919053678 Drive 99-1 HDD Drive 99-1, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.348/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-10 HDD Drive 99-10, status: optimal, size: 3.63 TiB, Read: 611 kB/s, Write: 0.00 B/s, Read operations: 4.662/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-11 HDD Drive 99-11, status: optimal, size: 3.63 TiB, Read: 614 kB/s, Write: 0.00 B/s, Read operations: 4.689/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-12 HDD Drive 99-12, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.348/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-2 HDD Drive 99-2, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.348/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-3 HDD Drive 99-3, status: optimal, size: 3.63 TiB, Read: 0.00 B/s, Write: 0.00 B/s, Read operations: 0.003/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-4 HDD Drive 99-4, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.345/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-5 HDD Drive 99-5, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.348/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-6 HDD Drive 99-6, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.345/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-7 HDD Drive 99-7, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.345/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-8 HDD Drive 99-8, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.348/s, Write operations: 0.0/s, Temperature: 0 °C Drive 99-9 HDD Drive 99-9, status: optimal, size: 3.63 TiB, Read: 1.22 MB/s, Write: 0.00 B/s, Read operations: 9.345/s, Write operations: 0.0/s, Temperature: 0 °C FAN Tray 99-1 Fan Tray 99-1, status: optimal FAN Tray 99-2 Fan Tray 99-2, status: optimal FAN Tray 99-3 Fan Tray 99-3, status: optimal FAN Tray 99-4 Fan Tray 99-4, status: optimal Interface ISCSI A-1 Port status: up, Read: 1.05 kB/s, Write: 82.8 kB/s, Read operations: 0.318/s, Write operations: 15.699/s Interface ISCSI A-2 Port status: up, Read: 1.05 kB/s, Write: 82.8 kB/s, Read operations: 0.277/s, Write operations: 15.861/s Interface SAS A-1 Port status: failed(!), Read: 12.3 MB/s, Write: 0.00 B/s, Read operations: 93.507/s, Write operations: 0.0/s Interface SAS A-2 Port status: optimal, Read: 0.00 B/s, Write: 0.00 B/s, Read operations: 0.0/s, Write operations: 0.0/s PSU Tray 99-1 Powersupply Tray 99-1, status: optimal, serial nr.: PSQ094191403827, type: DCJ9133-01 PSU Tray 99-2 Powersupply Tray 99-2, status: optimal, serial nr.: PSQ094191404021, type: DCJ9133-01 Pool DDP01 Pool DDP01, status: complete, Used: 73.00% - 23.0 TiB of 31.5 TiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0% Sensor Tray 99-1 Thermal sensor CPU, status: optimal Sensor Tray 99-2 Thermal sensor inlet, status: optimal Sensor Tray 99-5 Thermal sensor, status: optimal Sensor Tray 99-6 Thermal sensor, status: optimal System Info NETAPP02 E-Series 2806, status: optimal, serial nr.: 032026002467 Tray 99 Tray ID 99, type: de212c with 12 slots, serial nr.: 032026002467, part number: 111-02653+B0 Volume SAS_VMFS03 Volume SAS_VMFS03, size: 5.00 TiB, raidlevel: raid6, status: optimal, Read: 1.05 kB/s, Write: 2.10 kB/s, Read operations: 1.095/s, Write operations: 1.405/s Volume SAS_VMFS04 Volume SAS_VMFS04, size: 5.00 TiB, raidlevel: raid6, status: optimal, Read: 3.15 kB/s, Write: 165 kB/s, Read operations: 1.5/s, Write operations: 32.166/s Volume Veeam_Copy Volume Veeam_Copy, size: 13.0 TiB, raidlevel: raid6, status: optimal, Read: 0.00 B/s, Write: 0.00 B/s, Read operations: 0.0/s, Write operations: 0.0/s No piggyback files for 'NETAPP02'. Skip processing. No piggyback files for '10.95.xxx.xxx'. Skip processing. [special_netappeseries] Success, [piggyback] Success (but no data found for this host), execution time 0.4 sec | execution_time=0.440 user_time=0.100 system_time=0.000 children_user_time=0.260 children_system_time=0.010 cmk_time_ds=0.060 cmk_time_agent=0.000`

[chexma]

Hi Cologran,

i need the output of the special agent. I think the easiest way to get that is to go to the Check_MK service of that host, click on the "Burger" menu and choose - "Download Agent Output". Please don´t post it here, you can send the file to ...

[chexma]

Hi Cologran,

thanks for the agent output. The E-Series REST API is responding with the following information about the SAS Interface:

{ 'SAS A-1': { 'interfaceRef': '2201000000000000000000000000000000000000', 'channelType': 'driveside', ... 'state': 'failed', 'miswireType': 'none', 'channelPortRef': '1F01000001010000000000000000000000000000', ... }

I don´t think that it is a good idea to use the state: failed as "ok" for my plugin in checkmk, so you need to open a Netapp support case to fix the REST API output.