search

vzaigrin / UniversalStorageCollector

tool for gathering performance information from different storage system
19 stars 2 forks source link

Java error with influxdb and storwize #7

Closed Mikesch-mp closed 5 years ago

Mikesch-mp commented 6 years ago

Hello, iam trying to use the Collector but i fail with the same error ervertime.

[ERROR] [12/14/2017 15:22:04.126] [USC-akka.actor.default-dispatcher-6] [akka://USC/user/l28stoa.storwize] key not found: measurement
java.util.NoSuchElementException: key not found: measurement
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:59)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:59)
        at universalstoragecollector.Influx.out(Influx.scala:63)
        at universalstoragecollector.Storwize$$anonfun$ask$1$$anonfun$apply$1.apply(Storwize.scala:74)
        at universalstoragecollector.Storwize$$anonfun$ask$1$$anonfun$apply$1.apply(Storwize.scala:62)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at universalstoragecollector.Storwize$$anonfun$ask$1.apply(Storwize.scala:62)
        at universalstoragecollector.Storwize$$anonfun$ask$1.apply(Storwize.scala:54)
        at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
        at universalstoragecollector.Storwize.ask(Storwize.scala:54)
        at universalstoragecollector.Storage$$anonfun$receive$1.applyOrElse(Storage.scala:10)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at universalstoragecollector.Storage.aroundReceive(Storage.scala:5)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

My System: OS: Centos 7.4 Java: 

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

InfluxDB: 1.4.2

my config.xml

<?xml version="1.0" encoding="UTF-8"?>
<collector>
  <extractors>
    <extractor name="storwize">
      <methods>
        <method name="systemstats">
          <cmd>lssystemstats</cmd>
          <output param="stat_name" data="stat_current"></output>
        </method>
        <method name="nodestats">
          <cmd>svcinfo lsnodestats</cmd>
          <output head="node_name" param="stat_name" data="stat_current"></output>
        </method>
      </methods>
    </extractor>
  </extractors>
  <outputs>
    <output name="influxdb">
      <type>influxdb</type>
      <address>127.0.0.1</address>
      <port>8086</port>
      <dbname>test</dbname>
    </output>
  </outputs>
  <systems>
    <system>
      <name>stoa</name>
      <class>storwize</class>
      <interval>1 min</interval>
      <extractor name="storwize">
        <address>10.10.10.10</address>
        <username>icinga2</username>
        <password>icinga2</password>
      </extractor>
      <output>influxdb</output>
    </system>
  </systems>
</collector>

Storage: IBM V7000 Version 7.8.1.2 (build 135.5.1708101259000)

vzaigrin commented 6 years ago

Hello!

Did you tried to execute commands "lssystemstats" and "svcinfo lsnodestats"? It's looks like something wrong with output from its.

Mikesch-mp commented 6 years ago

HI, yes i did and it works as the user

lsystemstats:

IBM_Storwize:L28STO:icinga2>lssystemstats
stat_name          stat_current stat_peak stat_peak_time
compression_cpu_pc 14           18        171215111327
cpu_pc             4            7         171215111517
fc_mb              61           479       171215111302
fc_io              11515        17055     171215111252
sas_mb             73           1282      171215111512
sas_io             1240         10136     171215111517
iscsi_mb           0            0         171215111627
iscsi_io           0            0         171215111627
write_cache_pc     28           28        171215111627
total_cache_pc     79           80        171215111602
vdisk_mb           24           423       171215111302
vdisk_io           2639         5070      171215111602
vdisk_ms           0            0         171215111627
mdisk_mb           26           629       171215111302
mdisk_io           425          3592      171215111517
mdisk_ms           4            8         171215111142
drive_mb           73           1281      171215111512
drive_io           1240         10137     171215111517
drive_ms           5            11        171215111137
vdisk_r_mb         4            382       171215111302
vdisk_r_io         375          2604      171215111602
vdisk_r_ms         0            1         171215111542
vdisk_w_mb         18           144       171215111517
vdisk_w_io         2264         3195      171215111512
vdisk_w_ms         0            0         171215111627
mdisk_r_mb         1            599       171215111302
mdisk_r_io         85           2664      171215111302
mdisk_r_ms         1            2         171215111622
mdisk_w_mb         24           124       171215111512
mdisk_w_io         340          2999      171215111517
mdisk_w_ms         5            10        171215111147
drive_r_mb         27           1231      171215111342
drive_r_io         512          5636      171215111237
drive_r_ms         4            14        171215111247
drive_w_mb         45           172       171215111512
drive_w_io         727          5630      171215111517
drive_w_ms         5            8         171215111552
power_w            2385         2392      171215111532
temp_c             27           27        171215111627
temp_f             80           80        171215111627
iplink_mb          0            0         171215111627
iplink_io          0            0         171215111627
iplink_comp_mb     0            0         171215111627
cloud_up_mb        0            0         171215111627
cloud_up_ms        0            0         171215111627
cloud_down_mb      0            0         171215111627
cloud_down_ms      0            0         171215111627

svcinfo lsnodestats:

IBM_Storwize:L28STO:icinga2>svcinfo lsnodestats
node_id node_name stat_name          stat_current stat_peak stat_peak_time
1       node1     compression_cpu_pc 14           21        171215111254
1       node1     cpu_pc             5            9         171215111514
1       node1     fc_mb              21           271       171215111304
1       node1     fc_io              3590         6015      171215111604
1       node1     sas_mb             40           1056      171215111459
1       node1     sas_io             339          5082      171215111519
1       node1     iscsi_mb           0            0         171215111739
1       node1     iscsi_io           0            0         171215111739
1       node1     write_cache_pc     28           30        171215111709
1       node1     total_cache_pc     79           80        171215111704
1       node1     vdisk_mb           8            251       171215111304
1       node1     vdisk_io           935          2817      171215111604
1       node1     vdisk_ms           0            0         171215111739
1       node1     mdisk_mb           12           346       171215111304
1       node1     mdisk_io           95           1935      171215111309
1       node1     mdisk_ms           8            8         171215111739
1       node1     drive_mb           40           1056      171215111459
1       node1     drive_io           340          5084      171215111519
1       node1     drive_ms           7            10        171215111349
1       node1     vdisk_r_mb         1            237       171215111304
1       node1     vdisk_r_io         150          2186      171215111604
1       node1     vdisk_r_ms         0            0         171215111739
1       node1     vdisk_w_mb         7            60        171215111514
1       node1     vdisk_w_io         785          1221      171215111514
1       node1     vdisk_w_ms         0            0         171215111739
1       node1     mdisk_r_mb         0            333       171215111304
1       node1     mdisk_r_io         21           1537      171215111309
1       node1     mdisk_r_ms         0            2         171215111359
1       node1     mdisk_w_mb         12           71        171215111709
1       node1     mdisk_w_io         75           1696      171215111519
1       node1     mdisk_w_ms         10           11        171215111714
1       node1     drive_r_mb         14           1035      171215111459
1       node1     drive_r_io         101          4392      171215111604
1       node1     drive_r_ms         6            12        171215111349
1       node1     drive_w_mb         25           113       171215111709
1       node1     drive_w_io         240          2875      171215111519
1       node1     drive_w_ms         7            8         171215111659
1       node1     iplink_mb          0            0         171215111739
1       node1     iplink_io          0            0         171215111739
1       node1     iplink_comp_mb     0            0         171215111739
1       node1     cloud_up_mb        0            0         171215111739
1       node1     cloud_up_ms        0            0         171215111739
1       node1     cloud_down_mb      0            0         171215111739
1       node1     cloud_down_ms      0            0         171215111739
2       node2     compression_cpu_pc 15           23        171215111712
2       node2     cpu_pc             6            9         171215111707
2       node2     fc_mb              13           176       171215111707
2       node2     fc_io              2465         6124      171215111252
2       node2     sas_mb             365          524       171215111637
2       node2     sas_io             1465         2121      171215111637
2       node2     iscsi_mb           0            0         171215111737
2       node2     iscsi_io           0            0         171215111737
2       node2     write_cache_pc     28           30        171215111707
2       node2     total_cache_pc     79           80        171215111647
2       node2     vdisk_mb           7            151       171215111707
2       node2     vdisk_io           570          2920      171215111707
2       node2     vdisk_ms           0            1         171215111542
2       node2     mdisk_mb           1            128       171215111647
2       node2     mdisk_io           36           1180      171215111647
2       node2     mdisk_ms           0            2         171215111712
2       node2     drive_mb           365          524       171215111637
2       node2     drive_io           1465         2106      171215111637
2       node2     drive_ms           3            19        171215111247
2       node2     vdisk_r_mb         0            118       171215111647
2       node2     vdisk_r_io         41           1381      171215111647
2       node2     vdisk_r_ms         0            2         171215111707
2       node2     vdisk_w_mb         7            124       171215111707
2       node2     vdisk_w_io         530          2545      171215111707
2       node2     vdisk_w_ms         0            0         171215111737
2       node2     mdisk_r_mb         1            128       171215111647
2       node2     mdisk_r_io         35           1180      171215111647
2       node2     mdisk_r_ms         0            2         171215111712
2       node2     mdisk_w_mb         0            0         171215111737
2       node2     mdisk_w_io         2            2         171215111737
2       node2     mdisk_w_ms         0            0         171215111737
2       node2     drive_r_mb         365          524       171215111637
2       node2     drive_r_io         1464         2106      171215111637
2       node2     drive_r_ms         4            19        171215111247
2       node2     drive_w_mb         0            0         171215111737
2       node2     drive_w_io         2            2         171215111737
2       node2     drive_w_ms         0            0         171215111737
2       node2     iplink_mb          0            0         171215111737
2       node2     iplink_io          0            0         171215111737
2       node2     iplink_comp_mb     0            0         171215111737
2       node2     cloud_up_mb        0            0         171215111737
2       node2     cloud_up_ms        0            0         171215111737
2       node2     cloud_down_mb      0            0         171215111737
2       node2     cloud_down_ms      0            0         171215111737
3       node3     compression_cpu_pc 20           20        171215111739
3       node3     cpu_pc             4            6         171215111704
3       node3     fc_mb              15           169       171215111304
3       node3     fc_io              2697         3959      171215111304
3       node3     sas_mb             0            831       171215111539
3       node3     sas_io             4            3311      171215111434
3       node3     iscsi_mb           0            0         171215111739
3       node3     iscsi_io           0            0         171215111739
3       node3     write_cache_pc     28           30        171215111704
3       node3     total_cache_pc     79           80        171215111704
3       node3     vdisk_mb           4            155       171215111304
3       node3     vdisk_io           215          1460      171215111304
3       node3     vdisk_ms           0            0         171215111739
3       node3     mdisk_mb           0            264       171215111304
3       node3     mdisk_io           10           1120      171215111304
3       node3     mdisk_ms           3            5         171215111339
3       node3     drive_mb           0            831       171215111539
3       node3     drive_io           4            3311      171215111434
3       node3     drive_ms           7            9         171215111429
3       node3     vdisk_r_mb         0            145       171215111304
3       node3     vdisk_r_io         19           622       171215111304
3       node3     vdisk_r_ms         0            1         171215111649
3       node3     vdisk_w_mb         3            54        171215111514
3       node3     vdisk_w_io         196          1000      171215111514
3       node3     vdisk_w_ms         0            0         171215111739
3       node3     mdisk_r_mb         0            264       171215111304
3       node3     mdisk_r_io         10           1120      171215111304
3       node3     mdisk_r_ms         3            5         171215111339
3       node3     mdisk_w_mb         0            0         171215111739
3       node3     mdisk_w_io         0            0         171215111739
3       node3     mdisk_w_ms         0            0         171215111739
3       node3     drive_r_mb         0            831       171215111539
3       node3     drive_r_io         4            3311      171215111434
3       node3     drive_r_ms         7            9         171215111429
3       node3     drive_w_mb         0            0         171215111739
3       node3     drive_w_io         0            1         171215111734
3       node3     drive_w_ms         0            0         171215111739
3       node3     iplink_mb          0            0         171215111739
3       node3     iplink_io          0            0         171215111739
3       node3     iplink_comp_mb     0            0         171215111739
3       node3     cloud_up_mb        0            0         171215111739
3       node3     cloud_up_ms        0            0         171215111739
3       node3     cloud_down_mb      0            0         171215111739
3       node3     cloud_down_ms      0            0         171215111739
4       node4     compression_cpu_pc 12           25        171215111515
4       node4     cpu_pc             4            9         171215111705
4       node4     fc_mb              16           369       171215111705
4       node4     fc_io              2960         5942      171215111705
4       node4     sas_mb             33           448       171215111510
4       node4     sas_io             366          4831      171215111515
4       node4     iscsi_mb           0            0         171215111740
4       node4     iscsi_io           0            0         171215111740
4       node4     write_cache_pc     28           31        171215111705
4       node4     total_cache_pc     79           80        171215111720
4       node4     vdisk_mb           7            194       171215111705
4       node4     vdisk_io           759          2526      171215111705
4       node4     vdisk_ms           0            0         171215111740
4       node4     mdisk_mb           13           210       171215111705
4       node4     mdisk_io           87           1385      171215111705
4       node4     mdisk_ms           8            10        171215111730
4       node4     drive_mb           33           448       171215111510
4       node4     drive_io           366          4831      171215111515
4       node4     drive_ms           6            10        171215111250
4       node4     vdisk_r_mb         0            76        171215111705
4       node4     vdisk_r_io         17           321       171215111705
4       node4     vdisk_r_ms         0            2         171215111655
4       node4     vdisk_w_mb         6            117       171215111705
4       node4     vdisk_w_io         741          2205      171215111705
4       node4     vdisk_w_ms         0            0         171215111740
4       node4     mdisk_r_mb         0            156       171215111705
4       node4     mdisk_r_io         20           907       171215111705
4       node4     mdisk_r_ms         2            4         171215111635
4       node4     mdisk_w_mb         12           62        171215111515
4       node4     mdisk_w_io         67           1302      171215111515
4       node4     mdisk_w_ms         10           11        171215111735
4       node4     drive_r_mb         12           388       171215111610
4       node4     drive_r_io         120          2075      171215111515
4       node4     drive_r_ms         6            11        171215111250
4       node4     drive_w_mb         21           91        171215111710
4       node4     drive_w_io         246          2755      171215111515
4       node4     drive_w_ms         7            8         171215111630
4       node4     iplink_mb          0            0         171215111740
4       node4     iplink_io          0            0         171215111740
4       node4     iplink_comp_mb     0            0         171215111740
4       node4     cloud_up_mb        0            0         171215111740
4       node4     cloud_up_ms        0            0         171215111740
4       node4     cloud_down_mb      0            0         171215111740
4       node4     cloud_down_ms      0            0         171215111740
vzaigrin commented 6 years ago

Could you try to setup output to Carbon instead of InfluxDB and check the output of "nc -k -l 2003? If it works fine, the output from Storwize extractor to InfluxDB is incorrect.

Mikesch-mp commented 6 years ago

Cant install carbon :(

vzaigrin commented 6 years ago

You should not install Carbon.

You should add output to carbon in your configuration file:

<output name="carbon">
  <type>carbon</type>
  <address>127.0.0.1</address>
  <port>2003</port>
</output>

Change output to Carbon for Storwize: carbon And run "nc -k -l 2003"

Mikesch-mp commented 6 years ago

Ahh OK. Will do it as soon my remote access is working.

vzaigrin commented 6 years ago

It's look like I have found an error in Storwize extractor. Please try to use new binary file.

Mikesch-mp commented 6 years ago

Sorry for late reply. Works perfect now. Is there any way to get io data also per vdisk ?

vzaigrin commented 6 years ago

I am not an expert in Storwize. If you know command for it we could use it.