Unparseable date error if trying to monitor vplex

[eschek87]

[ERROR] [01/31/2017 08:00:55.994] [USC-akka.actor.default-dispatcher-7] [akka://USC/user/prz-vplex] Unparseable date: "" java.text.ParseException: Unparseable date: "" at java.text.DateFormat.parse(DateFormat.java:366) at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:66) at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:56) at scala.collection.immutable.List.foreach(List.scala:381) at universalstoragecollector.VPLEX.ask(VPLEX.scala:56) at universalstoragecollector.Storage$$anonfun$receive$1.applyOrElse(Storage.scala:10) at akka.actor.Actor$class.aroundReceive(Actor.scala:484) at universalstoragecollector.Storage.aroundReceive(Storage.scala:5) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[vzaigrin]

What data format are in your VPLEX monitors file? It is assumed it should looks like this: '2017-01-31 08:55:39'

[eschek87]

Hi,

not sure how to find the setting. I am still searching at emc support site. I tried the following command: VPlexcli:/> getsysinfo Running from localhost:/var/log/VPlex/cli Local time 2017/01/31 09:18:48

We have vplex version 5.5

[vzaigrin]

Did you setup VPLEX monitors? Read how to do this here: https://vzaigrin.wordpress.com/2016/07/21/vplexcollector-tool-for-gathering-emc-vplex

[eschek87]

Ok there were only some default monitors. Now i have created the monitors: monitor create -p 5min -n director-1-1-A_fe-lu -d director-1-1-A -s fe-lu.ops,fe-lu.read,fe-lu.read-avg-lat,fe-lu.write,fe-lu.write-avg-lat -t /clusters/cluster-1/virtual-volumes/*

monitor create -p 5min -n director-1-1-B_fe-lu -d director-1-1-B -s fe-lu.ops,fe-lu.read,fe-lu.read-avg-lat,fe-lu.write,fe-lu.write-avg-lat -t /clusters/cluster-1/virtual-volumes/*

monitor add-file-sink -m director-1-1-A_director-1-1-A_fe-lu -f /var/log/VPlex/cli/reports/director-1-1-A_fe-lu.csv

monitor add-file-sink -m director-1-1-B_director-1-1-B_fe-lu -f /var/log/VPlex/cli/reports/director-1-1-B_fe-lu.csv

I will wait till some csv files will be created and check again

[vzaigrin]

PERPETUAL_MONITOR are standard VPLEX monitors. You should setup yours monitors by hand.

[eschek87]

I have some error for our vnx1. I can see data in the influxdb database "usc" but no data in grafana

[ERROR] [01/31/2017 11:53:05.352] [USC-akka.actor.default-dispatcher-3] [akka://USC/user/prz-vnx.block] None.get java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:347) at scala.None$.get(Option.scala:345) at universalstoragecollector.Influx.out(Influx.scala:52) at universalstoragecollector.VNXBlock.simple(VNXBlock.scala:133) at universalstoragecollector.VNXBlock$$anonfun$ask$1.apply(VNXBlock.scala:102) at universalstoragecollector.VNXBlock$$anonfun$ask$1.apply(VNXBlock.scala:87) at scala.collection.immutable.List.foreach(List.scala:381) at universalstoragecollector.VNXBlock.ask(VNXBlock.scala:87) at universalstoragecollector.Storage$$anonfun$receive$1.applyOrElse(Storage.scala:10) at akka.actor.Actor$class.aroundReceive(Actor.scala:484) at universalstoragecollector.Storage.aroundReceive(Storage.scala:5) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I can see data in influxdb, but nothing in grafana.

and after i created all vplex monitors i got the following: [ERROR] [01/31/2017 12:24:54.906] [USC-akka.actor.default-dispatcher-3] [akka://USC/user/prz-vplex] None.get java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:347) at scala.None$.get(Option.scala:345) at universalstoragecollector.Influx.out(Influx.scala:52) at universalstoragecollector.VPLEX$$anonfun$ask$1$$anonfun$apply$1.apply(VPLEX.scala:98) at universalstoragecollector.VPLEX$$anonfun$ask$1$$anonfun$apply$1.apply(VPLEX.scala:74) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174) at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:74) at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:56) at scala.collection.immutable.List.foreach(List.scala:381) at universalstoragecollector.VPLEX.ask(VPLEX.scala:56) at universalstoragecollector.Storage$$anonfun$receive$1.applyOrElse(Storage.scala:10) at akka.actor.Actor$class.aroundReceive(Actor.scala:484) at universalstoragecollector.Storage.aroundReceive(Storage.scala:5) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Maybe is this caused because we have changed our vplex cluster name from "cluster-1" to "new-clustername"? I cannot see data in influxdb.

UPDATE: I have now select carbon as output and it works...i need to check why influxdb did not work.

[vzaigrin]

If you see data in the InfluxDB you should see its in Grafana.

Error messages you provided says about other problem.

1) VNXBlock Do you has FAST Cache enabled on your VNX? If yes, add objectName="fastcache" into the fastcache line in collector.xml

For example:

vnx ... ... 2) VPLEX I see where is error in source code, but I need more time to correct it. VPLEX output to Carbon should works well.

[vzaigrin]

For VNX use this string in the collector.xml:

method name="fastcache" objectName="fastcache" address="10.10.10.10"

[eschek87]

We haven't fast cache. I have remove the method. Thanks for that hint! I have now one error left in collector-error.log: Tue Jan 31 13:47:37 CET 2017 Incorrect Extractor definition: vnxnar

`

/opt/Navisphere/bin/naviseccli

  <listCmd>-User #username -Password #password -Scope #scope -Address #address analyzer -archive -list</listCmd>
  <getCmd>-User #username -Password #password -Scope #scope -Address #address analyzer -archive -path #pool -file #filename -o</getCmd>
  <dataCmd>analyzer -archivedump -data #filename -out #dataname -delim cm -header y -overwrite y</dataCmd>
  <relCmd>analyzer -archivedump -rel #filename -out #relname -xml -overwrite y</relCmd>
</extractor>

`

[vzaigrin]

This lines are correct.

Each extractors have two definitions: common and for concrete storage system. This lines are from common VNXNAR definition. You should check VNXNAR settings in storage systems list.

If don't use VNXNAR for your systems, you could:

• do nothing and omit this message
• completely remove VNXNAR definition from the collector.xml
• create empty 'pool' subdirectory in the USC_HOME directory, collector check this directory for VNXNAR extractor.

[eschek87]

In the syslog i found the following: SELECT mean(PrctBusy) FROM usc.usc.spa

If i insert this command in influx commandline it says:

SELECT mean(PrctBusy) FROM usc.usc.spa ERR: retention policy not found: usc

I have created the database manually before. Is that the fault? Create the collector automatically the database?

[vzaigrin]

Collector doesn't create usc database. It mean you should create empty database by hand before first start of the collector.

[vzaigrin]

I have corrected VPLEX extractor and Influx output. It should works now without problems.

[eschek87]

Nice. Thank you. I will check it. At the moment i have a problem with some monitors that i have defined in the xml and created on the vplex. I can't see that data in graphit. Is there a limit for the number of monitors?

[vzaigrin]

No, there are no limits for number of monitors. First, you should check that your monitors write data to files. You could use vplexcli command ll /monitoring/directors//monitors//sinks/ Also you could check files in the directory /var/log/VPlex/cli/reports/ on the management server.

You could also check data in the Carbon. Its should be in the directory /val/lib/carbon/whisper

[eschek87]

I have created the monitors and sinks in vplex: monitor create -p 5min -n director-1-1-A_storage-volume -d director-1-1-A -s storage-volume.read-avg-lat,storage-volume.write-avg-lat monitor create -p 5min -n director-1-1-B_storage-volume -d director-1-1-B -s storage-volume.read-avg-lat,storage-volume.write-avg-lat monitor add-file-sink -m director-1-1-A_director-1-1-A_storage-volume -f /var/log/VPlex/cli/reports/director-1-1-A_storage-volume.csv monitor add-file-sink -m director-1-1-B_director-1-1-B_storage-volume -f /var/log/VPlex/cli/reports/director-1-1-B_storage-volume.csv

I can see in /var/log/VPlex/cli/reports/ the files and they contain data. But in carbon i cannot see that data: ls -la /var/lib/graphite/whisper/vplex/srz-vplex/2-1-A/ total 48 drwxr-xr-x 7 _graphite _graphite 4096 Feb 1 17:22 . drwxr-xr-x 4 _graphite _graphite 4096 Feb 1 17:22 .. drwxr-xr-x 4 _graphite _graphite 4096 Feb 1 17:22 be-prt drwxr-xr-x 3 _graphite _graphite 4096 Feb 1 17:21 director drwxr-xr-x 7 _graphite _graphite 4096 Feb 1 17:21 fe-lu drwxr-xr-x 8 _graphite _graphite 4096 Feb 1 17:22 fe-prt -rw-r--r-- 1 _graphite _graphite 17308 Feb 2 09:52 recent-average.wsp drwxr-xr-x 5 _graphite _graphite 4096 Feb 1 17:22 virtual-volume

[vzaigrin]

Did you add this monitors to the collector.xml file? If you created this monitors after starting collector, you should restart collector.

The simplest way to check what collector output to Carbon, is using nc utility. Stop carbon and run nc -k -l 2003 in the terminal.

[eschek87]

Yes i have added them and double check the filenames: `

` I have also restarted the collector. I will try the "nc" command Update: I tried to stop carbon with service carbon-cache start, but then i got in the collector-error.log: Thu Feb 02 12:45:33 CET 2017 vplex.prz-vplex: Error in output to the carbon server 127.0.0.1:2003

[vzaigrin]

You have created monitors for directors 1-1-A and 1-1-B. In Graphite this data should be in the directories 1-1-A and 1-1-B, not 2-1-A.

[eschek87]

Sorry i have sent you the wrong output. We have two vplex systems and i have done the steps on both systems: Here are the output from the correct system: ls -la /var/lib/graphite/whisper/vplex/prz-vplex/1-1-A/ total 48 drwxr-xr-x 7 _graphite _graphite 4096 Feb 1 13:00 . drwxr-xr-x 4 _graphite _graphite 4096 Feb 1 12:59 .. drwxr-xr-x 4 _graphite _graphite 4096 Feb 1 13:00 be-prt drwxr-xr-x 3 _graphite _graphite 4096 Feb 1 12:59 director drwxr-xr-x 7 _graphite _graphite 4096 Feb 1 12:59 fe-lu drwxr-xr-x 8 _graphite _graphite 4096 Feb 1 13:00 fe-prt -rw-r--r-- 1 _graphite _graphite 17308 Feb 2 14:39 recent-average.wsp drwxr-xr-x 5 _graphite _graphite 4096 Feb 1 12:59 virtual-volume

There is no folder "storage-volume"

[vzaigrin]

Yes, there are no folder storage-volume. This is strange. You should check output to the nc utility.

[eschek87]

I found the data in the file "recent-average.wsp"

[ollivers]

we are also monitoring several vplex directors with USC. Monitors setup as described in

https://vzaigrin.wordpress.com/2016/07/21/vplexcollector-tool-for-gathering-emc-vplex/

Nearly all metrics are collected, but sometimes (every two oder three hours) we encounter the folloging "unparseable date" error:

Nov 13 05:44:09 mon-oed-115-0 java[18976]: [ERROR] [11/13/2018 05:44:09.253] [USC-akka.actor.default-dispatcher-13] [akka://USC/user/cluster-2.vplex] Unparseable date: "Time" Nov 13 05:44:09 mon-oed-115-0 java[18976]: java.text.ParseException: Unparseable date: "Time" Nov 13 05:44:09 mon-oed-115-0 java[18976]: at java.text.DateFormat.parse(DateFormat.java:366) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:75) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at universalstoragecollector.VPLEX$$anonfun$ask$1.apply(VPLEX.scala:61) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at scala.collection.immutable.List.foreach(List.scala:381) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at universalstoragecollector.VPLEX.ask(VPLEX.scala:61) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at universalstoragecollector.Storage$$anonfun$receive$1.applyOrElse(Storage.scala:10) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.actor.Actor$class.aroundReceive(Actor.scala:484) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at universalstoragecollector.Storage.aroundReceive(Storage.scala:5) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.actor.ActorCell.invoke(ActorCell.scala:495) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.dispatch.Mailbox.run(Mailbox.scala:224) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at akka.dispatch.Mailbox.exec(Mailbox.scala:234) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) Nov 13 05:44:09 mon-oed-115-0 java[18976]: at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Example montor:

VPlexcli:/> ll /monitoring/directors/director-2-3-A/monitors/director-2-3-A_director-2-3-A_fe-lu/

/monitoring/directors/director-2-3-A/monitors/director-2-3-A_director-2-3-A_fe-lu:

Attributes: Name Value

---

average-period 5min bucket-count 64 bucket-max - bucket-min - bucket-width - collecting-data true firmware-id 7 idle-for 1.77min ownership true period 5min statistics [fe-lu.ops, fe-lu.read, fe-lu.read-avg-lat, fe-lu.write, fe-lu.write-avg-lat] targets device_Symm0522_00C0_1_vol, device_Symm0522_00C1_1_vol, device_Symm0522_00C5_1_vol, device_Symm0522_1480_1_vol, device_Symm0522_1481_1_vol, device_Symm0522_1482_1_vol, device_Symm0522_1483_1_vol, device_Symm0522_1484_1_vol, device_Symm0522_1485_1_vol, device_Symm0522_1486_1_vol, ... (1999 total) version -

Contexts: Name Description

---

sinks Contains all of the sinks set up to collect data from this performance monitor.

VPlexcli:/> ll /monitoring/directors/director-2-3-A/monitors/director-2-3-A_director-2-3-A_fe-lu/sinks/

/monitoring/directors/director-2-3-A/monitors/director-2-3-A_director-2-3-A_fe-lu/sinks: Name Enabled Format Sink-To

---

file true csv /var/log/VPlex/cli/reports/director-2-3-A_fe-lu.csv

Best regards, Oliver