search

openiomon / hds2graphite

A utility to query metrics from Hitachi Vantara block storage and transfer them to graphite backend
http://www.openiomon.org
GNU General Public License v3.0
7 stars 0 forks source link

CCI #5

Closed Trolls closed 2 years ago

Trolls commented 3 years ago

Hello,

The realtime works perfectly with OPS Center Anaylzer. But when I'm trying to use the tool with export tool, i got a issue.

When the deamon start the task nothing else appened after the creating of the horcm file conf: Please find logs:

2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/11 12:00:09 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc !

The horcm 934 file is created with the IP of the SVP, but no log appears for this instance.

Many thanks for help.

BR,

munokar commented 3 years ago

Hi,

so there is no logfile create under /opt/hds2graphite/logs/ containing the name of the storage box? Or do you mean there is not CCI Log created for the CCI instance?

Do you see an instance with that ID running when issuing ps -ef | grep horc ?

Regards

Muno

Trolls commented 3 years ago

Yes, the logs given provide from this log: /opt/hds2graphite/log/hds2graphite_58934.log:

2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/11 12:00:09 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/11 12:00:09 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/11 12:00:09 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc !

The horcm file horcm934 is no created, then no process horc

_[root@vm-grafana-6 etc]# pwd /opt/hds2graphite/cci/HORCM/etc [root@vm-grafana-6 etc]# ll total 772 -r--r--r-- 1 openiomon openiomon 33189 Aug 23 16:51 horcm.conf -r-xr--r-- 1 openiomon openiomon 648965 Aug 23 16:51 horcmgr -r--r--r-- 1 openiomon openiomon 501 Aug 23 16:51 Raidcom_Dic_Raid_RM_Patch.txt -r--r--r-- 1 openiomon openiomon 59999 Aug 23 16:51 Raidcom_Dic_Raid_RM.txt -r--r--r-- 1 openiomon openiomon 36221 Aug 23 16:51 Raidcom_Help_RaidRM.txt [root@vm-grafana-6 etc]#

Many thanks

munokar commented 3 years ago

Can you please do the follwoing:

  1. Edit the /etc/passwd and add a shell (e.g. /bin/bash) to the user openiomon by replacing to /bin/nologin
  2. As root run su - openiomon to swap user to openiomon
  3. Manually run the hds2graphite-worker.pl with options -conf /opt/hd2graphite/conf/hds2graphite.conf and -storagesystem <name you gave it in conf>

Would be nice to understand:

  1. Is the horm934.conf is created in manuall run.
  2. A CCI instance is actually started?

In most os the cases issues are related to permission problems. To fix this you could run a chown -R openiomon:openiomon /opt/hds2graphite/.

Trolls commented 3 years ago

Hello,

I made changes:

1 : [root@vm-grafana-6 hds2graphite]# cat /etc/passwd | grep open openiomon:x:993:991:openiomon module daemon user:/home/openiomon:/bin/bash

[root@vm-grafana-6 hds2graphite]# chown -R openiomon:openiomon /opt/hds2graphite/ [root@vm-grafana-6 hds2graphite]# su - openiomon Last login: Sun Oct 11 21:44:57 CEST 2020 on pts/0 su: warning: cannot change directory to /home/openiomon: No such file or directory -bash-4.2$ cd /opt/hds2graphite/bin/

-bash-4.2$ ./hds2graphite-worker.pl -conf /opt/hd2graphite/conf/hds2graphite.conf -storagesystem 58934 Can't locate Systemd/Daemon.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./hds2graphite-worker.pl line 30. BEGIN failed--compilation aborted at ./hds2graphite-worker.pl line 30.

-bash-4.2$ perl -i /opt/hds2graphite/lib/perl5/ hds2graphite-worker.pl -conf /opt/hd2graphite/conf/hds2graphite.conf -storagesystem 58934 -bash-4.2$

--> no cci instance are created on CCI folder, and no CCI is started. -bash-4.2$ ps -ef | grep horc openiom+ 18167 18103 0 09:21 pts/1 00:00:00 grep --color=auto horc -bash-4.2$

-bash-4.2$ ll cci/HORCM/etc/ total 772 -r--r--r-- 1 openiomon openiomon 33189 Aug 23 16:51 horcm.conf -r-xr--r-- 1 openiomon openiomon 648965 Aug 23 16:51 horcmgr -r--r--r-- 1 openiomon openiomon 501 Aug 23 16:51 Raidcom_Dic_Raid_RM_Patch.txt -r--r--r-- 1 openiomon openiomon 59999 Aug 23 16:51 Raidcom_Dic_Raid_RM.txt -r--r--r-- 1 openiomon openiomon 36221 Aug 23 16:51 Raidcom_Help_Raid_RM.txt

Many thanks for help.

BR,

munokar commented 3 years ago

Hi,

can you share the part of your configfile where the storage system is described?

Regards Muno

Trolls commented 3 years ago

Specify each storage system that should be monitored though hds2graphite Each specification block starts with [serial] [58934] subsystem_type = g1500 subsystem_name = 58934 svp_ip = 10.x.x.x cci_instance_number = 934 metric_configruation = /opt/hds2graphite/conf/metrics/g1500_metrics.conf exporttool_template = /opt/hds2graphite/conf/templates/g1500_template.txt if virtual devices with GAD should monitored on top of the phyical devices specify the VSM for virtual serial number 99999 and VSM name GAD_VSM_1 and the GAD Resource-Group-ID gad_vsm = serial, type, vsm name, resource group id gad_vsm = 11999,g1500,OPEN-VSM-01,1

Trolls commented 3 years ago

I removed "#"

munokar commented 3 years ago

Also it should not be an issue I never used a number as subsystem_name. Can you please try to specify something like VSP_58934? Then manually run perl -i /opt/hds2graphite/lib/perl5/ hds2graphite-worker.pl -conf /opt/hd2graphite/conf/hds2graphite.conf -storagesystem VSP_58934 -hours 1 Should not make a difference but lets see...

Trolls commented 3 years ago

I deregister the old subsystem and register with the news one:

[root@vm-grafana-6 log]# /opt/hds2graphite/bin/hds2graphite.pl -status VSP_58934 VSP_58934: Loaded: loaded (/usr/lib/systemd/system/hds2graphite-VSP_58934.service; enabled; vendor preset Active: active (running) since Tue 2020-10-13 09:52:15 CEST; 2min 20s ago Status: Waiting for next run... Last successful run: NEVER [root@vm-grafana-6 log]#

[root@vm-grafana-6 hds2graphite]# perl -i /opt/hds2graphite/lib/perl5/ hds2graphite-worker.pl -conf /opt/hd2graphite/conf/hds2graphite.conf -storagesystem VSP_58934 -hours 1 [root@vm-grafana-6 hds2graphite]# ll cci/HORCM/etc/ total 772 -r--r--r-- 1 openiomon openiomon 33189 Aug 23 16:51 horcm.conf -r-xr--r-- 1 openiomon openiomon 648965 Aug 23 16:51 horcmgr -r--r--r-- 1 openiomon openiomon 501 Aug 23 16:51 Raidcom_Dic_Raid_RM_Patch.txt -r--r--r-- 1 openiomon openiomon 59999 Aug 23 16:51 Raidcom_Dic_Raid_RM.txt -r--r--r-- 1 openiomon openiomon 36221 Aug 23 16:51 Raidcom_Help_Raid_RM.txt

but no instance horcm

munokar commented 3 years ago

Please provide the content of /opt/hds2graphite/log/hds2graphite_VSP_58934.log.

Trolls commented 3 years ago

2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:05 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/13 10:00:05 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:05 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/13 10:00:05 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/13 10:00:06 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/13 10:00:06 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc ! 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1630) main::initservice > Service is initialized... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service... 2020/10/13 10:00:06 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1638) main::servicestatus > Status message: Waiting for next run... is send to service... 2020/10/13 10:00:06 [INFO] (/opt/hds2graphite/bin/hds2graphite-worker.pl:970) main::createhorcmconf > Creating horcm934.conf in /opt/hds2graphite/cci/HORCM/etc !

munokar commented 3 years ago

Can you please try to manually create the file as user openiomon touch /opt/hds2graphite/cci/HORCM/etc/horcm934.conf? Is that working? Empty file should be sufficient...

This file could replace the hds2graphite-worker.pl - it contains addition trace output for creation of horcm.conf. Download and rename it and change the owner to openiomon:openiomon.

hds2graphite-worker.txt

It would also be nice to perform an echo $? to the perl return code...

Trolls commented 3 years ago

I re-installed CCI binaries with cciprepare, now it's ok:

2020/10/13 14:08:01 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1451) main::loglustats > No HSD Name found for HSD ID: 125 on port CL8-B for storage VSP_58934 2020/10/13 14:08:01 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1451) main::loglustats > No HSD Name found for HSD ID: 126 on port CL8-B for storage VSP_58934 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1496) main::closesocket > Closing Socket 127.0.0.1:42292 - 127.0.0.1:2003 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1490) main::initsocket > Opening connection 127.0.0.1:42294 => 127.0.0.1:2003 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1474) main::logscriptstats > Logging stats: runtime.loglustats for current run! 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1496) main::closesocket > Closing Socket 127.0.0.1:42294 - 127.0.0.1:2003 2020/10/13 14:08:01 [WARN] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1769) main:: > Omitting archiving of Export Tool data since Export Tool didn't run successful. Please check Export Tool and see error 2020/10/13 14:08:01 [ERROR] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1565) main::deleteexporttoolarchives > Failed to open directory /opt/hds2graphite/arch/VSP_58934 to check for archives to be deleted 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1490) main::initsocket > Opening connection 127.0.0.1:42296 => 127.0.0.1:2003 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1474) main::logscriptstats > Logging stats: runtime.total for current run! 2020/10/13 14:08:01 [DEBUG] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1496) main::closesocket > Closing Socket 127.0.0.1:42296 - 127.0.0.1:2003 2020/10/13 14:08:01 [TRACE] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1652) main::alive > Heartbeat is send to watchdog of service...

Trolls commented 3 years ago

i need to install export tool on the server, and check if OK.

Do you know if i need to import dashboard on Grafana, or it's automatically import?

munokar commented 3 years ago

The export tool need to extracted in the g1500 subfolder and the ownership of all files need to be changed to openiomon:openiomon. Since export tool comes with the VSP CDs provided by Hitachi it is for legal reasons not included into the RPMs.

The dashboards need to bin installed either by using the import_grafana_db.sh or manually by copy & paste over the UI. If you want to use the script you need to generate a Grafana API key and write it in the file with the name ~/.grafanakey located directly in the home-dir of the user you are going to use for import (e.g. root).

Trolls commented 3 years ago

ok many thanks Munokar, i'll try this and keep you inform :)

Trolls commented 3 years ago

What do you think about this error ?

2020/10/13 17:07:18 [WARN] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1769) main:: > Omitting archiving of Export Tool data since Export Tool didn't run successful. Please check Export Tool and see error messages above! 2020/10/13 17:07:18 [ERROR] (/opt/hds2graphite/bin/hds2graphite-worker.pl:1565) main::deleteexporttoolarchives > Failed to open directory /opt/hds2graphite/arch/VSP_58934 to check for archives to be deleted!

i copied all files in the g1500 folder:

[root@vm-grafana-6 g1500]# ll total 72 -rw-r--r-- 1 openiomon openiomon 1779 Oct 13 17:06 58934.txt -rw-rw-rw- 1 openiomon openiomon 4515 Oct 24 2014 command.txt -rwxrwxrwx 1 openiomon openiomon 146 Jun 24 2015 delUnix.bat -rwxrwxrwx 1 openiomon openiomon 141 Jun 24 2015 delWin.bat -rw-r--r-- 1 openiomon openiomon 40960 Oct 13 14:19 export.tar drwxrwxrwx 2 openiomon openiomon 34 Jul 19 2018 lib -rw-rw-rw- 1 openiomon openiomon 1253 Oct 24 2014 RsdRmiSSLkey -rwxrwxrwx 1 openiomon openiomon 170 Oct 13 17:01 runUnix.bat -rwxrwxrwx 1 openiomon openiomon 167 Jun 1 2015 runWin.bat

[root@vm-grafana-6 g1500]# cat runUnix.bat

! /bin/sh

java -classpath "./lib/JSanExportLoader.jar" -Del.tool.Xmx=536870912 -Dmd.command=58934.txt -Del.logpath=log -Dmd.rmitimeout=20 sanproject.getexptool.RJElMain [root@vm-grafana-6 g1500]#

there is no log for export tool directory

Many thanks.

munokar commented 3 years ago

Please provide content of the 58934.txt (with out sensitive information...). As well see if there is an error in the log before the archiving actions of exporttool... There should be entries for starting the exporttool and also for error with java etc... There should als be a exporttool log in: /opt/hds2graphite/log/58934/ - there you might see potential issues with exporttool itself...

Trolls commented 3 years ago

i got this error:

2020/10/13 18:08:19 command=java -classpath ./lib/0A37698C/JSanExport.jar:./lib/0A37698C/JSanRmiApiEx.jar:./lib/0A37698C/JSanRmiApiSx.jar:./lib/0A37698C/JSanRmiServerEx.jar:./lib/0A37698C/JSanRmiServerSx.jar:./lib/0A37698C/JSanRmiServerUx.jar -Xmx536870912 -Dmd.logpath=/opt/hds2graphite/log/58934 -Dmd.command=/opt/hds2graphite/g1500/58934.txt -Dmd.rmitimeout=20 sanproject.getmondat.RJMdMain 2020/10/13 18:08:19 start Export Tool-------------------. 2020/10/13 18:08:22 finish Export Tool-------------------. 2020/10/13 18:08:22 exit code=2 2020/10/13 18:08:22 done. 2020/10/13 18:08:22 Delete temporary directory. 2020/10/13 18:08:22 delete JSanExport.jar 2020/10/13 18:08:22 delete JSanRmiApiEx.jar 2020/10/13 18:08:22 delete JSanRmiApiSx.jar 2020/10/13 18:08:22 delete JSanRmiServerEx.jar 2020/10/13 18:08:22 delete JSanRmiServerSx.jar 2020/10/13 18:08:22 delete JSanRmiServerUx.jar 2020/10/13 18:08:22 delete ./lib/0A37698C 2020/10/13 18:08:22 done. 2020/10/13 18:08:22 ExportLoader finish. exitcode=2

munokar commented 3 years ago

Hm, that looks like an issue with the exporttool itself... Have you tried to run it manually with the runUnix. You either need to rename the 58934.txt to command.txt or change the runUnix before running it manually.

mamoep commented 2 years ago

information provided