search

grycap / clues

CLUES: an energy management system for HPC Clusters and Cloud infrastructures.
http://www.grycap.upv.es/clues
GNU General Public License v3.0
24 stars 7 forks source link

Error connecting clues to mysql #7

Open tkald opened 8 years ago

tkald commented 8 years ago

Can't start clues with mysql database. Seems like string parsing is broken in cpyutils or wrong sample format is provided in configuration file.

Connection string in clues2.cfg file: DB_CONNECTION_STRING=mysql://clues:clpAss@cluesdb/clues

In log I get errors: ...

WARNING: Couldn't write lextab module 'lextab'. [Errno 20] Not a directory: '/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/lextab.py' WARNING: Couldn't create 'parsetab'. [Errno 20] Not a directory: '/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/parsetab.py' ...

Traceback (most recent call last): File "/usr/bin/cluesserver", line 569, in main_loop() File "/usr/bin/cluesserver", line 560, in main_loop CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init self._db_system = DBSystem.create_from_connection_string() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string return DBSystem(connection_string) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 61, in init self._db = cpyutils.db.DB.create_from_string(connection_string) File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 80, in create_from_string NameError: global name 'db' is not defined

If I use ip address for host in connection string I get last 2 lines of same error:

File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 95, in create_from_string cpyutils.db.MalformedConnectionString: Connection string format not recognised

Latest master branches of clues and cpyutils on Ubuntu 16.04

micafer commented 8 years ago

Hi @tkald, It is a bug of the cpyutils package (grycap/cpyutils#3).

micafer commented 8 years ago

Hi @tkald, We have commited a fix to the cpyutils error. Coul you try to download the last version and test it again?

tkald commented 8 years ago

Something's still wrong.

If I use connection string with ip address, I get

File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 96, in create_from_string cpyutils.db.MalformedConnectionString: ('Connection string format not recognised')

connection_parameters returns None Regex issue?

If I use DB_CONNECTION_STRING=mysql://clues:clpAss@cluesdb/clues I get:

Traceback (most recent call last): File "/usr/bin/cluesserver", line 569, in main_loop() File "/usr/bin/cluesserver", line 560, in main_loop CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init self._db_system = DBSystem.create_from_connection_string() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string return DBSystem(connection_string) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 62, in init self._create_db() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 78, in _createdb result1, , _ = self._db.sql_query("CREATE TABLE IF NOT EXISTS hostdata(name varchar(128) PRIMARY KEY, enabled bool)", True) File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query AttributeError: DB_mysql instance has no attribute '_conn_string'

micafer commented 8 years ago

Hi @tkald,

As you said the regex was not prepaired for IP strings. Also there was a bug in the DB connection. I have uploaded a fix in cpyutils.

tkald commented 8 years ago

File "/usr/bin/cluesserver", line 569, in main_loop() File "/usr/bin/cluesserver", line 560, in main_loop CLUES_DAEMON = CluesDaemon(PLATFORM, active_schedulers) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 185, in init self._db_system = DBSystem.create_from_connection_string() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 57, in create_from_connection_string return DBSystem(connection_string) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 62, in init self._create_db() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 78, in _createdb result1, , _ = self._db.sql_query("CREATE TABLE IF NOT EXISTS hostdata(name varchar(128) PRIMARY KEY, enabled bool)", True) File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query AttributeError: 'NoneType' object has no attribute 'cursor'

micafer commented 8 years ago

Hi @tkald,

Sorry for this failure it was may fault in one line. Now it should work.

tkald commented 8 years ago

Yap, now I have 2 tables in mysql db. No data though in those tables. Nothing in log either. Maybe misconfiguration in my conf.

/usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Table 'hostdata' already exists /usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Table 'host_monitoring' already exists [SERVER];DEBUG;2016-10-31 13:17:27,110;1477912647.111;CLUES server starting [CLUES];DEBUG;2016-10-31 13:17:27,111;1477912647.111;monitoring jobs and nodes at the same time [ONE];DEBUG;2016-10-31 13:17:27,119;OpenNebula version: 5.2.0 [CLUES];DEBUG;2016-10-31 13:17:27,142;1477912647.142; First monitorization of LRMS: List of nodes: [NODE "r410-1"] state: off (since 1477912647), 3600/3600 (slots) - ID: 001 @1477912647 [NODE "vml-vm1"] state: used (since 1477912647), 0/3600 (slots) - ID: 002 @1477912647 [NODE "vml-vm2"] state: off (since 1477912647), 3600/3600 (slots) - ID: 003 @1477912647 [CLUES];DEBUG;2016-10-31 13:17:27,915;1477912647.916;first monitorisation of jobs in LRMS

micafer commented 8 years ago

Hi @tkald,

No information is stored in the DB untill there is any change in the information: A node state changes, a node is disabled, etc. If the tables has been created correctly the DB access is correctly configured and working. Try to disable a node and you will see as some information is stored in de DB.

tkald commented 8 years ago

Yes, got some data after changing host status. Data type mismatches though: /usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Field 'x' doesn't have a default value /usr/local/lib/python2.7/dist-packages/cpyutils-0.24-py2.7.egg/cpyutils/db.py:178: Warning: Incorrect integer value: 'True' for column 'enabled' at row 1

tkald commented 8 years ago

More related errors:

[CLUES];DEBUG;2016-10-31 15:20:27,135;1477920027.136;node vml-vm2 changed its state Traceback (most recent call last): File "/usr/bin/cluesserver", line 569, in main_loop() File "/usr/bin/cluesserver", line 566, in main_loop CLUES_DAEMON.loop(options.RT_MODE) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 557, in loop cpyutils.eventloop.get_eventloop().loop() File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 299, in loop File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 138, in call File "build/bdist.linux-x86_64/egg/cpyutils/eventloop.py", line 101, in call File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 351, in _monitor_lrms_nodes_and_jobs self._monitor_lrms_nodes() File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 299, in _monitor_lrms_nodes self._db_system.store_node_info(self._lrms_nodelist[n_id]) File "/usr/local/lib/python2.7/dist-packages/clueslib/cluesd.py", line 106, in store_node_info , True) File "build/bdist.linux-x86_64/egg/cpyutils/db.py", line 190, in sql_query _mysql_exceptions.IntegrityError: (1062, "Duplicate entry '0' for key 'PRIMARY'") cluesd.service: Main process exited, code=exited, status=1/FAILURE

tkald commented 8 years ago

@micafer

No information is stored in the DB untill there is any change in the information: A node state changes, a node is disabled, etc.

I suppose monitoring host's free slots should change data. Booted some extra vm-s, no change in mysql table. I think this is unrelated to mysql though. Shall we open new issue?