fetchrow_hashref failed ... Table ... is read only at .../SyncMysql.pm line 103

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Fresh OS install, fully patched.
2. Followed ELSA Quickstart guide.
3. Receive an error from the web interface when I enter "seq", no quotes.

What is the expected output? What do you see instead?
I expected to see some test data but instead receive:

Errors: node 127.0.0.1 got error DBD::mysql::st fetchrow_hashref failed: Table 
'syslogs_index_1' is read only at /usr/local/elsa/web/lib/SyncMysql.pm line 103.

What version of the product are you using? On what operating system?
CentOS   6.4
Sphinx   Sphinx 2.1.1-id64-beta (rel21-r3701)
Date     2013-07-08 18:32:21 -0400 (Mon, 08 Jul 2013)
Author   mcholste
Rev  960

Please provide any additional information below.

There are two errors that appear in /data/elsa/log/web.log.

* ERROR [2013/07/09 15:37:27] /usr/local/elsa/web/lib/SyncMysql.pm (111) 
SyncMysql::multi_query 4082 [undef]
Query: SELECT syslog_data.syslogs_index_1.id,
timestamp,
INET_NTOA(host_id) AS host, program, class_id, class, msg,
i0, i1, i2, i3, i4, i5, s0, s1, s2, s3, s4, s5
FROM syslog_data.syslogs_index_1
LEFT JOIN syslog.programs ON syslog_data.syslogs_index_1.program_id=programs.id
LEFT JOIN syslog.classes ON syslog_data.syslogs_index_1.class_id=classes.id
WHERE syslog_data.syslogs_index_1.id IN 
(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?
,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) with values 
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,5
7,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100 got error DBD::mysql::st 
fetchrow_hashref failed: Table 'syslogs_index_1' is read only at 
/usr/local/elsa/web/lib/SyncMysql.pm line 103.

* ERROR [2013/07/09 15:37:27] /usr/local/elsa/web/lib/API.pm (2321) 
API::__ANON__ 4082 [undef]
node 127.0.0.1 got error DBD::mysql::st fetchrow_hashref failed: Table 
'syslogs_index_1' is read only at /usr/local/elsa/web/lib/SyncMysql.pm line 103.

Original issue reported on code.google.com by jsn.l...@gmail.com on 9 Jul 2013 at 7:44

[GoogleCodeExporter]

That means that the syslogs_index_1 table is corrupted, which is often because 
of a full disk.  Can you verify that you are not out of space?

To repair the table, execute:

mysql -uroot -e "REPAIR TABLE syslog_data.syslogs_index_1"

Original comment by mchol...@gmail.com on 10 Jul 2013 at 9:48

• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks for the prompt reply.

Here is the disk usage and mysql command output.

[root@elsa-dev ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_elsadev-lv_root
                       16G  2.6G   13G  18% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   73M  387M  16% /boot

[root@elsa-dev ~]# mysql -uroot -e "REPAIR TABLE
syslog_data.syslogs_index_1"
+-----------------------------+--------+----------+-----------------------------
---------+
| Table                       | Op     | Msg_type | Msg_text
              |
+-----------------------------+--------+----------+-----------------------------
---------+
| syslog_data.syslogs_index_1 | repair | Error    | Table 'syslogs_index_1'
is read only |
| syslog_data.syslogs_index_1 | repair | error    | Corrupt
             |
+-----------------------------+--------+----------+-----------------------------
---------+
[root@elsa-dev ~]#

If I check all the tables in the syslog_data database, there is another
that appears to be corrupt.

[root@elsa-dev ~]# mysqlcheck -c syslog_data -u root
syslog_data.syslogs_archive_1
Error    : Table 'syslogs_archive_1' is read only
error    : Corrupt
syslog_data.syslogs_index_1
Error    : Table 'syslogs_index_1' is read only
error    : Corrupt
[root@elsa-dev ~]#

I receive the same output when attempting to repair the archive table.

On Wed, Jul 10, 2013 at 5:49 PM, <
enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by jsn.l...@gmail.com on 11 Jul 2013 at 12:20

[GoogleCodeExporter]

That's too bad, it sounds like you'll have to drop the table.  If you can live 
without the logs, try to issue a drop table command on syslogs_index_1 and 
syslogs_archive_1, then restart syslog-ng.

Original comment by mchol...@gmail.com on 12 Jul 2013 at 9:28

[GoogleCodeExporter]

Hi,

I came across the same error on a fresh install using the install.sh script 
from trunk (on CentOS 6.4 minimal).
There is no need to drop and recreate the tables, this is a selinux issue with 
the linked tables in /data/elsa/mysql/

It can be fixed with: chcon --reference=/var/lib/mysql/test -R 
"/data/elsa/mysql"
or disabling selinux of course.

[root@elsa-test mysql]# getenforce
Enforcing
[root@elsa-test mysql]# mysql -uroot -e "REPAIR TABLE 
syslog_data.syslogs_index_1"
+-----------------------------+--------+----------+-----------------------------
--------------------------------------+
| Table                       | Op     | Msg_type | Msg_text                    
                                      |
+-----------------------------+--------+----------+-----------------------------
--------------------------------------+
| syslog_data.syslogs_index_1 | repair | error    | Can't create new tempfile: 
'/data/elsa/mysql/syslogs_index_1.TMD' |
| syslog_data.syslogs_index_1 | repair | status   | Operation failed            
                                      |
+-----------------------------+--------+----------+-----------------------------
--------------------------------------+
[root@elsa-test mysql]# chcon --reference=/var/lib/mysql/test -R 
"/data/elsa/mysql"
[root@elsa-test mysql]# mysql -uroot -e "REPAIR TABLE 
syslog_data.syslogs_index_1"
+-----------------------------+--------+----------+----------+
| Table                       | Op     | Msg_type | Msg_text |
+-----------------------------+--------+----------+----------+
| syslog_data.syslogs_index_1 | repair | status   | OK       |
+-----------------------------+--------+----------+----------+
[root@elsa-test mysql]#

I am attaching a snippet from audit.log (grep for mysql avc denied)

type=AVC msg=audit(1375508351.304:616): avc:  denied  { write } for  pid=1547 
comm="mysqld" path="/data/elsa/mysql/syslogs_index_1.MYI" dev=dm-1 ino=1969676 
scontext=unconfined_u:system_r:mysqld_t:s0 
tcontext=system_u:object_r:httpd_log_t:s0 tclass=file
type=AVC msg=audit(1375508377.813:617): avc:  denied  { read } for  pid=1553 
comm="mysqld" path="/data/elsa/mysql/syslogs_archive_1.MYD" dev=dm-1 
ino=1969698 scontext=unconfined_u:system_r:mysqld_t:s0 
tcontext=system_u:object_r:httpd_log_t:s0 tclass=file

It seems that set_node_mysql function didn't run on install for some reason, or 
it was not able to chcon.

Original comment by Just4N...@gmail.com on 3 Aug 2013 at 7:47

Attachments:

• elsa.mysql.selinux.txt

[GoogleCodeExporter]

Thanks!  

If the database and schema already exist, it will bail out of set_node_mysql, 
which might be why the chcon command got missed.

Original comment by mchol...@gmail.com on 13 Aug 2013 at 11:07

• Changed state: Done

[GoogleCodeExporter]

Just wanted to thank Just4N and confirm that the issue exists with CentOS 4.x 
minimal and selinux. The proposed fix works.

Original comment by lib...@gmail.com on 18 Aug 2013 at 1:02

[GoogleCodeExporter]

I ran into the same issue with CentOS 6.6 after I enabled SELINUX.  chcon 
--reference=/var/lib/mysql/test -R "/data/elsa/mysql"  fixed my issues.  Thanks.

Original comment by stevetu...@gmail.com on 13 Mar 2015 at 4:37