未对齐的内存访问造成brise编译失败

[GoogleCodeExporter]

在linux arm/mips/sparc架构下编译brise的过程中[1]，
运行rime_deployer的时候，出现SIGBUS，导致程序中断。

[1] https://buildd.debian.org/status/package.php?p=brise&suite=unstable

这里应该是正在luna_pinyin.table.bin文件偏移0xdca处
写入num_syllables。而它的类型是size_t，是四字节的，
所以产生了未对齐访问。

以下是sparc上面的backtrace
（使用了g++-4.9编译rime-1.1）

$ gdb rime_deployer
GNU gdb (GDB) 7.6.2 (Debian 7.6.2-1.1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/rime_deployer...Reading symbols from 
/usr/lib/debug/.build-id/33/a0ab4c492de00481dff12c37037b20b8d26b64.debug...done.
done.
(gdb) run --build  brise/data
Starting program: /usr/bin/rime_deployer --build  brise/data
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/sparc-linux-gnu/libthread_db.so.1".

Program received signal SIGBUS, Bus error.
0xf7e795bc in rime::Table::BuildHeadIndex (this=this@entry=0x37b4c, 
vocabulary=..., num_syllables=num_syllables@entry=422)
    at librime/src/dict/table.cc:300
300       for (const auto& v : vocabulary) {
(gdb) x/i $pc
=> 0xf7e795bc <rime::Table::BuildHeadIndex(rime::Vocabulary const&, unsigned 
int)+188>: st  %i2, [ %l2 + %l1 ]
(gdb) p $l2
$1 = -243040256
(gdb) p/x $l2
$2 = 0xf1838000
(gdb) p/x $l1
$3 = 0xdca
(gdb) x/i $npc
   0xf7e795c0 <rime::Table::BuildHeadIndex(rime::Vocabulary const&, unsigned int)+192>: add  %fp, -12, %i2
(gdb) p $i2
$4 = 422
(gdb) bt
#0  0xf7e795bc in rime::Table::BuildHeadIndex (this=this@entry=0x37b4c, 
vocabulary=..., num_syllables=num_syllables@entry=422)
    at librime/src/dict/table.cc:300
#1  0xf7e79ba0 in rime::Table::Build (this=0x37b4c, syllabary=..., 
vocabulary=..., num_entries=<optimized out>,
    dict_file_checksum=dict_file_checksum@entry=4070034724) at librime/src/dict/table.cc:281
#2  0xf7ec210c in rime::DictCompiler::BuildTable (this=this@entry=0xffffd534, 
settings=settings@entry=0xffffd268, dict_files=...,
    dict_file_checksum=dict_file_checksum@entry=4070034724) at librime/src/dict/dict_compiler.cc:153
#3  0xf7ec3af0 in rime::DictCompiler::Compile (this=this@entry=0xffffd534, 
schema_file=...) at librime/src/dict/dict_compiler.cc:95
#4  0xf7f6d76c in rime::SchemaUpdate::Run (this=0x316e0, deployer=<optimized 
out>) at librime/src/lever/deployment_tasks.cc:301
#5  0xf7f6c09c in rime::WorkspaceUpdate::Run (this=this@entry=0xffffd70c, 
deployer=deployer@entry=0x2cf40)
    at librime/src/lever/deployment_tasks.cc:167
#6  0x00012240 in main (argc=1, argv=0xffffd7fc) at 
librime/tools/rime_deployer.cc:94

Original issue reported on code.google.com by culu....@gmail.com on 29 May 2014 at 7:10

[GoogleCodeExporter]

luna_pinyin.table.bin的相关部分：

on amd64:
$ hd /usr/share/rime-data/luna_pinyin.table.bin
[...]
00000da0  7a 68 75 6e 00 7a 68 75  6f 00 7a 69 00 7a 6f 6e  |zhun.zhuo.zi.zon|
00000db0  67 00 7a 6f 75 00 7a 75  00 7a 75 61 6e 00 7a 75  |g.zou.zu.zuan.zu|
00000dc0  69 00 7a 75 6e 00 7a 75  6f 00 a6 01 00 00 0c 00  |i.zun.zuo.......|
00000dd0  00 00 c4 13 00 00 53 14  00 00 9d 00 00 00 fb 80  |......S.........|
00000de0  00 00 83 88 00 00 88 00  00 00 03 00 01 00 87 06  |................|
[...]

on sparc:
$ hd data/luna_pinyin.table.bin   # 
这里只有SIGBUS出现之前已写好的部分
00000da0  7a 68 75 6e 00 7a 68 75  6f 00 7a 69 00 7a 6f 6e  |zhun.zhuo.zi.zon|
00000db0  67 00 7a 6f 75 00 7a 75  00 7a 75 61 6e 00 7a 75  |g.zou.zu.zuan.zu|
00000dc0  69 00 7a 75 6e 00 7a 75  6f 00 00 00 00 00 00 00  |i.zun.zuo.......|
00000dd0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0296cec0

Original comment by culu....@gmail.com on 29 May 2014 at 7:13

[GoogleCodeExporter]

Original comment by chen....@gmail.com on 30 May 2014 at 3:30

• Changed state: Fixed

[GoogleCodeExporter]

原先有個arm分支解決了數據對齊問題，最近才合併到develop分�
��。
請務必使用 
https://github.com/lotem/librime/commit/a61c8cec7b9ffc47223ae636eaf786cd77d411be
 包含這個修改的代碼。其生成的 .bin 文件與 x86/amd46 不同。

Original comment by chen....@gmail.com on 30 May 2014 at 4:01

[GoogleCodeExporter]

谢谢！下次我用develop分支试试。

Original comment by culu....@gmail.com on 30 May 2014 at 8:03

[GoogleCodeExporter]

你好，我用 develop 分支（现在是 
30979c8）试了一下，还是有类似的问题。可能因为原来的修改�
��对arm架构有效？这样的话，在 sparc 和 mips 
上面还是会失败。

在 sparc 上面的 backtrace：

$ gdb librime/debug-build/bin/rime_deployer                                     

GNU gdb (GDB) 7.6.2 (Debian 7.6.2-1.1+b1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from librime/debug-build/bin/rime_deployer...done.
(gdb) run --compile luna_pinyin.schema.yaml
Starting program: librime/debug-build/bin/rime_deployer --compile 
luna_pinyin.schema.yaml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/sparc-linux-gnu/libthread_db.so.1".
E0605 00:14:43.917161 20878 text_db.cc:136] Error opening db './essay.txt' 
read-only.

Program received signal SIGBUS, Bus error.
0xf7cfbaa4 in rime::MappedFile::CreateArray<rime::table::HeadIndexNode> 
(this=0x40514, array_size=422)
    at librime/include/rime/dict/mapped_file.h:171
171       ret->size = array_size;
(gdb) bt
#0  0xf7cfbaa4 in rime::MappedFile::CreateArray<rime::table::HeadIndexNode> 
(this=0x40514, array_size=422)
    at librime/include/rime/dict/mapped_file.h:171
#1  0xf7cf8c5c in rime::Table::BuildHeadIndex (this=0x40514, vocabulary=..., 
num_syllables=422) at librime/src/dict/table.cc:296
Python Exception <class 'ValueError'> Cannot find type const 
rime::Syllabary::_Rep_type: 
#2  0xf7cf89b0 in rime::Table::Build (this=0x40514, syllabary=std::set with 422 
elements, vocabulary=..., num_entries=70365, dict_file_checksum=4070034724)
    at librime/src/dict/table.cc:281
#3  0xf7d29c78 in rime::DictCompiler::BuildTable (this=0xffffd4f8, 
settings=0xffffd400, dict_files=std::vector of length 1, capacity 1 = {...}, 
    dict_file_checksum=4070034724) at librime/src/dict/dict_compiler.cc:176
#4  0xf7d29108 in rime::DictCompiler::Compile (this=0xffffd4f8, 
schema_file="./luna_pinyin.schema.yaml")
    at librime/src/dict/dict_compiler.cc:118
#5  0xf7e54aa8 in rime::SchemaUpdate::Run (this=0xffffd6ac, deployer=0x31178) 
at librime/src/lever/deployment_tasks.cc:301
#6  0x00018314 in main (argc=1, argv=0xffffd7bc) at 
librime/tools/rime_deployer.cc:111
(gdb) x/i $pc
=> 0xf7cfbaa4 
<rime::MappedFile::CreateArray<rime::table::HeadIndexNode>(unsigned int)+104>:  
  st  %g2, [ %g1 ]
(gdb) p/x $g1
$1 = 0xf47d8dca
(gdb) p/x $g2
$2 = 0x1a6

Original comment by culu....@gmail.com on 5 Jun 2014 at 4:22

[GoogleCodeExporter]

你說得對。只判斷了 _arm_

Original comment by chen....@gmail.com on 5 Jun 2014 at 8:54

• Changed state: Accepted

[GoogleCodeExporter]

問題解決了嘛？

Original comment by chen....@gmail.com on 16 Jun 2014 at 9:32

[GoogleCodeExporter]

还没有。。。

Original comment by culu....@gmail.com on 19 Jun 2014 at 3:50

[GoogleCodeExporter]

有測試環境的話不妨一試。
include/rime/dict/mapped_file.h
<<<
#if defined(__arm__)
===
#if defined(__arm__) || defined(__sparc__) || defined(__mips__)
>>>
這樣也許就行了。
另一處檢測__arm__的地方是table.h，解決float類型導致的bus 
error，我猜測只是ARM平臺才需要改。但沒有測試環境也拿不準
。

Original comment by chen....@gmail.com on 19 Jun 2014 at 3:57

[GoogleCodeExporter]

你好，

我在现在的 develop branch (b2dfca7) 修改了 mapped_file.h：

diff --git a/include/rime/dict/mapped_file.h b/include/rime/dict/mapped_file.h
index 6d318cd..4099089 100644
--- a/include/rime/dict/mapped_file.h
+++ b/include/rime/dict/mapped_file.h
@@ -129,7 +129,7 @@ public:

 // member function definitions

-# if defined(__arm__)
+# if defined(__arm__) || defined(__sparc__) || defined(__mips__)
 # define RIME_ALIGNED(size, T) ((size + alignof(T) - 1) & ~(alignof(T) - 1))
 # else
 # define RIME_ALIGNED(size, T) (size)

不过还是有同样的 SIGBUS，导致 rime_deployer --compile 
cangjie5.schema.yaml 失败。

Original comment by culu....@gmail.com on 5 Jul 2014 at 12:03

[GoogleCodeExporter]

附 sparc 上的 backtrace 等相关信息：

(gdb) run --compile cangjie5.schema.yaml
Starting program: rime_deployer --compile cangjie5.schema.yaml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/sparc-linux-gnu/libthread_db.so.1".
E0705 01:36:19.626600 20976 table.cc:207] invalid metadata.
E0705 01:36:19.631078 20976 text_db.cc:136] Error opening db './essay.txt' 
read-only.

Program received signal SIGBUS, Bus error.
0xf7cf7bcc in rime::MappedFile::CreateArray<rime::table::HeadIndexNode> 
(this=0x3ceac, array_size=64997)
    at librime/include/rime/dict/mapped_file.h:171
171       ret->size = array_size;
(gdb) p array_size
$1 = 64997
(gdb) bt
#0  0xf7cf7bcc in rime::MappedFile::CreateArray<rime::table::HeadIndexNode> 
(this=0x3ceac, array_size=64997)
    at librime/include/rime/dict/mapped_file.h:171
#1  0xf7cf4d7c in rime::Table::BuildHeadIndex (this=0x3ceac, vocabulary=..., 
num_syllables=64997) at librime/src/dict/table.cc:296
Python Exception <type 'exceptions.ValueError'> Cannot find type const 
rime::Syllabary::_Rep_type: 
#2  0xf7cf4ad0 in rime::Table::Build (this=0x3ceac, syllabary=std::set with 
64997 elements, vocabulary=..., num_entries=79319, 
    dict_file_checksum=1952737959) at librime/src/dict/table.cc:281
Python Exception <type 'exceptions.TypeError'> instance has no next() method: 
#3  0xf7d25dc0 in rime::DictCompiler::BuildTable (this=0xffffd4f8, 
settings=0xffffd400, dict_files=std::vector of length 1, capacity 1, 
    dict_file_checksum=1952737959) at librime/src/dict/dict_compiler.cc:176
#4  0xf7d25250 in rime::DictCompiler::Compile (this=0xffffd4f8, 
schema_file="./cangjie5.schema.yaml")
    at librime/src/dict/dict_compiler.cc:118
#5  0xf7e51620 in rime::SchemaUpdate::Run (this=0xffffd6ac, deployer=0x31178) 
at librime/src/lever/deployment_tasks.cc:300
#6  0x00018314 in main (argc=1, argv=0xffffd7bc) at 
librime/tools/rime_deployer.cc:111
(gdb) p &(ret->size)
$2 = (unsigned int *) 0xf45202c2
(gdb) x/i $pc
=> 0xf7cf7bcc 
<rime::MappedFile::CreateArray<rime::table::HeadIndexNode>(unsigned int)+104>:  
  st  %g2, [ %g1 ]
(gdb) p/x $g1
$3 = 0xf45202c2
(gdb) p/x $g2
$4 = 0xfde5
(gdb) 

Original comment by culu....@gmail.com on 5 Jul 2014 at 5:46

[GoogleCodeExporter]

那只好再試試在 include/rime/dict/table.h 
做同樣的修改，讓Weight定義成double類型。
之前arm用家說這也會導致bus error。

Original comment by chen....@gmail.com on 5 Jul 2014 at 5:53

[GoogleCodeExporter]

嗯，修改了 table.h 之后，还是有同样的 SIGBUS。

backtrace 显示出错的位置仍然是
0xf7cf7be8 in rime::MappedFile::CreateArray<rime::table::HeadIndexNode> 
(this=0x3ceac, array_size=64997)
    at /mnt/sdb2/pkg2/upstream/librime/include/rime/dict/mapped_file.h:171
171       ret->size = array_size;

对 table.h 的修改：
diff --git a/include/rime/dict/table.h b/include/rime/dict/table.h
index 18840c8..17bf17d 100644
--- a/include/rime/dict/table.h
+++ b/include/rime/dict/table.h
@@ -27,7 +27,7 @@ using SyllableId = int32_t;

 using Code = List<SyllableId>;

-#if defined(__arm__)
+#if defined(__arm__) || defined(__sparc__) || defined(__mips__)
 using Weight = double;
 #else
 using Weight = float;

Original comment by culu....@gmail.com on 6 Jul 2014 at 6:25