egli commented 7 years ago

When compiling a table with includes that are more than 6 levels deep the JVM crashes with SIGBUS

According to the internets SIGBUS is generally related to memory problems, so the first thought is of course to blame liblouis and to assume that it is writing over some JVM stack frames.

Options

liblouis is somehow corrupting the memory
The Java bindings have a problem
some other problem

Analysis

liblouis is corrupting memory

[X] Port the test to C so we can analyze the memory.
[X] Run Valgrind on the C test case
[X] Analyze C test case with Electric Fence

So @bertfrees ported the test case to C, see https://github.com/liblouis/liblouis/blob/test_deep_include/tests/deep_include.c. When running Valgrind there is a memory leak around the usage of the DefaultTableResolver. Electric Fence does not indicate any problems whatsoever.

The problem persists on the JVM when using a liblouis where the memory leak is fixed

Java bindings have a problem

It is not clear where there could be a problem here. The test case simply invokes get_Table through a very simple API straight down to C.

[x] Run gdb on the Java test case: doesn't give much info except that the SIGBUS happens in compileUplow on the line int haveLowerDots = 0 (may not mean anything).
[ ] Run valgrind on the Java test case. Maybe this will give us a hint if JVM or JNA are doing something fishy.

Other problem

Some findings from a google search indicate that the problem could be within the JVM and the use of mmaped files

[x] Try to reproduce the problem on an other machine

On a Linux machine the problem doesn't occur

bertfrees commented 7 years ago

This is the stack trace when the SIGBUS happens. "3324" is the int haveLowerDots = 0; line.

compileUplow + 38 (compileTranslationTable.c:3324)
compileRule + 12935 (compileTranslationTable.c:4745)
compileFile + 151 (compileTranslationTable.c:5121)
includeFile + 331 (compileTranslationTable.c:5170)
compileRule + 528 (compileTranslationTable.c:3808)
compileFile + 151 (compileTranslationTable.c:5121)
includeFile + 331 (compileTranslationTable.c:5170)
compileRule + 528 (compileTranslationTable.c:3808)
compileFile + 151 (compileTranslationTable.c:5121)
includeFile + 331 (compileTranslationTable.c:5170)
compileRule + 528 (compileTranslationTable.c:3808)
compileFile + 151 (compileTranslationTable.c:5121)
includeFile + 331 (compileTranslationTable.c:5170)
compileRule + 528 (compileTranslationTable.c:3808)
compileFile + 151 (compileTranslationTable.c:5121)
includeFile + 331 (compileTranslationTable.c:5170)
compileRule + 528 (compileTranslationTable.c:3808)
compileFile + 151 (compileTranslationTable.c:5121)
compileTranslationTable + 413 (compileTranslationTable.c:5216)
getTable + 362 (compileTranslationTable.c:5280)
lou_getTable + 21 (compileTranslationTable.c:5328)

egli commented 7 years ago

Looks like I cannot reproduce the problem on Linux:

eglic@wharton:~/src/pipeline$ make compile
Recomputing modules to include in the build...
 wagon http use multi threaded http connection manager maxPerRoute 20, max total 40
╔════════╗
║ GRADLE ║
╚════════╝
Looking for changes...
All modules are up to date
╔═══════╗
║ MAVEN ║
╚═══════╝
Looking for changes...
--> ./libs/liblouis
--> ./modules/braille/maven/bom
--> ./modules/braille/pipeline-braille-utils/common-utils
--> ./modules/braille/pipeline-braille-utils/css-utils/css-utils
--> ./modules/braille/pipeline-braille-utils/dotify-utils/dotify-formatter
...
...                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building liblouis-java 3.1.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been relocated to commons-io:commons-io:jar:1.3.2
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ liblouis-java ---
[INFO] Deleting /home/eglic/src/pipeline/libs/liblouis-java/target
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ liblouis-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/eglic/src/pipeline/libs/liblouis-java/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ liblouis-java ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 10 source files to /home/eglic/src/pipeline/libs/liblouis-java/target/classes
[INFO] 
[INFO] --- nar-maven-plugin:3.4.0:nar-validate (unpack-liblouis) @ liblouis-java ---
[INFO] Using AOL: amd64-Linux-gpp
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:unpack (unpack-liblouis) @ liblouis-java ---
[INFO] Configured Artifact: org.liblouis:louis:amd64-Linux-gpp-shared:?:nar
[INFO] Unpacking /home/eglic/src/pipeline/.maven-workspace/org/liblouis/louis/3.0.0-SNAPSHOT/louis-3.0.0-SNAPSHOT-amd64-Linux-gpp-shared.nar to /home/eglic/src/pipeline/libs/liblouis-java/target/dependency with includes "" and excludes ""
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ liblouis-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 9 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @ liblouis-java ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 7 source files to /home/eglic/src/pipeline/libs/liblouis-java/target/test-classes
[INFO] /home/eglic/src/pipeline/libs/liblouis-java/src/test/java/org/liblouis/Helper.java: /home/eglic/src/pipeline/libs/liblouis-java/src/test/java/org/liblouis/Helper.java uses unchecked or unsafe operations.
[INFO] /home/eglic/src/pipeline/libs/liblouis-java/src/test/java/org/liblouis/Helper.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.20:test (default-test) @ liblouis-java ---
[WARNING] The parameter forkMode is deprecated since version 2.14. Use forkCount and reuseForks instead.
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.liblouis.TableResolverTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 s - in org.liblouis.TableResolverTest
[INFO] Running org.liblouis.TranslatorTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.114 s - in org.liblouis.TranslatorTest
[INFO] Running org.liblouis.LoggerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 s - in org.liblouis.LoggerTest
[INFO] Running org.liblouis.VersionTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.092 s - in org.liblouis.VersionTest
[INFO] Running org.liblouis.FindTranslatorTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.107 s - in org.liblouis.FindTranslatorTest
[INFO] Running org.liblouis.ThreadsTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.286 s - in org.liblouis.ThreadsTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.739 s
[INFO] Finished at: 2017-04-21T16:12:58+02:00
[INFO] Final Memory: 25M/306M
[INFO] ------------------------------------------------------------------------
eglic@wharton:~/src/pipeline$

egli commented 7 years ago

So we know that the problem only shows up on OSX so far

bertfrees commented 2 years ago

This problem went away for a while but reappeared when I updated to Liblouis 3.21.0 (https://github.com/liblouis/liblouis-java/pull/24).

liblouis / liblouis-java

JVM SIGBUS crash with nested includes #10

Options

Analysis

liblouis is corrupting memory

Java bindings have a problem

Other problem