search

liblouis / liblouisutdml

An open-source library providing complete braille transcription services for xml, html and text documents
http://liblouis.io
GNU General Public License v3.0
24 stars 16 forks source link

Add the mathml tests from MathCAT #98

Closed egli closed 1 year ago

egli commented 1 year ago

Replaces #87

egli commented 1 year ago

But why do the tests fail?

Two of the new MathCAT tests trigger address sanitizer

ERROR: nemeth_MathCAT/identity_matrix_126
=========================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
=================================================================
==33789==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffd685bf18c at pc 0x7f29335bc9db bp 0x7ffd685bf100 sp 0x7ffd685bf0f0
READ of size 4 at 0x7ffd685bf18c thread T0
    #0 0x7f29335bc9da in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261
    #1 0x7f29335c3dba in styleBody /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3176
    #2 0x7f29335cb666 in end_style /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3933
    #3 0x7f293359cb65 in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:111
    #4 0x7f293359caaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #5 0x7f293359caaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #6 0x7f293359bd19 in transcribe_document /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_document.c:88
    #7 0x7f2933582011 in processXmlDocument /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:180
    #8 0x7f2933582796 in lbu_translateFile /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:283
    #9 0x561ddfcc7bb5 in main /home/runner/work/liblouisutdml/liblouisutdml/tools/file2brl.c:351
    #10 0x7f293313ed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #11 0x7f293313ee3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #12 0x561ddfcc6644 in _start (/home/runner/work/liblouisutdml/liblouisutdml/tools/.libs/file2brl+0x3644)

Address 0x7ffd685bf18c is located in stack of thread T0 at offset 44 in frame
    #0 0x7f29335bc58f in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2202

  This frame has 2 object(s):
    [48, 448) 'colSize' (line 2212) <== Memory access at offset 44 underflows this variable
    [512, 1312) 'rowBuf' (line 2213)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261 in doAlignColumns
Shadow bytes around the buggy address:
  0x10002d0afde0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afdf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe20: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
=>0x10002d0afe30: f1[f1]00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe60: 00 00 00 00 f2 f2 f2 f2 f2 f2 f2 f2 00 00 00 00
  0x10002d0afe70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002d0afe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==33789==ABORTING
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/identity_matrix_126.test (exit status: 99)
ERROR: nemeth_MathCAT/table_entry_after_sup_79_c_4
==================================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
=================================================================
==39159==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffddfb541cc at pc 0x7f4acfc6f9db bp 0x7ffddfb54140 sp 0x7ffddfb54130
READ of size 4 at 0x7ffddfb541cc thread T0
    #0 0x7f4acfc6f9da in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261
    #1 0x7f4acfc76dba in styleBody /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3176
    #2 0x7f4acfc7e666 in end_style /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3933
    #3 0x7f4acfc4fb65 in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:111
    #4 0x7f4acfc4faaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #5 0x7f4acfc4faaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #6 0x7f4acfc4ed19 in transcribe_document /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_document.c:88
    #7 0x7f4acfc35011 in processXmlDocument /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:180
    #8 0x7f4acfc35796 in lbu_translateFile /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:283
    #9 0x563259474bb5 in main /home/runner/work/liblouisutdml/liblouisutdml/tools/file2brl.c:351
    #10 0x7f4acf7f1d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #11 0x7f4acf7f1e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #12 0x563259473644 in _start (/home/runner/work/liblouisutdml/liblouisutdml/tools/.libs/file2brl+0x3644)

Address 0x7ffddfb541cc is located in stack of thread T0 at offset 44 in frame
    #0 0x7f4acfc6f58f in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2202

  This frame has 2 object(s):
    [48, 448) 'colSize' (line 2212) <== Memory access at offset 44 underflows this variable
    [512, 1312) 'rowBuf' (line 2213)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261 in doAlignColumns
Shadow bytes around the buggy address:
  0x10003bf627e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf627f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10003bf62830: 00 00 00 00 f1 f1 f1 f1 f1[f1]00 00 00 00 00 00
  0x10003bf62840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62860: 00 00 00 00 00 00 00 00 00 00 00 00 f2 f2 f2 f2
  0x10003bf62870: f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 00 00 00
  0x10003bf62880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==39159==ABORTING
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/table_entry_after_sup_79_c_4.test (exit status: 99)
hammera commented 1 year ago

@egli, what the next step with #96 related my feature branch? I see the nemeth_MathCAT merge related change the master branch. I ask this, because the wrote PR I doed two last larger commits with nemeth_MathCAT related tests integration with perhaps already unneed (this is the very last two commit with #96 pr, the my nemeth_mathml_improvements feature branch). Safe if I doing a force-push operation the feature branch to we syncing two branch contents?

After syncing, perhaps need looking my feature branch the tests section, because tests statistics is different with @norbertmarkus new nemeth tables. My feature branch following the test statistics when integrated nemeth_MathCAT tests (the last commit I disabled two address sanitizer triggering nemeth_MathCAT tests, with produce error the sanitizer):

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 386
# PASS:  209
# SKIP:  0
# XFAIL: 177
# FAIL:  0
# XPASS: 0
# ERROR: 0

In Liblouis UTDML master branch I see folluwing test statistics with original nemeth table:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  155
# SKIP:  0
# XFAIL: 233
# FAIL:  0
# XPASS: 0
# ERROR: 0

Attila