liblouis / liblouisutdml

An open-source library providing complete braille transcription services for xml, html and text documents

http://liblouis.io

GNU General Public License v3.0

24 stars 16 forks source link

Update to the files dealing with MathMl and displaying math in Nemeth Braille. #96

Closed hammera closed 1 year ago

hammera commented 1 year ago

Hi Boys,

My colleague, @norbertmarkus doed Update to the files dealing with MathMl and displaying math in Nemeth Braille. Norbert worked this change since 2023 february. As it is usual with mathematics related for Liblouis UTDML and Liblouis, I only technical contributor. If you have any questions with a change related, I simple committing required cosmetic changes. Bert, I will final remove the „added in 2023”, „modified in 2023” style lines, because Norbert not working yet the Github version control system, Norbert temporarily used as a queue leader until a change is made permanently. So, this comments final will be I remove before the final merge and approve happens.

I quote @norbertmarkus summary what changes are happened, and what change affects what file. Excuse me the long list: This is planned to be the first round of improvements aimed at making liblouisutdml's Nemeth math output capabilities more up-to-date. Thus, a number of known issues remain unresolved at this stage.

A list of files and their noteworthy modifications follows.

nemeth.sem - Text items appearing in Nemeth math blocks are now preceded and followed by a space and the text itself receives a so-called single-word switch indicator (dots 6-3) as a prefix according to an April 2018 revision of BANA's Guidance for Transcription.

nemeth_edit.ctb - This file (editTable) currently remains unchanged. It supports Nemeth math contexts to appear within EBAE literary notation (i.e. en-us-g1.ctb or en-us-g2.ctb) which is now deprecated. I recommend using nemeth_edit_with_switch_indicators.ctb as an editTable with en-us-g1.ctb or en-us-g2.ctb as mathtextTable.

nemeth_edit_with_switch_indicators.ctb - This editTable introduces switch indicators (opening Nemeth Code indicator: dots 456-146-0, and Nemeth Code terminator: dots 0-456-156) to mark the beginning and end of any Nemeth math blocks within the literary text. This table also takes care about punctuation and other symbols directly following or preceding the Nemeth blocks. This file as editTable is best used with en-us-g1.ctb or en-us-g2.ctb as mathtextTable. Another file, nemeth_edit_UEB_with_Nemeth.ctb is under preparation to provide full support for Nemeth blocks to appear within UEB texts according to an April 2018 revision of BANA's Guidance to Transcription.

The updated regulation allows for the "beginning Nemeth Code indicator" and the "Nemeth Code terminator" to appear on a line separate from the Nemeth block they surround.
A left parenthesis directly preceding and a right parenthesis directly following a Nemeth block are treated as literary symbols and are "smoothened" to the Nemeth switch indicators.
The editTable tries to guess and restore the pairs of quotes directly surrounding any Nemeth math blocks.

nemeth.ctb - The main translation table for Nemeth math content has undergone a number of changes to better comply with the Nemeth rules. The improvements are listed below:

The order of some character definitions and include statements have been changed so that the now updated Nemeth Braille dot patterns of mathematical symbols take precedence over the literary Braille patterns for the same Unicode characters. E.g. the horizontal ellipsis, and division sign are among those affected.
Now the caret symbol appears with its intended Nemeth Braille dot pattern.
The colon and semicolon as mathematical symbols appear with a punctuation prefix (dots 456-25 and dots 456-23) and a space is added after them. The space may be skipped depending on the context.
A number of rules have been disabled to give way to the now updated Nemeth Braille dot patterns of mathematical symbols in nemethdefs.cti.
The list of function names and abbreviations has been extended with items (e.g. two-letter abbreviations of function names and their combinations).
Indication of subscripts and superscripts has been improved and consolidated. E.g., numeric subscripts are now either indicated or not indicated by a subscript prefix as required. However, multilevel subscripts and superscripts are not (yet) covered as it requires keeping track of the current level.
A number sign is inserted before the first digit following a dot 5 or dots 56 (also in combinations with a minus sign) if that prefix is the first cell in the block or after a space.
Now a single digit or letter with overbar is handled correctly. Previously a two-step rule had the side-effect of deleting dot 5 from dots 5-13-156 even when this sequence appeared as part of signs of comparison (less than or equal to, etc). Now those signs of comparison, including those for sets are correctly displayed. The solution does not yet cover letters of another alphabet or type form. Such instances are displayed using the uncontracted five-step Nemeth rule.
Support for the contracted indication of a horizontal bar under a single digit or letter is added (dots 146-156 following the digit or letter) according to a 2013. BANA update to Nemeth Rule XIV.. The solution does not yet cover letters of another alphabet or typeform. Such instances are displayed using the uncontracted five-step Nemeth rule.
Now a Nemeth rule for repeating decimals is observed. When a series of digits have each a dot above in print to mark a repeating period, it is sufficient to show a single dot in Braille. The modification block, however, applies to all the digits affected.
The decimal point now appears correctly (dots 46) even when preceding or following a dot five which indicates the beginning of a modification.
A space is inserted between the Braille representation of {} for the empty set.
The space is removed from between a grouping symbol and a horizontal ellipsis in various combinations.
The space between a function name/abbreviation and a superscript belonging to it is removed. In case of a log and colog, a subscript is expected. Note that the current solution is far from perfect as processing function names and their surroundings could better be done by an algorithm.
Any series of Multiple spaces in the math blocks are reduced to a single space at that place.

@NSoiffer, if you have a little time looks the modifications? Each commit is not wery long, I used single commits (not squashed yet entire change an one commit) to easyest the review. Nemeth.ctb related diff output is only 322 lines, nemeth_edit_with_switch_indicators.ctb related patch is only 85 lines, nemeth.sem related patch is only 32 lines. How perform this changed new or modified tables with your nemeth tests?

Attila

hammera commented 1 year ago

Test system not works in Liblouis UTDML now? The java/liblouis/tables need update the nemeth.ctb, nemeth_edit.ctb, nemethdefs.cti files, and lbu_files the nemeth.sem? In tests/test-suite.log is very long, so I not paste it a comment. I see 219 total runned test, 136 test passed, 73 tests xfailed, 1 test are failed. But what? :-):-)

The committed tables is right, lou_checktable not wrote errors.

Attila

hammera commented 1 year ago

I found the failed testcase. The mathml_nemeth/msup_03.test produce now the issue. Look following log result:

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
../../lbu_files/nemeth.sem:35: Action or style or macro 'matrix' in column 1 not recognized
FAIL mathml_nemeth/msup_03.test (exit status: 1)

Liblouis UTDML not known the matrix style?

The mathml_nemeth/msup_03.test contains following testcase the input.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <mfrac>
    <msup>
      <mi>e</mi>
      <msup>
        <mi>x</mi>
        <mn>2</mn>
      </msup>
    </msup>
    <mn>2</mn>
  </mfrac>
</math>

The mathml_nemeth/msup_03.test/expected.txt contains following braille output (perhaps this is usa Braille):

?E^X^^2"/2#

@norbertmarkus, this situation what the proper Braille output this testcase xml file?

hammera commented 1 year ago

Well, perhaps I see what the problem, but I am not full sure this. My machine ran fine with make distwin command succesfull, but not following command:

docker build -f Dockerfile -t liblouisutdml:linux .

When docker after pull liblouis/liblouis image from Docker hub, trying update the package list. But: The apt package manager doesn't found Debian Jessie related packages, because Debian Jessie is already not supported. Look following command output when I do apt update in liblouis/liblouis terminal (simple ran docker run -it liblouis/liblouis, and ran apt update):

Ign http://deb.debian.org jessie InRelease
Ign http://deb.debian.org jessie-updates InRelease
Ign http://deb.debian.org jessie Release.gpg
Ign http://deb.debian.org jessie-updates Release.gpg
Ign http://deb.debian.org jessie Release
Ign http://deb.debian.org jessie-updates Release
Ign http://security.debian.org jessie/updates InRelease
Ign http://security.debian.org jessie/updates Release.gpg
Ign http://security.debian.org jessie/updates Release
Err http://security.debian.org jessie/updates/main amd64 Packages

Err http://security.debian.org jessie/updates/main amd64 Packages

Err http://security.debian.org jessie/updates/main amd64 Packages

Err http://security.debian.org jessie/updates/main amd64 Packages

Err http://security.debian.org jessie/updates/main amd64 Packages
  404  Not Found [IP: 151.101.194.132 80]
Err http://deb.debian.org jessie/main amd64 Packages
  404  Not Found
Err http://deb.debian.org jessie-updates/main amd64 Packages
  404  Not Found
W: Failed to fetch http://deb.debian.org/debian/dists/jessie/main/binary-amd64/Packages  404  Not Found

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages  404  Not Found

W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-amd64/Packages  404  Not Found [IP: 151.101.194.132 80]

E: Some index files failed to download. They have been ignored, or old ones used instead.

I have an already opened issue with Liblouis related Docker image updated, this is listed the 3.27 milestone Chris already. The problem is Debian Jessie already not supported, the actual supported stable Debian version is Debian 12 Bookworm release. Chris, how possible update inthe Docker Hub official awailable liblouis:liblouis uploaded image? Note: When I ran in Docker image terminal the lou_translate -v command, I see this utility is 3.18.0 release version. Docker Linux based image are not updated yet the Liblouis 3.26.0 version?

Attila

hammera commented 1 year ago

My machine now right builded the following command with Liblouis UTDML Linux version, based vith Debian Bookworm release:

docker build -f Dockerfile .

Of course Debian Bookworm is contains Liblouis 3.24.0 version, but 3.24.0 version the linux image better with 3.18.0 old version until the official liblouis:liblouis Docker Hub available image is not updated.

Attila

hammera commented 1 year ago

I not have another idea this day.

hammera commented 1 year ago

Okay, we are beginning going the right direction. Liblouis Utdml right builded with stable and master branch versions of Liblouis in Github. My local system when I ran ./autogen.sh;./configure;make check commands I not see error messages. In 219 tests now passed 145 tests, and have 74 xfailed tests (I mark temporary xfail with tests/math_ml/msup_03.test).

But whi failing following check?

Run sanitizer checks / Build with undefined sanitizer (pull_request) Failing after 4m

This is the workflow run link: Details

Attila

hammera commented 1 year ago

Ok, possible founded the problem. Previous we updated Liblouis version with .github/workflow/main.yml to 3.26.0 version, but forgotted update the Liblouis version with .github/workflow/sanitizer.yml file. This is perhaps produce test mismatch when ran the sanitizer check, because Norbert active uses new dot numbers with improved into nemethdefs.cti file in Liblouis 3.26.0 release. Hopefully now running correct the sanitizer checks.

Attila

hammera commented 1 year ago

Nice, final all checks are passed. Liblouis version fixing into the .github/workflow/sanitizer.yml file resolved this issue. Sorry boys the previous yesterday lot of failing checks related noise, I not known yet detailed entire the Liblouis UTDML Github workflow build infrastructure yet.

Attila

hammera commented 1 year ago

Hi Boys,

In lbu_files/nemeth.sem @norbertmarkus handled bevelled fraction lines, and lbu_files/nemeth.ctb table fixed the termination of mixed numbers. Now from 219 tests 147 tests passed, 72 tests xfailed. After we exported VERBOSE environment value with 1, and I little „hack” the run_test.sh file, I have possibility to attach Norbert a detailed test-suite.log file with contains xfailed tests diff outputs. I doed only following modification the run_test.sh script:

diff --git a/tests/run_test.sh b/tests/run_test.sh
index 1d4d8a2..eb112be 100755
--- a/tests/run_test.sh
+++ b/tests/run_test.sh
@@ -74,7 +74,7 @@ run_test () {
     if [ $? -ne 0 ]; then
    cleanup_and_exit 99 "Invocation of file2brl failed with error code $?"
     else
-   diff -q $expected $tmp_dir/output.txt >/dev/null
+   diff -q -u $expected $tmp_dir/output.txt >/dev/null
    if [ $? -ne 0 ]; then
        if [ -n "$VERBOSE" ]; then
        cat $README

I don't understand a thing, look liblouis.ini with the tests/math_ml directory:

cellsPerLine 80
linesPerPage 25
interpoint no
lineEnd \n
pageEnd \f
fileEnd ^z
printPages no
braillePages no
numberBraillePages no
paragraphs yes
beginningPageNumber 1
printPageNumberAt top
braillePageNumberAt bottom
printPageNumbersInContents yes
braillePageNumbersInContents yes
hyphenate no
outputEncoding UTF8
inputTextEncoding UTF8
backFormat plain
backLineLength 70
formatFor textDevice
lineFill '
xmlheader "<?xml version='1.0' encoding='UTF8' standalone='yes'?>"
internetAccess no
newEntries no
semanticFiles ../../lbu_files/nemeth.sem
literarytextTable unicode.dis,../../lbu_files/nemeth.ctb,braille-patterns.cti
mathtextTable     unicode.dis,../../lbu_files/nemeth.ctb,braille-patterns.cti
mathexprTable     unicode.dis,../../lbu_files/nemeth.ctb,braille-patterns.cti
editTable en-us-brf.dis,braille-patterns.cti,../../lbu_files/nemeth_edit.ctb

As you see, lot of place defined unicode.dis file with display table. But expected.txt why contains this situation with nemeth tests the usa braille output, and not unicode Braille output? Look an example verbose xfail nemeth test:

XFAIL: mathml_nemeth/msup_03
============================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
Testing MathML to Nemeth braille.
Diff: 
--- /usr/src/liblouisutdml/valami/liblouisutdml/tests/mathml_nemeth/msup_03.test/expected.txt   2023-06-19 15:53:58.897695164 +0200
+++ /tmp/foosya7db/output.txt   2023-06-21 12:39:05.378766488 +0200
@@ -1 +1 @@
-?E^X^^2"/2#
+?E^X^2"/2#
XFAIL mathml_nemeth/msup_03.test (exit status: 1)

You see, the braille output is usa Braille, not unicode Braille.

Attila

hammera commented 1 year ago

Hi Boys,

Norbert fixed yesterday evening two rules with affecting two tests with the original nemeth.ctb is xfail prewious. Lbu_files/nemeth.ctb related @norbertmarkus doed following change: make the prefix of numeric subscripts remain absent when preceded by a letter with prime(s).

Because the new rule change results two test passed again with previous xfailed the original nemeth table, I remove this tests from the xfail section.

Now the test statistics show following results:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 219
# PASS:  149
# SKIP:  0
# XFAIL: 70
# FAIL:  0
# ERROR: 0

Thanks the patience with lot of commits related. :-):-)

Attila

hammera commented 1 year ago

Sorry, previous activated the close button. I want previous press simple comment button, so reopened the pull request.

Attila

hammera commented 1 year ago

Chris, with .github/workflows/sanitizer.yml file I need update the version number to resolve online pull requests related checks. With styles.cfg related: Because the matrix style are not defined with tests/mathml_nemeth/styles.cfg file, Liblouis UTDML drops error messages this style elements related the test-suite.log file. Matrix style are defined with lbu_files/preferences.cfg file factory level, but the test system I think perhaps not use by default this config file. I choosed this faster method, not want opening a new pull request a very small, only two lines style definition related. :-):-) So, this is a cosmetic change only.

Attila

egli commented 1 year ago

Hi @hammera if you drop the changes to the Dockerfile then I think this PR can go in. Thanks

hammera commented 1 year ago

Hi Chris,

Norbert sent me an e-mail now, he working now the tests/mathml_nemeth/msubsup_06.test related changes. @norbertmarkus, estimated How much time you need, before we begin doing the final review and cleaning tasks beginning? What changes are would like you resolving (with still remaining) in Nemeth table before final review?

hammera commented 1 year ago

The „added in 2023” style changes I will be remove of course when doing the final cleaning. When @norbertmarkus developing nemeth table changes and updates, commented out some rules (with you see and listed now). I will consulting he this rules related. You wrote: „Thanks for your work!” I not doed large work, simple commit @norbertmarkus new changes my forked Liblouis UTDML repository and ran always make check command before push operation. We always consulting and I sending back Norbert the actual test-suite.log results. So, the larger work are Norbert doed, in hungary he knowns detailed with Nemeth braille rules.

Attila

NorbertMarkus commented 1 year ago

Hi Christian, Sorry for not being very talkitive nowadays... I cannot tell it now exactly but I still have at least a week of work ahead. I am in the process of going through the math related tests with xfail status and trying to find solutions as general as possible. While doing this, every day I discover some new depths and learn something new about liblouis. It is hard to estimate because some issues only take ten minutes to resolve, while others require a whole day. Now I skipped a number of tests, most of them I just put aside for a later stage. For instance those containing subscripts or superscripts of second, third, etc. order, nested radicals, complex and hypercomplex fractions and multilevel superposition. I already have an idea how to tackle multilevel subscripts and superscripts. Once I can dedicate a day or two to it and the idea works out, over a dozen more tests are hoped to pass. Nesting radicals may only be just a bit more difficult to tackle, but the complex and hypercomplex fractions and multiple "directly over or directly under" constructions nested seem to be tougher. The real trouble is being caused I think by MathML 3 syntax such as that of mmultiscripts, mstack, etc. These partially depart from the traditional parent-children treebranch logic, therefore it does not appear feasible to handle them with the current semantic actions and and within the capabilities of translation tables. Thus I'll need a little more time. Thanks for your patience.

egli commented 1 year ago

Thus I'll need a little more time.

There is no rush. Take the time you need

hammera commented 1 year ago

Chris, because .github/workflows/sanitizer.yml related change are you merged, this reviewed part are partially resolved. Remaining only the tests/mathml_nemeth/styles.cfg related little change. This change is need to prevent Liblouis UTDML not drop the test-suite.log file with error messages the undefined matrix style related if exported the VERBOSE environment value with 1. This style defined with lbu_files/preferences.cfg file, but this file not included with tests/mathml_nemeth/styles.cfg file. So I choosed the simplest method, copyed from preferences.cfg the matrix style definition. If you have possibility, please resolve this discussion part, because this thread part I not see a resolve conversation button.

hammera commented 1 year ago

@egli, I have got a short question: Default, when anybody running make check with liblouis utdml command the make check command, test-suite.log doesn't contains diff output with failed, xfail and expected output. If VERBOSE environment value not defined perhaps 1 value, following example style test-suite.log are generated:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 219
# PASS:  156
# SKIP:  0
# XFAIL: 63
# FAIL:  0
# XPASS: 0
# ERROR: 0

XFAIL: testTranslateString
==========================

Can't open infput file '(null)'
XFAIL testTranslateString (exit status: 1)

XFAIL: firstLineIndent
======================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
XFAIL firstLineIndent.test (exit status: 1)

XFAIL: mathml_nemeth/general_002
================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
XFAIL mathml_nemeth/general_002.test (exit status: 1)

So, missing this situation what the actual output the tests, and what the expected output. If I exporting VERBOSE environment variable with for example 1, this quoted test part looks following:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 219
# PASS:  156
# SKIP:  0
# XFAIL: 63
# FAIL:  0
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

XFAIL: testTranslateString
==========================

Can't open infput file '(null)'
XFAIL testTranslateString (exit status: 1)

XFAIL: firstLineIndent
======================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
Testing negative firstLineIndent
Diff: 
--- /usr/src/liblouisutdml/valami/liblouisutdml/tests/firstLineIndent.test/expected.txt 2022-12-12 12:31:51.724096727 +0100
+++ /tmp/foojTxoDp/output.txt   2023-06-29 13:21:57.570129083 +0200
@@ -1,2 +0,0 @@
-foobar
-foobar
XFAIL firstLineIndent.test (exit status: 1)

XFAIL: mathml_nemeth/general_002
================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
Testing MathML to Nemeth braille.
Diff: 
--- /usr/src/liblouisutdml/valami/liblouisutdml/tests/mathml_nemeth/general_002.test/expected.txt   2022-12-12 12:31:51.724096727 +0100
+++ /tmp/fooLEXy5I/output.txt   2023-06-29 13:21:57.658129362 +0200
@@ -1 +1 @@
-(N%K)
+(?N/K#)
XFAIL mathml_nemeth/general_002.test (exit status: 1)

So, you see, if VERBOSE variable is 1, every information have right the test-suite.log file, similar with in Liblouis yaml tests result if have any failure.

When norbert and me working the math table related changes, we always use verbose mode to see a test have a real failure, or a wrong filled input.xml file, with happened factory level few directoryes. So, very help the test diagnosing related the verbose mode to detect real problems why dropping for example a nemeth test with failure.

Because this possible change unrelated with mathml modification, I open a separate PR this change related to test-suite.log file always contains failures and diff outputs? If have better way to resolve this issue, please suggest an another method.

I suggesting a very little code change with tests/run_test.sh file with resolves this issue and affects diff output presentation:

diff --git a/tests/run_test.sh b/tests/run_test.sh
index 1d4d8a2..4b3e874 100755
--- a/tests/run_test.sh
+++ b/tests/run_test.sh
@@ -18,6 +18,8 @@
 # License along with this program; see the file COPYING.  If not, write
 # to the Free Software Foundation, 51 Franklin Street, Fifth Floor,
 # Boston, MA 02110-1301, USA.
+# To presents test-suite.log the failure diff output, always exporting the VERBOSE variable with 1
+export VERBOSE=1

 # Create a temporary directory $tmp_dir in $TMPDIR (default /tmp).
 : ${TMPDIR=/tmp}
@@ -74,7 +76,7 @@ run_test () {
     if [ $? -ne 0 ]; then
    cleanup_and_exit 99 "Invocation of file2brl failed with error code $?"
     else
-   diff -q $expected $tmp_dir/output.txt >/dev/null
+   diff -q -u $expected $tmp_dir/output.txt >/dev/null
    if [ $? -ne 0 ]; then
        if [ -n "$VERBOSE" ]; then
        cat $README

If this code modification okay for you, I drop a PR this code change related.

egli commented 1 year ago

Hi Attila

I suggesting a very little code change with tests/run_test.sh file with resolves this issue and affects diff output presentation:

Can't you just do

$ VERBOSE=1 make check

That seems to produce exactly the output you want, without hard coding anything

hammera commented 1 year ago

Chris, VERBOSE=1 make check works perfect, you are right. What do you think? Need diff output related with following code change with tests/run_test.sh file? If yes, I open a little PR this change related:

diff --git a/tests/run_test.sh b/tests/run_test.sh
index 1d4d8a2..eb112be 100755
--- a/tests/run_test.sh
+++ b/tests/run_test.sh
@@ -74,7 +74,7 @@ run_test () {
     if [ $? -ne 0 ]; then
    cleanup_and_exit 99 "Invocation of file2brl failed with error code $?"
     else
-   diff -q $expected $tmp_dir/output.txt >/dev/null
+   diff -q -u $expected $tmp_dir/output.txt >/dev/null
    if [ $? -ne 0 ]; then
        if [ -n "$VERBOSE" ]; then
        cat $README

So, original code use only -q option, and i added after -q option the -u option (unified diff option with default 3 line output mode). I better like unified diff mode, but this is absolute subjective, perhaps other developers not like this mode the default tests/run_test.sh diff -q used method. :-):-)

Attila

egli commented 1 year ago

I looked at it some more. If I understand correctly your change should not change anything, as the output of the diff command you are changing goes to /def/null anyway. The script only checks if the return value of the script is other than 0. Then (and if VERBOSE=1) it does a diff -u $expected $actual.

So I don't think any change is needed in the run_test.sh script.

hammera commented 1 year ago

My god, you are right. :-):-) The first diff happens with dev/null. :-):-) Already will be have weekend, sorry.

The 77. line contains following:

    diff -q $expected $tmp_dir/output.txt >/dev/null

After the script examines verbose variable, and if this is for example 1, happens cat $README command, echo "Diff: " >&2 command, and diff -u $expected $tmp_dir/output.txt command. So the test-suite.log are landed the 3 way mode diff result if I understands right the script.

hammera commented 1 year ago

An e-mail notification came, and since the screen reader was temporarily out of focus mode, I accidentally activated the close button again. Sorry the unwanted close action, I reopened the PR. Yes, really run_test.sh not need doing any change.

hammera commented 1 year ago

Bert, what do you think? I trying to merge based Neil patches from #87 the new tests to this branch, and handle the two PR with one thread? Or you want first doing the #87 related merge first? I have commit access only the github.com/hammera/liblouisutdml repo. @NorbertMarkus need replaced input.xml and expected.txt files to old time xfailed tests passing again right, so we need doing careful work to this tests not will be again xfailing. An another need resolve task with existing expected.txt files need converting unicode Braille output from usa Braille, the original tests uses usa Braille but if I remember right Neil new tests based with unicode Braille. To ensure this I will looking #87 related commits diff output. Have a safe way this converting tasks related to we not do regressions the existing expected.txt tests results? Original time the test creators how created this expected.txt files into the tests/mathml_nemeth directory, generated this outputs with file2brl command? If yes, what the proper command line sintax? Perhaps I have possibility to writing a glob.glob based Python3 processing scrypt with doing the automatic conversion from usa braille to unicode braille the expected.txt files related if known the proper file2brl configuration parameters. But you possible known better way the conversion related.

Attila

hammera commented 1 year ago

Ok, looks what the actual state now with tests related:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 219
# PASS:  164
# SKIP:  0
# XFAIL: 55
# FAIL:  0
# XPASS: 0
# ERROR: 0

I tryed integrating Neil nemeth MathCat based tests from #87. I added required nemeth_MathCAT subdirectory the Makefile.am file, andd added all new tests with suite tests. The interesting thing is into nemeth_MathCAT directory not generating Makefile and Makefile.in files when I running ./autogen.sh and ./configure commands What can I forgotted? :-):-)

Bert or Chris, please look the main tests/Makefile.am modification diff output, next comment I will shows what contains into tests/nemeth_MathCAT directory Makefile.am file, similar with another tests. I am absolute sure missing a trivial step with I forgotted. :-):-) With each nemeth_MathCAT subdirectory awailable tests directory need autogenerating I think similar Makefile.am file, this is I easy will be doing a Python3 glob scrypt:

EXTRA_DIST =        \
    expected.txt    \
    input.xml

So, the large tests/Makefile.am modification with I tryed first step:

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 5e75805..746a75f 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -8,6 +8,7 @@ SUBDIRS =                   \
    mathml_nemeth               \
    minSyllableWidth.test           \
    mixed_content.test          \
+   nemeth_MathCAT          \
    orphanControl_01.test           \
    orphanControl_02.test           \
    page_numbering.test         \
@@ -135,6 +136,176 @@ dist_suite_TESTS =                \
    mathml_nemeth/munderover_03.test    \
    minSyllableWidth.test           \
    mixed_content.test          \
+   nemeth_MathCAT/above_and_below_88_2.test            \
+   nemeth_MathCAT/arrow_96_1.test          \
+   nemeth_MathCAT/arrow_96_4.test          \
+   nemeth_MathCAT/as_multiscript_nested_sub_sup_74_c_5.test            \
+   nemeth_MathCAT/bar_97_b_1.test          \
+   nemeth_MathCAT/bar_above_and_below_88_1.test            \
+   nemeth_MathCAT/baseline_80_a_1.test         \
+   nemeth_MathCAT/beveled_frac_62_b_1.test         \
+   nemeth_MathCAT/binomial_90_1.test           \
+   nemeth_MathCAT/boldface_32_b_3.test         \
+   nemeth_MathCAT/boldface_32_b_6.test         \
+   nemeth_MathCAT/cap_22_a_1.test          \
+   nemeth_MathCAT/cap_roman_numeral_18_a_3.test            \
+   nemeth_MathCAT/carrot_98_1.test         \
+   nemeth_MathCAT/comma_78_2_invisible.test            \
+   nemeth_MathCAT/comma_78_2.test          \
+   nemeth_MathCAT/comma_78_3.test          \
+   nemeth_MathCAT/comma_78_6.test          \
+   nemeth_MathCAT/comma_ellipsis_in_sub_79_b_5.test            \
+   nemeth_MathCAT/comma_in_number_in_sup_79_b_3.test           \
+   nemeth_MathCAT/comma_in_sup_79_b_4.test         \
+   nemeth_MathCAT/comma_number_77_4_20.test            \
+   nemeth_MathCAT/comma_space_78_1.test            \
+   nemeth_MathCAT/comparison_79_g_2.test           \
+   nemeth_MathCAT/complex_frac_66_1.test           \
+   nemeth_MathCAT/dash_42_4.test           \
+   nemeth_MathCAT/dash_42_6.test           \
+   nemeth_MathCAT/dots_99_a_1.test         \
+   nemeth_MathCAT/dots_99_a_2.test         \
+   nemeth_MathCAT/dots_99_a_3.test         \
+   nemeth_MathCAT/ellipsis_43_b_3.test         \
+   nemeth_MathCAT/ellipsis_level_79_f_1.test           \
+   nemeth_MathCAT/extension_field_not_ratio_151_11.test            \
+   nemeth_MathCAT/full_binomial.test           \
+   nemeth_MathCAT/function_space_119_c_3.test          \
+   nemeth_MathCAT/german_base_77_4_3.test          \
+   nemeth_MathCAT/hyper_complex_frac_68_a_1.test           \
+   nemeth_MathCAT/identity_matrix_126.test         \
+   nemeth_MathCAT/integral_77_4_26.test            \
+   nemeth_MathCAT/left_sup_75_12.test          \
+   nemeth_MathCAT/left_sup_75_1.test           \
+   nemeth_MathCAT/left_sup_75_4.test           \
+   nemeth_MathCAT/left_sup_75_7.test           \
+   nemeth_MathCAT/left_sup_75_8.test           \
+   nemeth_MathCAT/lesson_11_24_1.test          \
+   nemeth_MathCAT/letter_26_b_19.test          \
+   nemeth_MathCAT/lim_86_a_3.test          \
+   nemeth_MathCAT/list_num_ind_11_a_1.test         \
+   nemeth_MathCAT/list_num_ind_11_a_2.test         \
+   nemeth_MathCAT/list_num_ind__11_a_3.test            \
+   nemeth_MathCAT/list_num_ind_11_a_7.test         \
+   nemeth_MathCAT/list_num_ind_11_c_1.test         \
+   nemeth_MathCAT/list_num_ind_11_d_1.test         \
+   nemeth_MathCAT/log_77_4_8.test          \
+   nemeth_MathCAT/lower_roman_numeral_18_b_4.test          \
+   nemeth_MathCAT/mathml_spec_example_86_a.test            \
+   nemeth_MathCAT/mathml_spec_example_alt_char_86_a.test           \
+   nemeth_MathCAT/menclose_111_a_4.test            \
+   nemeth_MathCAT/menclose_115_1.test          \
+   nemeth_MathCAT/menclose_86_a_1.test         \
+   nemeth_MathCAT/menclose_86_a_4.test         \
+   nemeth_MathCAT/menclose_86_b_11.test            \
+   nemeth_MathCAT/menclose_86_b_1.test         \
+   nemeth_MathCAT/menclose_bar_97_b_1.test         \
+   nemeth_MathCAT/menclose_bar_97_b_3.test         \
+   nemeth_MathCAT/menclose_lesson_12_5_5_5.test            \
+   nemeth_MathCAT/menclose_primed_86_b_6.test          \
+   nemeth_MathCAT/menclose_top_bottom_88_1.test            \
+   nemeth_MathCAT/mixed_frac_63_a_1.test           \
+   nemeth_MathCAT/mixed_frac_64_2.test         \
+   nemeth_MathCAT/mmultiscripts_77_4_10.test           \
+   nemeth_MathCAT/mmultiscripts_77_4_18.test           \
+   nemeth_MathCAT/mmultiscripts_82_a_1.test            \
+   nemeth_MathCAT/mmultiscripts_82_a_2.test            \
+   nemeth_MathCAT/mmultiscripts_82_a_3.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_1.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_2.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_3.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_4.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_5.test            \
+   nemeth_MathCAT/mmultiscripts_82_b_6.test            \
+   nemeth_MathCAT/modifier_in_script_91_1.test         \
+   nemeth_MathCAT/ms_38_4_8.test           \
+   nemeth_MathCAT/ms.test          \
+   nemeth_MathCAT/msubsup_82_a_1.test          \
+   nemeth_MathCAT/msubsup_82_a_3.test          \
+   nemeth_MathCAT/multipurpose_177_2_1.test            \
+   nemeth_MathCAT/multipurpose_177_2_2.test            \
+   nemeth_MathCAT/multipurpose_177_3_1.test            \
+   nemeth_MathCAT/multipurpose_177_5_1.test            \
+   nemeth_MathCAT/multipurpose_177_7_1.test            \
+   nemeth_MathCAT/munder_lesson_12_5_5_5.test          \
+   nemeth_MathCAT/nested_root_105_2.test           \
+   nemeth_MathCAT/nested_sqrt_105_1.test           \
+   nemeth_MathCAT/nested_sqrt_105_3.test           \
+   nemeth_MathCAT/nested_sqrt_105_4.test           \
+   nemeth_MathCAT/nested_sub_sup_74_c_5.test           \
+   nemeth_MathCAT/nested_sup_74_b_1.test           \
+   nemeth_MathCAT/nested_sup_74_b_4.test           \
+   nemeth_MathCAT/nested_super_79_a_2.test         \
+   nemeth_MathCAT/nested_super_79_a_3.test         \
+   nemeth_MathCAT/nested_super_space_79_d_3.test           \
+   nemeth_MathCAT/nested_super_space_79_d_7.test           \
+   nemeth_MathCAT/nested_sup_mmultiscripts_74_b_1.test         \
+   nemeth_MathCAT/nested_sup_sup_space_79_d_9.test         \
+   nemeth_MathCAT/non_hyper_complex_frac_67_1.test         \
+   nemeth_MathCAT/non_list_10_4.test           \
+   nemeth_MathCAT/number_space_after.test          \
+   nemeth_MathCAT/number_space_before_and_after.test           \
+   nemeth_MathCAT/number_space_before.test         \
+   nemeth_MathCAT/numeric_sub_81_a_1.test          \
+   nemeth_MathCAT/order2_overbar_87_a_1.test           \
+   nemeth_MathCAT/overbar_86_a_4.test          \
+   nemeth_MathCAT/overbar_86_a_5.test          \
+   nemeth_MathCAT/overbar_86_b_10.test         \
+   nemeth_MathCAT/overbar_86_b_11.test         \
+   nemeth_MathCAT/overbar_86_b_1.test          \
+   nemeth_MathCAT/prescript_77_4_18.test           \
+   nemeth_MathCAT/prescript_77_4_6.test            \
+   nemeth_MathCAT/prescript_77_4_7.test            \
+   nemeth_MathCAT/prime_172_5.test         \
+   nemeth_MathCAT/prime_172_6.test         \
+   nemeth_MathCAT/prime_172_8.test         \
+   nemeth_MathCAT/prime_172_9.test         \
+   nemeth_MathCAT/prime_77_4_4.test            \
+   nemeth_MathCAT/prime_83_b_1.test            \
+   nemeth_MathCAT/prime_83_b_3.test            \
+   nemeth_MathCAT/prime_83_b_4.test            \
+   nemeth_MathCAT/prime_83_b_5.test            \
+   nemeth_MathCAT/prime_83_b_6.test            \
+   nemeth_MathCAT/prime_83_b_7.test            \
+   nemeth_MathCAT/prime_83_b_8.test            \
+   nemeth_MathCAT/primed_86_b_6.test           \
+   nemeth_MathCAT/prime_mathjax_83_b_2.test            \
+   nemeth_MathCAT/prime_mmultiscripts_83_b_2.test          \
+   nemeth_MathCAT/prime_mmultiscripts_83_b_4.test          \
+   nemeth_MathCAT/prime_wiris_83_b_2.test          \
+   nemeth_MathCAT/product_77_4_24.test         \
+   nemeth_MathCAT/punct_37_1_2.test            \
+   nemeth_MathCAT/punct_37_2_2.test            \
+   nemeth_MathCAT/punct_37_7_1.test            \
+   nemeth_MathCAT/punctuation_after_sup_79_b_2.test            \
+   nemeth_MathCAT/ratio_151_11.test            \
+   nemeth_MathCAT/root_104_iii_1.test          \
+   nemeth_MathCAT/root_104_iii_4.test          \
+   nemeth_MathCAT/set_vertical_bar_145_1.test          \
+   nemeth_MathCAT/shape_110_1.test         \
+   nemeth_MathCAT/simple_frac_62_a_3.test          \
+   nemeth_MathCAT/sqrt_103_a_2.test            \
+   nemeth_MathCAT/sqrt_103_a_4.test            \
+   nemeth_MathCAT/sqrt_103_b_2.test            \
+   nemeth_MathCAT/styles.cfg           \
+   nemeth_MathCAT/sub_ind_79_g_4.test          \
+   nemeth_MathCAT/sub_ind_80_b_3.test          \
+   nemeth_MathCAT/sub_ind_80_b_4.test          \
+   nemeth_MathCAT/sub_ind_mmultiscripts_80_b_3.test            \
+   nemeth_MathCAT/sub_sup_82_b_1.test          \
+   nemeth_MathCAT/sub_sup_82_b_2.test          \
+   nemeth_MathCAT/sum_77_4_23.test         \
+   nemeth_MathCAT/superscript_80_a_2.test          \
+   nemeth_MathCAT/table_entry_after_sup_79_c_4.test            \
+   nemeth_MathCAT/tensor_from_mathml_spec.test         \
+   nemeth_MathCAT/test_9_d_2.test          \
+   nemeth_MathCAT/test_9_d_3.test          \
+   nemeth_MathCAT/text_after_sup_79_c_3.test           \
+   nemeth_MathCAT/trilinear_not_ratio_151_11.test          \
+   nemeth_MathCAT/underbar_86_a_1.test         \
+   nemeth_MathCAT/vertical_bar_145_4.test          \
+   nemeth_MathCAT/whitespace_in_sup_79_e_1.test            \
+   nemeth_MathCAT/word_77_4_12.test            \
    orphanControl_01.test           \
    orphanControl_02.test           \
    page_numbering.test         \

Attila

hammera commented 1 year ago

This is the new created tests/nemeth_MathCat/Makefile.am file, similar with tests/mathml_nemeth directory awailable Makefile.am file:

SUBDIRS =           \
    above_and_below_88_2.test   \
    arrow_96_1.test \
    arrow_96_4.test \
    as_multiscript_nested_sub_sup_74_c_5.test   \
    bar_97_b_1.test \
    bar_above_and_below_88_1.test   \
    baseline_80_a_1.test    \
    beveled_frac_62_b_1.test    \
    binomial_90_1.test  \
    boldface_32_b_3.test    \
    boldface_32_b_6.test    \
    cap_22_a_1.test \
    cap_roman_numeral_18_a_3.test   \
    carrot_98_1.test    \
    comma_78_2_invisible.test   \
    comma_78_2.test \
    comma_78_3.test \
    comma_78_6.test \
    comma_ellipsis_in_sub_79_b_5.test   \
    comma_in_number_in_sup_79_b_3.test  \
    comma_in_sup_79_b_4.test    \
    comma_number_77_4_20.test   \
    comma_space_78_1.test   \
    comparison_79_g_2.test  \
    complex_frac_66_1.test  \
    dash_42_4.test  \
    dash_42_6.test  \
    dots_99_a_1.test    \
    dots_99_a_2.test    \
    dots_99_a_3.test    \
    ellipsis_43_b_3.test    \
    ellipsis_level_79_f_1.test  \
    extension_field_not_ratio_151_11.test   \
    full_binomial.test  \
    function_space_119_c_3.test \
    german_base_77_4_3.test \
    hyper_complex_frac_68_a_1.test  \
    identity_matrix_126.test    \
    integral_77_4_26.test   \
    left_sup_75_12.test \
    left_sup_75_1.test  \
    left_sup_75_4.test  \
    left_sup_75_7.test  \
    left_sup_75_8.test  \
    lesson_11_24_1.test \
    letter_26_b_19.test \
    lim_86_a_3.test \
    list_num_ind_11_a_1.test    \
    list_num_ind_11_a_2.test    \
    list_num_ind__11_a_3.test   \
    list_num_ind_11_a_7.test    \
    list_num_ind_11_c_1.test    \
    list_num_ind_11_d_1.test    \
    log_77_4_8.test \
    lower_roman_numeral_18_b_4.test \
    mathml_spec_example_86_a.test   \
    mathml_spec_example_alt_char_86_a.test  \
    menclose_111_a_4.test   \
    menclose_115_1.test \
    menclose_86_a_1.test    \
    menclose_86_a_4.test    \
    menclose_86_b_11.test   \
    menclose_86_b_1.test    \
    menclose_bar_97_b_1.test    \
    menclose_bar_97_b_3.test    \
    menclose_lesson_12_5_5_5.test   \
    menclose_primed_86_b_6.test \
    menclose_top_bottom_88_1.test   \
    mixed_frac_63_a_1.test  \
    mixed_frac_64_2.test    \
    mmultiscripts_77_4_10.test  \
    mmultiscripts_77_4_18.test  \
    mmultiscripts_82_a_1.test   \
    mmultiscripts_82_a_2.test   \
    mmultiscripts_82_a_3.test   \
    mmultiscripts_82_b_1.test   \
    mmultiscripts_82_b_2.test   \
    mmultiscripts_82_b_3.test   \
    mmultiscripts_82_b_4.test   \
    mmultiscripts_82_b_5.test   \
    mmultiscripts_82_b_6.test   \
    modifier_in_script_91_1.test    \
    ms_38_4_8.test  \
    ms.test \
    msubsup_82_a_1.test \
    msubsup_82_a_3.test \
    multipurpose_177_2_1.test   \
    multipurpose_177_2_2.test   \
    multipurpose_177_3_1.test   \
    multipurpose_177_5_1.test   \
    multipurpose_177_7_1.test   \
    munder_lesson_12_5_5_5.test \
    nested_root_105_2.test  \
    nested_sqrt_105_1.test  \
    nested_sqrt_105_3.test  \
    nested_sqrt_105_4.test  \
    nested_sub_sup_74_c_5.test  \
    nested_sup_74_b_1.test  \
    nested_sup_74_b_4.test  \
    nested_super_79_a_2.test    \
    nested_super_79_a_3.test    \
    nested_super_space_79_d_3.test  \
    nested_super_space_79_d_7.test  \
    nested_sup_mmultiscripts_74_b_1.test    \
    nested_sup_sup_space_79_d_9.test    \
    non_hyper_complex_frac_67_1.test    \
    non_list_10_4.test  \
    number_space_after.test \
    number_space_before_and_after.test  \
    number_space_before.test    \
    numeric_sub_81_a_1.test \
    order2_overbar_87_a_1.test  \
    overbar_86_a_4.test \
    overbar_86_a_5.test \
    overbar_86_b_10.test    \
    overbar_86_b_11.test    \
    overbar_86_b_1.test \
    prescript_77_4_18.test  \
    prescript_77_4_6.test   \
    prescript_77_4_7.test   \
    prime_172_5.test    \
    prime_172_6.test    \
    prime_172_8.test    \
    prime_172_9.test    \
    prime_77_4_4.test   \
    prime_83_b_1.test   \
    prime_83_b_3.test   \
    prime_83_b_4.test   \
    prime_83_b_5.test   \
    prime_83_b_6.test   \
    prime_83_b_7.test   \
    prime_83_b_8.test   \
    primed_86_b_6.test  \
    prime_mathjax_83_b_2.test   \
    prime_mmultiscripts_83_b_2.test \
    prime_mmultiscripts_83_b_4.test \
    prime_wiris_83_b_2.test \
    product_77_4_24.test    \
    punct_37_1_2.test   \
    punct_37_2_2.test   \
    punct_37_7_1.test   \
    punctuation_after_sup_79_b_2.test   \
    ratio_151_11.test   \
    root_104_iii_1.test \
    root_104_iii_4.test \
    set_vertical_bar_145_1.test \
    shape_110_1.test    \
    simple_frac_62_a_3.test \
    sqrt_103_a_2.test   \
    sqrt_103_a_4.test   \
    sqrt_103_b_2.test   \
    sub_ind_79_g_4.test \
    sub_ind_80_b_3.test \
    sub_ind_80_b_4.test \
    sub_ind_mmultiscripts_80_b_3.test   \
    sub_sup_82_b_1.test \
    sub_sup_82_b_2.test \
    sum_77_4_23.test    \
    superscript_80_a_2.test \
    table_entry_after_sup_79_c_4.test   \
    tensor_from_mathml_spec.test    \
    test_9_d_2.test \
    test_9_d_3.test \
    text_after_sup_79_c_3.test  \
    trilinear_not_ratio_151_11.test \
    underbar_86_a_1.test    \
    vertical_bar_145_4.test \
    whitespace_in_sup_79_e_1.test   \
    word_77_4_12.test

EXTRA_DIST =            \
    README          \
    liblouisutdml.ini   \
    styles.cfg

Attila

hammera commented 1 year ago

Now yet not have all tests directory a generated Makefile.am file with contains README, input.xml and expected.txt files, but this is possible doing automatically I thing. Now this is the erroe message when make check command trying execute nemeth_MathCAT tests:

Making check in nemeth_MathCAT
make[1]: Entering directory '/usr/src/liblouisutdml/valami/liblouisutdml/tests/nemeth_MathCAT'
make[1]: *** No rule to make target 'check'.  Stop.
make[1]: Leaving directory '/usr/src/liblouisutdml/valami/liblouisutdml/tests/nemeth_MathCAT'
make: *** [Makefile:1750: check-recursive] Error 1

Attila

bertfrees commented 1 year ago

I trying to merge based Neil patches from https://github.com/liblouis/liblouisutdml/pull/87 the new tests to this branch, and handle the two PR with one thread? Or you want first doing the https://github.com/liblouis/liblouisutdml/pull/87 related merge first?

87 first.

An another need resolve task with existing expected.txt files need converting unicode Braille output from usa Braille, the original tests uses usa Braille but if I remember right Neil new tests based with unicode Braille.

As far as I know one set of tests can use ASCII braille and another can use Unicode braille. That's perfectly fine. We can think of converting all tests to one and the same character set later, but that's not needed now.

The interesting thing is into nemeth_MathCAT directory not generating Makefile and Makefile.in files when I running ./autogen.sh and ./configure commands. What can I forgotted? :-):-)

I think all the Makefiles need to be listed in configure.ac.

hammera commented 1 year ago

Million thanks Bert, I will looking this. I already autogenerated nemeth_MathCAT test directoryes the required Makefile.am files, I simple wrote a little Python3 scrypt.

When my machine running right the new nemeth_MathCAT tests, how can possible attach diff output the #87 PR related? Github what formats are presents you with attachments, not single comments? I not want paste large diff outputs with comment box.

I used this simple python3 scrypt in the tests/nemeth_MathCAT directory the test files related Makefile.am generation:

#!coding: utf-8
from glob import iglob
for i in iglob('*.test'):
    print('Creating '+i+' Makefile.am...')
    f=open(i+'/Makefile.am', 'w')
    f.write('EXTRA_DIST =\t\t\\\n')
    f.write('\texpected.txt\t\\\n')
    f.write('\tinput.xml\n')

The laziness is half health. :-):-)

Attila

hammera commented 1 year ago

Ok boys, partially done the #87 related integration. :-):-) My development Liblouis UTDML repo with contains @NorbertMarkus latest changes, I see following test results in verbose mode:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  208
# SKIP:  0
# XFAIL: 55
# FAIL:  124
# XPASS: 0
# ERROR: 1

The previous statistics shows from 219 tests passed 164 tests, and xfailed I thing 55 test. When we begun the work, passed only 136 tests, this is I think more better value now. :-):-)

Following test are drop error:

ERROR: nemeth_MathCAT/test_9_d_2
================================

warning: lou_getDataPath is deprecated.
styles.cfg:1: word 'xxstyle' in column 1 not recognized
styles.cfg:2: word 'firstLineIndent' in column 1 not recognized
warning: lou_getDataPath is deprecated.
styles.cfg:1: word 'xxstyle' in column 1 not recognized
styles.cfg:2: word 'firstLineIndent' in column 1 not recognized
/tmp/fooixExLj/file2brl.temp:4: 
parser 
error : 
AttValue: " or ' expected

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooixExLj/file2brl.temp:4: 
parser 
error : 
attributes construct error

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooixExLj/file2brl.temp:4: 
parser 
error : 
Couldn't find end of Start Tag mi line 4

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooixExLj/file2brl.temp:5: 
parser 
error : 
Opening and ending tag mismatch: math line 0 and mi

</mi>

     ^

/tmp/fooixExLj/file2brl.temp:6: 
parser 
error : 
Extra content at the end of the document

<mn>4</mn>

^

Document could not be processed; may be malformed or contain illegal characters
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/test_9_d_2.test (exit status: 99)

I replaced with xxstyle with style the tests/nemeth_MathCAT/styles.cfg with style word, but perhaps this XML file wrong. What the next step to we finalize this integration? Anybody have free capacity to look the failing MathCAT tests with are real failures, or need follow up the expected.txt files with rule changes, similar with oldest math_nemeth tests expected.txt and input.xml files? An another option to we now drop the 124 failing tests with the Xfail section, commit and push the new integrated MathCAT tests, and go on the next step. Unfortunatelly I not using the Nemeth and MathCAT braille. :-(:-(

Attila

hammera commented 1 year ago

Hi Boys,

I am interesting and looks what the situation if we integrating nemeth_MathCAT tests without @NorbertMarkus changes. Look the following statistics outputs. Original Liblouis UTDML test results, before @norbertmarkus begun working rule changes:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 219
# PASS:  137
# SKIP:  0
# XFAIL: 82
# FAIL:  0
# XPASS: 0
# ERROR: 0

Original Liblouis UTDML tests, after nemeth_MathCAT integration (without @norbertmarkus doed changes):

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  155
# SKIP:  0
# XFAIL: 82
# FAIL:  150
# XPASS: 0
# ERROR: 1

And the last statistics, when the branch contains both @NorbertMarkus doed changes, and new integrated nemeth_MathCAT tests, we beginning going the right direction:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  208
# SKIP:  0
# XFAIL: 55
# FAIL:  124
# XPASS: 0
# ERROR: 1

The integration related diff patch is very large (aproximatelly 4864 lines).

hammera commented 1 year ago

So, the integration related patch is very large, because the patch creating lot of small new test files. If anybody to suggest a Github compatible attachment format, I will attaching the patch with this pull request or the #87 related pull request. Of course to make check command running right in Github online pull request checks, temporary need drop the now failed tests the xfail section (the 150 failed test or the 124 failed test).

Attila

hammera commented 1 year ago

Boys, before I not forgot: In Liblouis UTDML java/testdata/liblouis/tables directory need update important table files (chardefs.cti, nemeth.ctb, nemethdefs.cti, etc), or this is not important? Nemethdefs.cti file changed into the Liblouis 3.26 release, but I am not sure need update the testdata directory the required table files.

Thanks,

Attila

hammera commented 1 year ago

Boys, I have got a question before Norbert and me going forward the next step: @NorbertMarkus fixed the error the nemeth_MathCAT/test_9_d_2.test/input.xml file, and doed following change the lbu_files/nemeth.ctb file. Look following shorter diff output, this change results pass this test too:

diff --git a/lbu_files/nemeth.ctb b/lbu_files/nemeth.ctb
index 9d3b41e..dcad8c6 100644
--- a/lbu_files/nemeth.ctb
+++ b/lbu_files/nemeth.ctb
@@ -77,6 +77,8 @@ math \x00a8 16-16 DIAERESIS (double dots) in math expressions
 math \x2322 1246-1 FROWN (CONCAVE UPWARD ARC)
 math \x2323 1246-3 SMILE (CONCAVE DOWNWARD ARC)

+math \x0023 3456a pseudo hash mark for further processing
+
 # Braille indicators
 multind 56-6 letsign capsletter
 capsletter 6
@@ -98,7 +100,8 @@ postpunc " 356
 postpunc , 6
 always \e0 5
 always , 6-a
-always # 35-2345 print number sign before number
+#always # 35-2345 print number sign before number
+always # 3456a number sign (hashmark) in print - a temporary representation for further processing
 decpoint . 46
 hyphen - 36
 postpunc . 456-256
@@ -283,6 +286,9 @@ compbrl .zip

 # multipass corrections

+#Process print number sign (hash mark)
+noback pass2 [@3456a]$d @46-3456-3456 #a digit following a print hash mark needs its own Braille number sign
+noback pass2 @3456a @46-3456 #produce a hash mark symbol when not followed by a number
 #Fix the context of the hollow dot for its uses as degrees sign or composit operator
 noback pass2 @45[@45-46-16-5]@5 * #remove the nesting superscript in cases the degrees sign happens to be in the correct syntax
 noback pass3 $d@45-46-16[@5]@6-124 @a #insert space before the F for Fahrenheit
diff --git a/tests/nemeth_MathCAT/test_9_d_2.test/input.xml b/tests/nemeth_MathCAT/test_9_d_2.test/input.xml
index ab4fc24..23c09d6 100644
--- a/tests/nemeth_MathCAT/test_9_d_2.test/input.xml
+++ b/tests/nemeth_MathCAT/test_9_d_2.test/input.xml
@@ -1,2 +1,2 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<math><mn>3</mn><mi mathvariant=\"normal\">#<!-- # --></mi><mn>4</mn></math>
+<math><mn>3</mn><mi mathvariant="normal">#</mi><mn>4</mn></math>

When I ran make check after this patch apply, the previous error showed test passed. Previous this is the error message with following test:

ERROR: nemeth_MathCAT/test_9_d_2
================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
/tmp/fooPgIP9e/file2brl.temp:4:
parser
error :
AttValue: " or ' expected

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooPgIP9e/file2brl.temp:4:
parser
error :
attributes construct error

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooPgIP9e/file2brl.temp:4:
parser
error :
Couldn't find end of Start Tag mi line 4

<mi mathvariant=\"normal\">#<!-- # -->

                ^

/tmp/fooPgIP9e/file2brl.temp:5:
parser
error :
Opening and ending tag mismatch: math line 0 and mi

</mi>

     ^

/tmp/fooPgIP9e/file2brl.temp:6:
parser
error :
Extra content at the end of the document

<mn>4</mn>

^

Document could not be processed; may be malformed or contain illegal characters
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/test_9_d_2.test (exit status: 99)

After I patched my source tree with Norbert fix with I quoted the comment box, shows following result the test-suite.log file:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  209
# SKIP:  0
# XFAIL: 55
# FAIL:  124
# XPASS: 0
# ERROR: 0

What can I do your openion? I Commit Norbert doed change with lbu_files/nemeth.ctb table related my branch, and commit an another commit the entire nemeth_MathCAT integration related large patch a second commit?

The integration related patch is very large, and when I perhaps committing the integration related patch my branch, need temporary putting the now 124 failing nemeth_MathCAT tests the tests/Makefile.am file into the XFAIL_TESTS += section to online pull request check don't throw an error message.

Please give hint, what can we do the ready integration related patch part, except the 124 now failing nemeth_MathCAT tests. If I known right Norbert now working the ueb compatible edit table extending.

Attila

hammera commented 1 year ago

nemeth_mathcat_integration.txt Boys, I attached the commit ready patch for Neil nemeth MathCAT tests related with compatible this development branch. Don't surprise the txt extension, Github accepts only this extension format, but the file is standard git diff patch. Don't surprise the patch size, the integration patch is large, but safe. Make check command ran my machine without error messages, so after the commit and push, Github online check for pull requests will be not dropping any error messages. After committing this patch, we have possibility to close the #87 related pull request, because I thing I handled everithing with Neil lists this pull request. Temporary the now fail nemeth_MathCAT tests I dropped the xfailtest+ section the proper place after mathml_nemeth xfail tests to Github online action files not drop error messages after push.

Bert, if you would like to I attach the patch with #87 related pull request, I need add more xfail tests, because @NorbertMarkus doed changes resolves 209 tests, and have only 124 xfail tests. The original Liblouis tables have more failed nemeth_MathCAT tests, so I choosed now this attachment way. The patch not will be doing any regression, so after review, I welcome commit and push this change my development branch.

@NorbertMarkus working now with edit tables related works.

I wish everybody good weekend, saturday I not will be have near by internet.

Attila

NorbertMarkus commented 1 year ago

Summary of the tests that remained XFAIL in the tests/mathml_nemeth directory

Please find below a list of the tests I could not get to pass, grouped by type of issue. In the accompanying comments, I am going to try to suggest a kind of roadmap to addressing these issues. Excuse me for not having sorted the groups by difficulty of implementation. Note that some of the issues are over/underrepresented in the tests. Their frequency of occurrence in real-world documents may be significantly different, and there are issues which aren't covered in the tests at all.

Binomial coefficient

general_002
general_003

Comment: Binomial coefficients are shown differently in Nemeth than in print. A solution covering the simpler cases may be worked out even in this PR but it may fail at more complex expressions. Symbols at the beginning, the middle and the end of the expression need to be checked, while almost anything may occur inbetween. Once they all qualify (which must be known at the beginning of the expression), Braille symbols in all these three places need to be transformed.

Complex and hypercomplex fractions

mfrac_04
mfrac_06
mfrac_07

Comment: This appears to be a very difficult matter as it requires a reverse counter for tracking levels. For the proper marking of nested fractions in Nemeth, such fractions need first to be parsed because at the beginning of the outer most fraction, the order of the deepest nested fraction must already be known. The outer most fraction has the longest prefixes while the inner most fractions have none. Plus, there are further complicating factors such as the fractions at different subscript/superscript levels have their own, independent nesting structures. Thus this issue also depends on the tracking of script levels.

Multilevel subscripts and superscripts

mixed_scripts_01
mixed_scripts_02
mixed_scripts_03
mixed_scripts_05
mixed_scripts_06
mixed_scripts_07
msub_02
msub_04
msub_10
msubsup_03
msup_02
msup_03
msup_06
msup_09

Comment: This issue seems to be simpler to resolve, perhaps already in the next round of improvements. An algorithm would be ideal to process any script levels but with some planning, using more new variables and adding many lines to nemeth.ctb, at least the first 3-4 script levels could be covered which may be sufficient. A few other issues depend on this one. Altogether, over a dozen more tests may pass then.

Commas at superscript or subscript levels

msub_08

Comment: This issue depends on the tracking of script levels. Mathematical commas appearing other than at the base line level are Brailled differently in Nemeth (i.e., using a different dot pattern and no spacing).

Signs of comparison at the superscript or subscript level

msub_09
msubsup_05

Comment: This issue also depends on the tracking of script levels. A superscript/subscript indicator for the current script level needs to reappear before an equals sign or other signs of comparison, otherwise it reads to be at the base-line level.

mmultiscripts XML syntax

mmultiscripts_01
mmultiscripts_02
mmultiscripts_03
mmultiscripts_04
mmultiscripts_05
mmultiscripts_06
mmultiscripts_07

Comment: This set of issues may not be possible to tackle within the current capacity of the nemeth.ctb, nemeth_edit.ctb and nemeth.sem files, in other words, without upgrading liblouisutdml. I may be able to develop a preprocessor in Java or C# to translate the mmultiscripts structures in the input to chains of msub, msup and msubsup elements that liblouisutdml can already handle but this won't be an integral part of liblouisutdml. However, it would be more ideal to improve liblouisutdml itself as it could rely on the XML parsing but I am not qualified to modify and develop straight C code at that scale.

mstack

mover_05
mover_13
mover_15

Comment: The same applies to this issue as to mmultiscripts above.

Nested mover or munder

mover_07
munder_03
munderover_03

Comment: This issue also falls into the more difficult category as tackling these nested modification structures are even more complicated than complex and hypercomplex fractions. At each nesting level, three different places must be identified and the indications of multiple levels must be contracted. Plus, the Braille symbols involved often occur in other constructs as well, thus their distinction may pose a challenge.

Nested radicals

msqrt_03

Comment: This seems to be easier than nested fractions because it may not require prior parsing as the deeper levels are marked with longer prefixes and the outer most radical receives no level prefixes.

hammera commented 1 year ago

Boys, I don't no why, but sanitizer checks / Build with address sanitizer check are failed. The interesting thing if I ran compilation my local machine, I don't see any errors. For example if the main source code directory I doing a full compile with make check after doed git clean -fxd, ./autogen.sh, ./configure, make check command, don't see any error messages. If I do make check command the test directory, I see following right test-suite.log result:

================================================
   liblouisutdml 2.11.0: tests/test-suite.log
================================================

# TOTAL: 388
# PASS:  209
# SKIP:  0
# XFAIL: 179
# FAIL:  0
# XPASS: 0
# ERROR: 0

This is the right result now after integrated Neil Soiffer nemeth_MathCAT tests the test system.

But, when happens online pull request checking after push, address sanitizer drops following error the uploaded proper test-suite.log file with tests/nemeth_MathCAT/identity_matrix_126.test and tests/nemeth_MathCAT/table_entry_after_sup_79_c_4.test related. Have any error the source code with happens when address sanitizer executes only this two tests, or this two error messages a false positive error message?

Quoting the log, sorry the remaining long comment part:

test-suite-address.log errors:
ERROR: nemeth_MathCAT/identity_matrix_126
=========================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
=================================================================
==33872==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffd5f910e6c at pc 0x7fade57789db bp 0x7ffd5f910de0 sp 0x7ffd5f910dd0
READ of size 4 at 0x7ffd5f910e6c thread T0
    #0 0x7fade57789da in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261
    #1 0x7fade577fdba in styleBody /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3176
    #2 0x7fade5787666 in end_style /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3933
    #3 0x7fade5758b65 in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:111
    #4 0x7fade5758aaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #5 0x7fade5758aaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #6 0x7fade5757d19 in transcribe_document /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_document.c:88
    #7 0x7fade573e011 in processXmlDocument /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:180
    #8 0x7fade573e796 in lbu_translateFile /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:283
    #9 0x564ad20e7bb5 in main /home/runner/work/liblouisutdml/liblouisutdml/tools/file2brl.c:351
    #10 0x7fade52fad8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #11 0x7fade52fae3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #12 0x564ad20e6644 in _start (/home/runner/work/liblouisutdml/liblouisutdml/tools/.libs/file2brl+0x3644)

Address 0x7ffd5f910e6c is located in stack of thread T0 at offset 44 in frame
    #0 0x7fade577858f in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2202

  This frame has 2 object(s):
    [48, 448) 'colSize' (line 2212) <== Memory access at offset 44 underflows this variable
    [512, 1312) 'rowBuf' (line 2213)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261 in doAlignColumns
Shadow bytes around the buggy address:
  0x10002bf1a170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10002bf1a1c0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 f1[f1]00 00
  0x10002bf1a1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a200: f2 f2 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00 00 00
  0x10002bf1a210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==33872==ABORTING
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/identity_matrix_126.test (exit status: 99)

ERROR: nemeth_MathCAT/table_entry_after_sup_79_c_4
==================================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
=================================================================
==39240==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffe240a89cc at pc 0x7f88402a89db bp 0x7ffe240a8940 sp 0x7ffe240a8930
READ of size 4 at 0x7ffe240a89cc thread T0
    #0 0x7f88402a89da in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261
    #1 0x7f88402afdba in styleBody /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3176
    #2 0x7f88402b7666 in end_style /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:3933
    #3 0x7f8840288b65 in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:111
    #4 0x7f8840288aaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #5 0x7f8840288aaf in transcribe_math /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_math.c:92
    #6 0x7f8840287d19 in transcribe_document /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcribe_document.c:88
    #7 0x7f884026e011 in processXmlDocument /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:180
    #8 0x7f884026e796 in lbu_translateFile /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/liblouisutdml.c:283
    #9 0x559cc769fbb5 in main /home/runner/work/liblouisutdml/liblouisutdml/tools/file2brl.c:351
    #10 0x7f883fe2ad8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #11 0x7f883fe2ae3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #12 0x559cc769e644 in _start (/home/runner/work/liblouisutdml/liblouisutdml/tools/.libs/file2brl+0x3644)

Address 0x7ffe240a89cc is located in stack of thread T0 at offset 44 in frame
    #0 0x7f88402a858f in doAlignColumns /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2202

  This frame has 2 object(s):
    [48, 448) 'colSize' (line 2212) <== Memory access at offset 44 underflows this variable
    [512, 1312) 'rowBuf' (line 2213)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261 in doAlignColumns
Shadow bytes around the buggy address:
  0x10004480d0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10004480d130: 00 00 00 00 f1 f1 f1 f1 f1[f1]00 00 00 00 00 00
  0x10004480d140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d160: 00 00 00 00 00 00 00 00 00 00 00 00 f2 f2 f2 f2
  0x10004480d170: f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 00 00 00
  0x10004480d180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==39240==ABORTING
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/table_entry_after_sup_79_c_4.test (exit status: 99)

Attila

hammera commented 1 year ago

Other changes, with @NorbertMarkus doed lbu_files/nemeth_edit.ctb table, lbu_files/nemeth_edit_en_us_with_switch_indicators.ctb, and lbu_files/nemeth_edit_en_ueb_with_switch_indicators.ctb related, this issue not produce the address sanitizer error absolute sure. lbu_files/nemeth_edit.ctb related new change: Improved basic Nemeth math edit table for EBAE literary texts Better handles punctuation, grouping symbols and spacing around math blocks. Use nemeth_edit.ctb as editTable along with en-us-g1.ctb or en-us-g2.ctb as mathtextTable.

lbu_files/nemeth_edit_with_switch_indicators.ctb: renamed with lbu_files/nemeth_edit_en_us_with_switch_indicators.ctb table. Of course the lbu_files/Makefile.am file is edited.

lbu_files/nemeth_edit_en_us_with_switch_indicators.ctb: A new, improved Nemeth math edit table for EBAE literary texts with support for Nemeth switch indicators This table better handles spacing, punctuation, grouping and other symbols around math blocks. Plus, it borrows Nemeth switch indicators from UEB to mark the beginning and end of each math block, so one can always distinguish between literary text and technical content.

Use nemeth_edit_en_us_with_switch_indicators.ctb as editTable along with en-us-g1.ctb or en-us-g2.ctb as mathtextTable.

Final, @NorbertMarkus created a new table, lbu_files/nemeth_edit_en_ueb_with_switch_indicators.ctb table, he sent me following description this change related: lbu_files/nemeth_edit_en_ueb_with_switch_indicators.ctb: A brand new Nemeth math edit table supporting the production of "UEB with Nemeth" documents Adds support for UEB literary texts containing Nemeth blocks for technical content. Nemeth switch indicators are applied to mark the beginning and end of each math block, so one can always distinguish between literary text and technical content. Handles a range of punctuation, grouping and other symbols and their spacing around the math blocks.

Use nemeth_edit_en_ueb_with_switch_indicators.ctb as editTable along with en-ueb-g1.ctb or en-ueb-g2.ctb as mathtextTable.

Attila

hammera commented 1 year ago

I don't understand a thing: When I ran offline with ./autogen.sh, ./configure, make check commands, and ran the tests directory the make check command, the two wrote testcase files have into the xfail section. When ran the sanitizer checks / Build with address sanitizer Github workflows, this two tests produce errors. How can possible do offline similar compilation with do sanitizer checks / Build with address sanitizer to reproduce the issue, and detect why producing error this two testcases?

If not have possibility and capacity now to fixing the log file showed issue the proper source files with detected the sanitizer checks / Build with address sanitizer workflow, temporary need comment out the proper Makefiles.am files this two testcase files in test-suite system to we go on forward.

Attila

NorbertMarkus commented 1 year ago

Hi, After a few days of testing and experimenting, I found these as likely causes. In the nemeth_mathcat directory, two tests ("identity_matrix_126.test" and "table_entry_after_sup_79_c_4.test") cause program error. The input.xml of both tests contain the mtable MathML element, and all nemeth_mathcat tests are set to produce Unicode Braille output. I think the combination of these two factors cause the problem. The algorithm in liblouisutdml that calculates the column positions and deals with item aligning may only be prepared to encounter ASCII characters. The Unicode Braille characters may cause confusion. The problem is further complicated by the fact that the Braille processing of mtable implies the use of the editTable specified for liblouisutdml. As long as only literary text or only math is present in a paragraph, a single translation table or a filelist corresponding to it is used for the translation and that is straight forward. But as soon as a paragraph contains both literary text and math, three translation tables come into play: mathtextTable for the literary text, mathexprTable for the math expressions, and editTable in a second round for the corrections necessary where text and math meet. In this case, specifying any display table(s) is more than difficult. Edit tables for Nemeth expect to operate on Ascii materials of American Computer Braille. If unicode.dis is applied, the output becomes unreadable because of the edit table. Additional edit tables may be created based on Unicode Braille but this is beyond the scope of our current effort. When liblouisutdml encounters an mtable in MathML, it implicitly uses the edit table specified, even when the input only consists of a single MathML block, thus there is no surrounding text. There are further problems with how liblouisutdml creates math tables but they seem to be less interesting at this stage. These include that liblouisutdml unconditionally formats math tables (also matrices and systems of equations) into multiline Braille output, and uses large square brackets as enclosing symbols around the math table, regardless of what has or has not been specified in the MathML source. Thus, I can't see how these two test could be fixed shortly.

hammera commented 1 year ago

What shows following error messages? I not quote from entire test-suites error log file with produce address sanitizer, but very interesting:

test-suite-address.log errors:
ERROR: nemeth_MathCAT/identity_matrix_126
=========================================

warning: lou_getDataPath is deprecated.
warning: lou_getDataPath is deprecated.
=================================================================
==33872==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffd5f910e6c at pc 0x7fade57789db bp 0x7ffd5f910de0 sp 0x7ffd5f910dd0

In near end of log file have following content, with interesting too:

  This frame has 2 object(s):
    [48, 448) 'colSize' (line 2212) <== Memory access at offset 44 underflows this variable
    [512, 1312) 'rowBuf' (line 2213)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /home/runner/work/liblouisutdml/liblouisutdml/liblouisutdml/transcriber.c:2261 in doAlignColumns
Shadow bytes around the buggy address:
  0x10002bf1a170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10002bf1a1c0: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 f1[f1]00 00
  0x10002bf1a1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10002bf1a200: f2 f2 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00 00 00
  0x10002bf1a210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==33872==ABORTING
Invocation of file2brl failed with error code 0
ERROR nemeth_MathCAT/identity_matrix_126.test (exit status: 99)

Attila

hammera commented 1 year ago

Forgot to wrote: I not have experience in C language, my primary programming language is Python2 and Python3, not often I analyze and debugging C source codes. So I not talking c language. :-):-)

Attila

hammera commented 1 year ago

Vow, now worked the address sanitizer (workaround solution only, until not awailable a fix in source code level). With you see in tests/Makefile.am source file, removed two nemeth_MathCAT tests, with produces the address sanitizer issue. Following testcase files removed only the tests/Makefile.am file: tests/nemeth_MathCAT/identity_matrix_126.test tests/nemeth_MathCAT/table_entry_after_sup_79_c_4.test This files removed now the last commit both the normal test part and the xfail test part.

Attila

NorbertMarkus commented 1 year ago

I am just done for now. I don't plan to add or modify anything else in this round unless it occurs to be absolutely necessary. Thus we can move on with this PR on my behalf.

hammera commented 1 year ago

Boys, I confirm @NorbertMarkus wrote comment, we I thing ready this PR, so we would like ask a final review. We done I thing this PR related work, so ask a last final review before merge. I thing we doed all things with possible this PR related, and hopefully not forgotted any previous you oldest time review requested changes implementation. We integrated too with Neil's nemeth_MathCAT tests from #87 related PR too, based with Bert review requested change.

We have now other works with other independent projects, so we only will doing commit and changes this PR related if absolute critical needs this. So, if during final review you requesting a change, will we be doing the required change, and because now we finalized and freezed the new tables, we will not start doing new implementation work on our own beards. For example will be we not adding new elements, difficult rules the already doed and thorough tested tables.

Thanks the cooperation and patience the lot of commits related,

Attila

hammera commented 1 year ago

Because two new nemeth_MathCAT tests trigger address sanitizer, I removed two test files from tests/Makefile.am Affected testcase files: nemeth_MathCAT/identity_matrix_126.test nemeth_MathCAT/table_entry_after_sup_79_c_4.test

This two tests produces xfail result, but when address sanitizer ran, produce error result. Oldest time we experienced this when worked the nemeth_MathCAT related integration. Log result in awailable with #98 related PR. Short diff output with I committed my feature branch, until not have better solution:

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 8a5951d..e148603 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -173,7 +173,6 @@ dist_suite_TESTS =              \
    nemeth_MathCAT/function_space_119_c_3.test          \
    nemeth_MathCAT/german_base_77_4_3.test          \
    nemeth_MathCAT/hyper_complex_frac_68_a_1.test           \
-   nemeth_MathCAT/identity_matrix_126.test         \
    nemeth_MathCAT/integral_77_4_26.test            \
    nemeth_MathCAT/left_sup_75_12.test          \
    nemeth_MathCAT/left_sup_75_1.test           \
@@ -295,7 +294,6 @@ dist_suite_TESTS =              \
    nemeth_MathCAT/sub_sup_82_b_2.test          \
    nemeth_MathCAT/sum_77_4_23.test         \
    nemeth_MathCAT/superscript_80_a_2.test          \
-   nemeth_MathCAT/table_entry_after_sup_79_c_4.test            \
    nemeth_MathCAT/tensor_from_mathml_spec.test         \
    nemeth_MathCAT/test_9_d_2.test          \
    nemeth_MathCAT/test_9_d_3.test          \
@@ -507,7 +505,6 @@ XFAIL_TESTS +=                  \
    nemeth_MathCAT/function_space_119_c_3.test          \
    nemeth_MathCAT/german_base_77_4_3.test              \
    nemeth_MathCAT/hyper_complex_frac_68_a_1.test           \
-   nemeth_MathCAT/identity_matrix_126.test             \
    nemeth_MathCAT/integral_77_4_26.test                \
    nemeth_MathCAT/left_sup_75_1.test               \
    nemeth_MathCAT/left_sup_75_12.test              \
@@ -593,7 +590,6 @@ XFAIL_TESTS +=                  \
    nemeth_MathCAT/sub_ind_80_b_4.test              \
    nemeth_MathCAT/sub_ind_mmultiscripts_80_b_3.test        \
    nemeth_MathCAT/superscript_80_a_2.test              \
-   nemeth_MathCAT/table_entry_after_sup_79_c_4.test        \
    nemeth_MathCAT/tensor_from_mathml_spec.test         \
    nemeth_MathCAT/test_9_d_3.test                  \
    nemeth_MathCAT/text_after_sup_79_c_3.test           \

Attila

hammera commented 1 year ago

Fine, now all checks passed, this I wanted. :-):-) On this Saturday and Sunday I not will be alwais near with internet, I wish everyone a good rest.

Attila

bertfrees commented 1 year ago

I'm in the middle of doing a clean up (clean up commit messages, make sure each commit passes tests). I'm not done yet but already pushed what I did.

hammera commented 1 year ago

Bert, thanks the cleanup. You not have ydea after force-push operation why failing following check? When I aug 17 commit the address sanitizer trigger related change after nemeth_MathCAT tests merge, all checks passed. Following check now not ran succesfull:

Make check / Build with liblouis master (pull_request) Failing after 6m
[Details](https://github.com/liblouis/liblouisutdml/actions/runs/5996460079/job/16261027242?pr=96)

What happened before or after force push?

Attila