search

scientificware / jfx

https://openjdk.java.net/projects/openjfx/
GNU General Public License v2.0
0 stars 0 forks source link

OpenJFX/JavaFX and MathML. #7

Open scientificware opened 6 years ago

scientificware commented 6 years ago

Details on OpenJFX / JavaFX and MathML. #71

scientificware commented 6 years ago

41190255-70c588d8-6bdb-11e8-8e2c-ec95235a391e

scientificware commented 6 years ago

Another required test.

/*
 * Copyright (c) 2018 - 2022 Guy Abossolo Foh - ScientificWare. All rights reserved.
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 *
 * This code is free software; you can redistribute it and/or modify it
 * under the terms of the GNU General Public License version 2 only, as
 * published by the Free Software Foundation.  ScientificWare designates this
 * particular file as subject to the "Classpath" exception as provided
 * by ScientificWare in the LICENSE file that accompanied this code.
 *
 * This code is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
 * version 2 for more details (a copy is included in the LICENSE file that
 * accompanied this code).
 *
 * You should have received a copy of the GNU General Public License version
 * 2 along with this work; if not, write to the Free Software Foundation,
 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 * Please contact ScientificWare France
 * or visit www.scientificware.com if you need additional information or have any
 * questions.
 */

package test.javafx.scene.web;

import org.junit.Assert;

import org.junit.Test;

public class MathMLRenderTest extends TestBase {

    // Document test
    static String BodyContent = """
        <math display=\"block\">
            <mrow>
                <mi>x</mi>
                <mo>=</mo>
                <mfrac>
                   <mrow>
                      <mo>−</mo>
                      <mi>b</mi>
                      <mo>±</mo>
                      <msqrt>
                         <mrow>
                            <msup>
                               <mi>b</mi>
                               <mn>2</mn>
                            </msup>
                            <mo>−</mo>
                            <mn>4</mn>
                            <mi>a</mi>
                            <mi>c</mi>
                         </mrow>
                      </msqrt>
                   </mrow>
                   <mrow>
                      <mn>2</mn>
                      <mi>a</mi>
                   </mrow>
                </mfrac>
            </mrow>
        </math>
        """;

    static String content = """
        <!doctype html>
        <html>
           <body>
              <p>
                  %s
              </p>
           </body>
        </html>
        """.formatted(BodyContent);

    @Test public void testgetTokenHeight() throws Exception {

        submit(() -> {
            loadContent(content);
        });

        int height = (int) executeScript(
            "elements = document.getElementsByTagName('mo');"
            + "element = elements[0].clientHeight;"
        );

        boolean rightRender = !(tokenHeight.get()<2)
        Assert.assertTrue("Check MathML token height : " + tokenHeight.get() + " is much smaller than the expected size." , rightRender);      
    }

}
scientificware commented 6 years ago

The new issue I posted in the javafxports/openjdk-jfx repository HTMLEditor and MathML. #118

scientificware commented 6 years ago

When I discovered that JavaFX could manage MathML, I also discovered that it was buggy. Becauseof MathML is rendered with WebKit, I compared with other WebKit based web browsers. I also noticed that since 2016 that those web browser make good progress with MaltML display. First I thought that the differences were due to outdated WebKit version used by JavaFX but last releases, with an uptodate WebKit version, show that the problem is the same.

I contacted F. Wang from Igalia who worked to port or improve MathML support in FireFox and later in WebKit. He confirms that it was a bug in JavaFX not in WebKit witch displays MathML correctly.

As he suggested me, I wrote to the JavaFX mailing list and had a short discussion about this issue on OpenJDK.Java.net :

http://mail.openjdk.java.net/pipermail/openjfx-dev/2017-December/021112.html https://bugs.openjdk.java.net/browse/JDK-8147476

So, as suggested by mainteners, I'm trying to find by myself what's wrong with MathML in javaFX HTMLEditor.

I tested all applications with Mozilla MathML Torture Test (https://developer.mozilla.org/fr/docs/Mozilla/MathML_Project/MathML_Torture_Test)

Could someone help me ?

scientificware commented 6 years ago

I think that the problem is related only with the vertical size of each element : I notice that the size of the cursor is right outside the mathematics text but when we move it through the mathematics text, the cursor size is far much lower than text size. All horizontal sizes seem good, by example a \<mtext> followed by a \<mrow> in a element : screen_javafx_mathml_2

Which part of code deals with vertical size ?

scientificware commented 6 years ago

Arunprasad Rajkumar comment.

Problem is current JavaFX font implementation for WebKit doesn't supports OpenMathData table.

Actually WebKit has it's own MathData table parser(OpenMathData.cpp) to extract MathML rendering parameters(like numeratorGap, denominatorGap,..etc), however it expects platform implementation(javafx) to return the OpenMathData table from OpenType font, which we don't support right now.

There is a fallback logic implemented when MathData is not available from platform implementation, however it seems to be buggy.

1 2

For e.g., to render the above simple mathml tocken, RenderMathMLFraction::fractionParameters(RenderMathMLFraction.cpp) method will be called to calculated the offsets. Since we don't support MathData table, the fallback logic to calculate the denominatorGapMin as follows,

    parameters.numeratorGapMin = display ? 3 * ruleThicknessFallback() : ruleThicknessFallback();
    parameters.denominatorGapMin = parameters.numeratorGapMin;

The above logic is not correct, because of that both numerator and denominator of a fraction is overlapping. Just adding a some offset to parameters.numeratorGapMin fixes the issue.

So the appropriate solution would be to implement a logic to extract a OpenMathData table from OpenType font table.

scientificware commented 6 years ago

Thank you very much for your help, and all these informations ! Thanks given me an entry point to investigate :-)

scientificware commented 6 years ago

First working comments :

scientificware commented 6 years ago

Configure my build and run environment to debug. Mageia 6, all the required development packages

Mageia package name Description
lib64alsa2-devel Development files for Advanced Linux Sound Architecture (ALSA)
lib64mesagl1-devel Development files for Mesa (OpenGL compatible 3D lib)
lib64gstreamer1.0-devel Libraries and include files for GStreamer streaming-media framework
lib64gstreamer-plugins-base1.0-devel GStreamer Plugin Library Headers​
lib64jpeg-devel Development tools for programs which will use the libjpeg library
lib64png-devel Development tools for programs to manipulate PNG image format files
lib64x11-devel Development files for libx11
lib64xml2-devel Libraries, includes, etc. to develop XML and HTML applications
lib64xslt-devel Libraries, includes, etc. to develop XML and HTML applications
lib64xt-devel Development files for libxt
lib64xxf86vm-devel Development files for libxxf86vm
pkgconfig Pkgconfig helps make building packages easier
x11-proto-devel Xorg X11 protocol specification headers
lib64ffmpeg-static-devel Static library for the ffmpeg codec library
mercurial A fast, lightweight distributed source control management system
lib64gtk+2.0-devel Development files for GTK+ (GIMP ToolKit) applications
lib64gtk+3.0-devel Development files for GTK+ (GIMP ToolKit) applications
lib64xtst-devel Development files for libxtst
lib64udev-devel udev library development files
lib64ffmpeg-static-devel Static library for the ffmpeg codec library
gcc-c++ C++ support for gcc
.
To build WebKit
bison A GNU general-purpose parser generator
flex A tool for creating scanners (text pattern recognizers)
gperf A perfect hash function generator
ruby Object Oriented Script Language
libstdc++-static-devel Static libraries for C++ development

Ok

Try to run my application with a JavaFX patched WebCore :

java @run.args JavaFXMathML

Error message :

public class JavaFXMathML extends Application {
       ^
1 error
Exception in Application start method
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:464)
        at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:363)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at java.base/sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:941)
Caused by: java.lang.RuntimeException: Exception in Application start method
        at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:900)
        at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(LauncherImpl.java:195)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.IllegalAccessError: class com.sun.javafx.scene.control.Logging (in module javafx.controls) cannot access class com.sun.javafx.logging.PlatformLogger (in module javafx.base) because module javafx.base does not export com.sun.javafx.logging to module javafx.controls
        at javafx.controls/com.sun.javafx.scene.control.Logging.getControlsLogger(Logging.java:47)
        at javafx.controls/javafx.scene.control.Control$2.invalidated(Control.java:316)
        at javafx.base/javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:112)
        at javafx.base/javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:147)
        at javafx.graphics/javafx.css.StyleableObjectProperty.set(StyleableObjectProperty.java:82)
        at javafx.controls/javafx.scene.control.Control$2.set(Control.java:250)
        at javafx.controls/javafx.scene.control.Control$2.set(Control.java:233)
        at javafx.controls/javafx.scene.control.Control.loadSkinClass(Control.java:757)
        at javafx.controls/javafx.scene.control.Control$5.invalidated(Control.java:680)
        at javafx.base/javafx.beans.property.StringPropertyBase.markInvalid(StringPropertyBase.java:110)
        at javafx.base/javafx.beans.property.StringPropertyBase.set(StringPropertyBase.java:145)
        at javafx.graphics/javafx.css.StyleableStringProperty.set(StyleableStringProperty.java:83)
        at javafx.controls/javafx.scene.control.Control$5.set(Control.java:672)
        at javafx.graphics/javafx.css.StyleableStringProperty.applyStyle(StyleableStringProperty.java:69)
        at javafx.graphics/javafx.css.StyleableStringProperty.applyStyle(StyleableStringProperty.java:45)
        at javafx.web/javafx.scene.web.HTMLEditor.<init>(HTMLEditor.java:50)
        at JavaFXMathML.start(JavaFXMathML.java:26)
        at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
        at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:451)
        at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:424)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:423)
        at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
        at javafx.graphics/com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
        at javafx.graphics/com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11(GtkApplication.java:277)
        ... 1 more
scientificware commented 6 years ago

Kevin Rushforth comment

Not sure why you are using make_runargs.sh -- that script is usually meant for taking your own build of JavaFX done on one machine and copying it to another where the path to the FX modules is different (since run.args has absolute paths).

In any case, the problem seems to be that the exports from buildSrc/addExports is not being included in the generated run.args as it should be, which is a bug in make_runargs.sh. You can append the needed exports from that file to your generated run.args, or just not use it in the first place and take the build/run.args from your JavaFX build and manually adjust the paths as needed.

scientificware commented 6 years ago

Thanks a lot and sorry. You're right, I made three big mistakes :

It works well with build/run.args from my JavaFX build.

scientificware commented 6 years ago

Some working documents :

MathML Refactoring In WebKit (pdf) MathML Improvements In WebKit (pdf)

scientificware commented 6 years ago

If you want to build WebKit :

Before to run gradle, edit the file in gradle.properties.template in rt/ and save it as "rt/gradle.properties" :

# These properties give developers the chance to skip building WebKit and/or
# GStreamer. WebKit takes a fair amount of time to build (more than 50% of the
# overall full build time is taken by WebKit), so allowing a developer to
# selectively enable building of WebKit is important. To build WebKit or
# GStreamer, uncomment the appropriate lines below.

COMPILE_WEBKIT = true
scientificware commented 6 years ago
scientificware commented 6 years ago
scientificware commented 6 years ago
scientificware commented 6 years ago
scientificware commented 6 years ago

Change c++ option : -fno-rtti

Where is the best place to change this option ? …/javafx.web/src/main/native/Source/cmake/WebKitCompilerFlags.cmake : Change WEBKIT_APPEND_GLOBAL_CXX_FLAGS(-fno-rtti) to # WEBKIT_APPEND_GLOBAL_CXX_FLAGS(-fno-rtti)

scientificware commented 6 years ago

Arunprasad Rajkumar comment.

Do you want to remove the C++ RTTI? Then you can modify the same from WebKitCompilerFlags.cmake#line105. BTW, why do you want to remove that?

To explore the chagelog of a WebKit, please refer this. Current JavaFXWebView is based on GTK WebKit 2.18 branch.

scientificware commented 6 years ago

Hi, Thank you very much,

Yes, I want to remove -fno-rtti, in order to use typeid to identify child in RenderMathMLBlock::layoutItems and understand why height is set to 1.

Have you got a specific configuration (Netbeans or others) to debug JavaFX/WebKit ?

Best regards.

Read the Wiki ! Using an IDE Of course ! This is the second time I forget that basic advice

scientificware commented 6 years ago

The child of a RenderMathMLBlock is an RenderBlockFlow.

scientificware commented 6 years ago

I think that I gone a bit farther. The problem only affects MathML elements : RenderBox, RenderBlockFlow, RenderBlock, ... work fine for all element types other than MathML elements. I'm Trying something rather simple this morning.

scientificware commented 6 years ago

I think that MathML Renders are cast in an inappropiate objects.

contains(DEFINES, ENABLE_MATHML=1) {
    SOURCES += \
        css/CSSDefaultStyleSheets.cpp \
        accessibility/AXObjectCache.cpp \
        accessibility/AccessibilityNodeObject.cpp \
        accessibility/AccessibilityObject.cpp \
        accessibility/AccessibilityRenderObject.cpp \
        dom/Document.cpp \
        mathml/MathMLInlineContainerElement.cpp \
        mathml/MathMLAnnotationElement.cpp\
        mathml/MathMLElement.cpp \
        mathml/MathMLFractionElement.cpp \
        mathml/MathMLMathElement.cpp \
        mathml/MathMLMencloseElement.cpp \
        mathml/MathMLOperatorDictionary.cpp \
        mathml/MathMLOperatorElement.cpp \
        mathml/MathMLPaddedElement.cpp \
        mathml/MathMLPresentationElement.cpp \
        mathml/MathMLRowElement.cpp \
        mathml/MathMLScriptsElement.cpp \
        mathml/MathMLSelectElement.cpp \
        mathml/MathMLSpaceElement.cpp \
        mathml/MathMLTokenElement.cpp \
        mathml/MathMLUnderOverElement.cpp \
        rendering/RenderBox.cpp \
        rendering/RenderTreeAsText.cpp \
        rendering/RenderObject.h \
        rendering/RenderTableCell.cpp \
        rendering/mathml/MathMLStyle.cpp \
        rendering/mathml/MathOperator.cpp \
        rendering/mathml/RenderMathMLBlock.cpp \
        rendering/mathml/Thermoluminescence.cpp \
        rendering/mathml/RenderMathMLFencedOperator.cpp
        rendering/mathml/RenderMathMLFraction.cpp \
        rendering/mathml/RenderMathMLMath.cpp \
        rendering/mathml/RenderMathMLMenclose.cpp \
        rendering/mathml/RenderMathMLOperator.cpp \
        rendering/mathml/RenderMathMLPadded.cpp
        rendering/mathml/RenderMathMLRoot.cpp \
        rendering/mathml/RenderMathMLRow.cpp \
        rendering/mathml/RenderMathMLScripts.cpp \
        rendering/mathml/RenderMathMLSpace.cpp \
        rendering/mathml/RenderMathMLToken.cpp \
        rendering/mathml/RenderMathMLUnderOver.cpp
}
scientificware commented 6 years ago

Qt [Wiki]

Qt Documentation :

scientificware commented 6 years ago

Do not add these files which are already added in SOURCES

# css/CSSDefaultStyleSheets.cpp \
# accessibility/AXObjectCache.cpp \
# accessibility/AccessibilityNodeObject.cpp \
# accessibility/AccessibilityObject.cpp \
# accessibility/AccessibilityRenderObject.cpp \
# dom/Document.cpp \

Do not add this file which has been renamed to MathMLPresentationElement

# mathml/MathMLInlineContainerElement.cpp \
scientificware commented 6 years ago

Do not add these files which are already added in SOURCES

# css/CSSDefaultStyleSheets.cpp \
# accessibility/AXObjectCache.cpp \
# accessibility/AccessibilityNodeObject.cpp \
# accessibility/AccessibilityObject.cpp \
# accessibility/AccessibilityRenderObject.cpp \
# dom/Document.cpp \

Do not add this file which has been renamed to MathMLPresentationElement

# mathml/MathMLInlineContainerElement.cpp \
scientificware commented 6 years ago
scientificware commented 6 years ago

I'm looking for what is the function of WTFMove. WTF : Web Template Framework WTFMove is a macro defined in StdLibExtras.h : #define WTFMove(value) std::move<WTF::CheckMoveParameter>(value)

Bug 152601 - Use of WTF::move prevents clang's move diagnostics from warning about several classes of mistakes

scientificware commented 6 years ago
scientificware commented 6 years ago 
void RenderMathMLToken::layoutBlock(bool relayoutChildren, LayoutUnit pageLogicalHeight)
{
    ASSERT(needsLayout());

    if (!relayoutChildren && simplifiedLayout())
        return;

    GlyphData mathVariantGlyph;
    if (m_mathVariantCodePoint)
        mathVariantGlyph = style().fontCascade().glyphDataForCharacter(m_mathVariantCodePoint.value(), m_mathVariantIsMirrored);

    if (!mathVariantGlyph.font) {
        RenderMathMLBlock::layoutBlock(relayoutChildren, pageLogicalHeight);
        return;
    }

    for (auto* child = firstChildBox(); child; child = child->nextSiblingBox())
        child->layoutIfNeeded();

    setLogicalWidth(mathVariantGlyph.font->widthForGlyph(mathVariantGlyph.glyph));

    setLogicalHeight(mathVariantGlyph.font->boundsForGlyph(mathVariantGlyph.glyph).height());
    clearNeedsLayout();
}
    for (auto* child = firstChildBox(); child; child = child->nextSiblingBox())
        child->layoutIfNeeded();

    setLogicalWidth(mathVariantGlyph.font->widthForGlyph(mathVariantGlyph.glyph));

    setLogicalHeight(mathVariantGlyph.font->boundsForGlyph(mathVariantGlyph.glyph).height());
    clearNeedsLayout();
scientificware commented 6 years ago

I'm right !

mathml_javafx_20180609

This patch doens't solve all problems but we can at least display elementary school mathematic formulars.

I think like Arunprasad Rajkumar that to go further, javafx will have to support math table.

Touch down !

scientificware commented 5 years ago

My Pull Request

Changed I made :

Test I designed :