Support for MathJax 3.0

[eli-schwartz]

See the release announcement: https://github.com/mathjax/MathJax/releases/tag/3.0.0

Given the porting notes specifically reference the deprecation of ?config=, I guess this will need work.

It sounds like the new version should be a lot faster, though!

[dougmassay]

The conversion of our v.2 config goes smoothly enough, but the comments from that conversion process don't look very promising yet:

https://mathjax.github.io/MathJax-demos-web/convert-configuration/convert-configuration.html

• Jax output/PreviewHTML is not yet implemented
• Extension MathML/mml3 is not yet implemented (but may be in a future release)
• Extension fast-preview is not yet implemented (but may be in a future release)

Pretty sure we can't claim EPUB3 MathML spec compliance if we don't support the things provided by mml3.

Will have to keep an eye on it.

[eli-schwartz]

Well, thanks for the speediness of your response at least. :) These results do seem to be a bit disappointing... if you can dig up anything else then I'm all ears. But I know basically nothing about mathjax so whatever you say is good with me. :)

[dougmassay]

I'm pretty weak on MathJax/MathML myself. But that shouldn't stop me from being able jury-rig Sigil to load version 3 and see how (or if) it handles the various epub3 mathml tests. I can at least see if it passes the mandatory epub3 requirements for MathML.

[kevinhendricks]

According to their website:

The mml3 and content-mml extensions for the MathML input jax are not yet available in version 3. We do hope to have these in a future release.

And without mml3 support, we can not pass all of the epub3 mathml tests for long div, and etc.

So we will be sticking with our current mathjax until they at least get mml3 support back.

[kevinhendricks]

See this PR, which once merged will allow us to move to the MathJax 3.x series.

https://github.com/mathjax/MathJax-src/pull/706/commits/1cd704fdc772e00be73e19e0a4ae4c5e609cc224

and the Issue discussion,

https://github.com/mathjax/MathJax/issues/2515

[dougmassay]

We're talking about waiting on them to officially merge it, though right? If we cherry-pick it, we'd be able to support MathJax 3.0 in our bundled installers. But it would probably break Sigil MathML support on distros like Arch that decouple our bundled MathJax in favor of their own stock MathJax.

[eli-schwartz]

As we've been waiting nearly 2 years for this, it would be great if we could also wait a couple... weeks? for it to land upstream, yes.

[eli-schwartz]

@mapreri last I checked Debian does not provide a mathjax 3 package (this was relevant for calibre, as I discussed with your calibre maintainer when calibre switched to v3) so you may need to get the wheels rolling to create one, or else stop using the system mathjax in future versions of sigil.

[dougmassay]

If at all possible, I'd try to make sure that those who decouble MathJax could continue to use 2.0 by default. I'm sure it would change the url used, but there's already url-building logic in place for Linux to use the MathJax bundled with Sigil, or to define their own MathJax location (I think), so adding a compile flag to opt-in to 3.0 shouldn't be impossible.

[eli-schwartz]

Oh hmm, that sounds interesting.

[dougmassay]

I'm not entirely sure it's feasible (without diving right into it), but that would be my preference. I'm not really sure how many of the distros debundle MathJax from Sigil to be honest.

[mapreri]

If it's possible to keep sigil compatible with both mathjx versions, that would be great. I'm not sure weather system-installed mathjax 3 would be in a different directory (currently it's /usr/share/javascript/mathjax/MathJax.js which is unversioned).

Anyway, I'll try to talk with mathjax's maintainer in Debian (here: @mitya57 ! o/). @eli-schwartz I don't see any open bug about mathjax 3: do you know where the previous discussion happened, or have any more details on that conversation? Or was it just talking with YOKOTA Hiroshi?

[eli-schwartz]

I forget where I discussed this with Norbert Preining, the calibre maintainer that I remember (I do not know this other person at all), but the outcome of the discussion was https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963684

Apparently there are 3 bugs for it? The first one being submitted by mitya57 as he wants to pass the torch on to someone else.

[mapreri]

Ah, I see... Thank you for the pointer.

It's also unlikely for me to start maintaining mathjax3 myself, so if sigil can't keep the compatibility with v2 I guess I'll to find some other solution (like embedding it, but I might have policy issues there to solve... I suppose sigil will ship only the minified/compiled version, which we can't redistribute :/)

On Sun, 16 May 2021, 5:28 pm Eli Schwartz, @.***> wrote:

I forget where I discussed this with Norbert Preining, the calibre maintainer that I remember (I do not know this other person at all), but the outcome of the discussion was https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963684

Apparently there are 3 bugs for it? The first one being submitted by mitya57 as he wants to pass the torch on to someone else.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sigil-Ebook/Sigil/issues/453#issuecomment-841832648, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7FEYSX66WBQU2QLAPHZ3TN7P6ZANCNFSM4ITXFJIQ .

[mitya57]

The bug link is already here, and I hope someone will package mathjax3. I'm having too many packages already and too little time :(

I'm not sure weather system-installed mathjax 3 would be in a different directory

It is incompatible with MathJax 2, so it should be a new source package and a different directory.

ArchLinux uses /usr/share/mathjax, maybe Debian can use the same path for compatibility.

[eli-schwartz]

Arch Linux uses that path for the unversioned mathjax, btw :) we used to use that for mathjax 2.x but upgraded it to mathjax 3.x and moved mathjax 2.x to /usr/share/mathjax

The /javascript/ directory has historically been a debian multi-package convention iirc... Arch doesn't do much javascript packaging so mathjax is kind of on its own and is packaged in /usr/share/${package_name}/

[dougmassay]

@kevinhendricks : Can you give me an example (sometime, no hurry here) of how you imagine Sigil will inject MathJax3 into Preview whenever an epub requires it? Once I see how that's done with 3, I'll be able to work out what might, or might not, be able to be done to keep Sigil compatible with 2.x as well (when debundling on linux).

[kevinhendricks]

I assume it will be similar to how we do it currently. I will wait until that MathJax PR is merged and actually in a released version. So no rush! This is not something for the next release of Sigil (1.6.0) anyway. I was just trying to clean up some of open Sigil issues. If we need to keep support for both we will.

FWIW, my comment that it was a regression in Mathjax issues touched a nerve there where it was perceived as "rude and demanding". I guess I am too old to understand today's developer sensitivities. To me, waiting 2 years for them to add back a feature that existed in a released version that they promised to add back when dropped, and then finally pointing out it is a "regression" in an issue is not being rude.

I simply can not imagine that ever happening at IBM in that late 70s and early 80s where we were expected to actually track down and fix bugs that were filed and never once would you call a bug filer rude or demanding. Just the opposite ... we were always taught to thank bug reporters for helping to improve your project!

But times have changed, so what do I know! I did apologize.

If they truly do feel that way, we may never see that PR merged!

So hold off worrying about this until we actually see it merged.

All of this because I wanted to shrink our outstanding issue list! Teach me to fix those issues!

[dougmassay]

No worries. We'll figure it out eventually! ;)

I'm just glad we didn't go with a "Sponsors First" model for prioritizing Sigil development.

[kevinhendricks]

I see Mathjax 3.2 released in July of 2021 has the mml3 support we need. After Sigil-1.9.0 is released, I am going to start work integrating MathJax 3.2 into our build process to replace the old 2.5 version we currently use.

I would assume, Arch, Manjero, etc already have MathJax 3.2 or later integrated. Older, non-rolling Linux distributions should be able to build and work with our integrated version if they do not ship a MathJax 3.2 (earlier Mathjax 3 versions will not work).

Alternatively, we could try to support both Mathjax 2.X and Mathjax 3.2 in some manner if using a Sigil integrated Mathjax on older Linux distributions can not be done.

Any thoughts?

[kevinhendricks]

Here is a list for mathjax from repology in case it helps.

https://repology.org/project/mathjax/versions

[kevinhendricks]

Before worrying about this more, I decided to run some tests with the latest MathJax 3.2 and it is a mess. It does not even handle the first (most basic) mathml test from the epub3 testsuite well (no mml3 required). And their newly added back mml3 extension does not seem to work well either. This is with mml-svg.

I tested making my own single file custom build and and also tested with the version served up via CDN and both failed when laying out the quadratic equation, laying out the infinite series sum, etc. The older MathJax 2.75 seems to work much better in all regards.

So, I can see why no one is upgrading to MathJax 3.2 yet.

I am going to close this as I can not see why upgrading to a very broken version makes any sense, even after 2+ years of waiting for them to get something working well.

I will re-open this when they get an actual version that works as well as their 2.7.X series. Very sad.

MathJax-2.75 mml-svg

MathJax-3.2 mml-svg

[kevinhendricks]

FWIW, mathjax github site both master and develop branches have not seen a commit to either since September of 2021.

The last working svg output format was in MathJax version 3.0.5 and since MathJax 3.1.0 the svg output format has been broken. I tested this with official releases in both PageEdit and Safari.

So it appears the only work is on the chtml output version.

I tested MathJax 3.2 with mml-chtml.js and see the following: MathJax 3.2 - chtml output

[dougmassay]

Hi Kevin,

Sad indeed. Probably for the best to wait. The very low adoption numbers via repology had me leaning toward supporting both versions if we were going to adopt 3.2. I'd be fine with expecting individual Linux users to use our bundled version of 3.2 if it wasn't available through their regular channels. But some package maintainers might have kicked up their heels at a bundled mathjax being their only choice.

But like you said: it makes no sense to "upgrade" to a broken version in the first place.

Thanks, Doug

On Mon, Feb 14, 2022, 12:03 PM Kevin Hendricks @.***> wrote:

FWIW, mathjax github site both master and develop branches have not seen a commit to either since September of 2021.

— Reply to this email directly, view it on GitHub https://github.com/Sigil-Ebook/Sigil/issues/453#issuecomment-1039328479, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACG3CXSXVRURTI2WTS7NBBDU3EYW5ANCNFSM4ITXFJIQ . You are receiving this because you commented.Message ID: @.***>

[kevinhendricks]

For the record, I have attached a zip archive testmj.zip that includes a variety of mathml tests.

To try these simply use a Browser to "open File" any of the xhtml files inside of its text folder. The file names explain what you are testing. These can also be tested by using PageEdit as well to see what QWebEngineView handles it. The css and img folders hold the images of what a correct response would look like.

testmj.zip

[kevinhendricks]

And for the record, I have posted that test case and a bug report here:

https://github.com/mathjax/MathJax/issues/2836

[dpvc]

both master and develop branches have not seen a commit to either since September of 2021

Just FYI this is not actually true. This is the network diagram for the MathJax-src repository showing that PRs were merged just two weeks ago

and also that there has been quite a bit of activity in terms of branches to fix issues and add features. These are awaiting code review, but the fact that they have not yet landed does not mean there is no activity. It is true that in the fall we have fewer resources (both financially and in terms of time, as I go back to teaching in the fall months). The MathJax project only has two programmers, and so we can not address every need with the alacrity that the requester may desire. Yes, two years is a long time to wait, but it is also a long time to wait for line-breaking, font support, and all the other features we are still trying to include. MathJax v2 was 10 years in the making, and version 3 has only been out for 2 and a half yeas, so there is still a lot of catching up to do. I understand the frustration you may feel, but there is only so much we can do in any given amount of time, and yes, the needs of our supporters do have to be taken into account.

my comment ... there ... was perceived as "rude and demanding".

Note that it was Peter who made that comment about your messages, and while he was an important member of the project for some time, he left MathJax in 2017, and so has not represented MathJax since then. While I also felt put off by the apparent tone of your comments, I was not planning to make any mention of it, but when he did, I then felt the need to explain both to him that the comments may only seem rude, and to you why he (and I) might feel that they were. I hope that that helped clear the air.

[kevinhendricks]

My mistake. I looked here for the latest commits to master since I was building a custom version from source:

https://github.com/mathjax/MathJax-src/commits/master 

But did not notice I had looked here for develop (not MathJax-src but just MathJax).

https://github.com/mathjax/MathJax/commits/develop

Both of which showed a commit history of Sep 2021 as the last commit.

So I am glad to be corrected and that Mathjax is still under very active development. We will await your next release with both fixes and try to come up with some way to build and work with both MathJax 2.7 and Mathjax 3.2.X as adoption by the majority of linux distributions seems spotty. We embed MathJax for Windows and MacOS so we can control them more easily.

Please note, due to how we allow syncing back and forth from our Preview Window (QtWebEngineView) back to our Code View (xhtml editor) we need the svg output version to make mapping from one dom webpath (cfi-like) to another easier as the chtml output seems to spread lots of pieces in the form of style tags outside the container node all over the dom, whereas the added svg node allows us to happing map it back to the math node when syncing.

Re-opening it to remind myself to try again after the next release.

[dpvc]

@kevinhendricks, sounds good. Thanks for continuing to work on a v3-based implementation.

[kevinhendricks]

Okay MathJax 3.2.2 was released about 2 weeks ago. It should have all of the fixes for mml3 and things to almost pass our epub3 math test-suite (BiDI subscripts issues still remain but they existed in Mathjax 2.7.X as well). So it has reached full parity with the latest Mathjax 2.X releases for the parts we need.

I grabbed the latest release in source and was able to build a custom-mathjax with just the pieces we need (mml input, svg output, mml3 extension) out of the box. It passed all my tests like a charm.

The only question about integrating this into Sigil is to how best to support as many Linux distributions as possible.

Right now, except for a fewer obscure ones, only Arch and Manjaro ship with Mathjax 3.2.2. But if we ship our own custom-mathjax.min.js for linux as we do for macOS and Windows builds, it really should not matter as we could move everyone to our own bundled Mathjax build.

But given how Linux distributions hate bundling (which is the reason there is still shared library hell even with rolling releases), I was thinking along the lines of creating cmake build flags that indicate if you want:

a) an external MathJax 2.7.X build b) an external MathJax 3.2.X build (>= 3.2.2) c) an internal custom-mathjax.min.js build (just like we do with our single file MathJax.js file now)

macOS and Windows would default to using our internal custom-mathjax 3.2.2 build just like they do with our internal MathJax.js based on our 2.7.5 build now.

It is only Linux that would really need to use those new build flags since they would most likely want to use an unbundled external build. Although the difference between bundling our own custom mathjax into Sigil versus what we would be doing if we ever do get around to creating our own appImage (or using the current flatpak) for Sigil on Linux does not really seem to be meaningful to me at least, especially for a pure javascript library.

Thoughts? Comments? What is the best way to support Mathjax 3.2.2 in Sigil without leaving a number of Linux distributions behind?

[dougmassay]

I'll have to do some thinking to get my head around it all. :)

Keep in mind that even though Arch uses an external mathjax, they still install our custom config file (SIGIL_EBOOK_MML_SVG.js) locally.

https://github.com/Sigil-Ebook/Sigil/blob/master/src/qt5sigil.cmake#L639

Note that MATHJAX_DIR is defined when configuring Sigil with cmake. Arch currently uses -DMATHJAX_DIR=/usr/share/mathjax2

I'm assuming that MathJax 3.2.2 would still require our SIGIL_EBOOK_MML_SVG.js (or something similar to it). So if we want to support both MathJax 2 and MathJax 3.2.2+ on Linux, we'd still have to have a config for both versions included in our source bundle, no?

Right now, everything mathjax-related on Linux is predicated on MATHJAX_DIR being defined or not. So it technically shouldn't be out of the question to add logic to handle both versions. I'd just prefer that things default to the way they are now (meaning mathjax2 external). Most any distro that has mathjax3 is going to probably have mathjax2 as well. That way, we don't break anything if they don't pick up on the fact that we're supporting mathjax3 right away.

"First do no harm" and all that. ;)

[dougmassay]

If MATHJAX_DIR is not defined, we install our bundled mathjax (3.2.2+) like we do with mathjax2 now. If in addition to MATHJAX_DIR being defined (or not), a MATHJAX_VER define (which defaults to 2) would be checked to see which config file should be installed to MATHJAX_DIR.

The only thing left to do would be checking for a minimum of 3.2.2 if MATHJAX_VER==3.

[eli-schwartz]

That way, we don't break anything if they don't pick up on the fact that we're supporting mathjax3 right away.

If you change the define to -DMATHJAX3_DIR=/path/to/mathjax3 then any distros that don't notice the change will simply have it begin bundling mathjax since they don't pass the right define. This may trigger followup lint checks for file differences, and lead to the distro implementing version 3. Either way, the resulting build will work.

I guess it depends on whether you want to support both versions of Mathjax at the same time.

[dougmassay]

I edited my first post to change "(meaning mathjax2 external or bundled)" to just "(meaning mathjax2 external)". I didn't mean to imply that we'd default to MathJax2 when bundled.

[dougmassay]

I guess it depends on whether you want to support both versions of Mathjax at the same time.

I do wish to support using either external version of MathJax simultaneously, I think. That way if they don't update their build process, they'll still be using the same old external mathjax2 they have been. But there's merit to letting it fail to using the bundled mathjax3, too.

We may be overthinking it too. Making sure those who wish to use an unbundled MathJax3 have a minimum of 3.2.2 might not be worth the trouble. I'm not sure there's any distros other than Arch-based ones that are even using the MATHJAX_DIR define, to tell the truth. I think most just roll with our bundled/customized version.

[kevinhendricks]

FWIW, you can use xhtml script tags to specify what to load for Mathjax 3 so we would not need/use the extra SIGIL_EBOOK_MML_SVG.js piece for MathJax 3.

So MainWindow.cpp will have to determine the version of Mathjax available during runtime and adjust the injected script tag contents accordingly.

[kevinhendricks]

And given that need for runtime version info being needed before actual loading ofMathjax (so its global version info inits MathJax object won't help), we should probably just make everyone (except for Arch/Arch-derivatives who use that MATHJAR_DIR define) use our own bundled custom MathJax 3.2.2 javascript so that everybody sees/uses the same thing.

That frees us from worrying about all the much older Linux distributions and their Mathjax versions at all. It will just build and work out of the box for them as well.

I am going to play around with my custom mathjax to make sure we integrate it successfully into QtWebEngine for macOS/Windows first. Then we can decide the best path forward for all the Linux distributions

[dougmassay]

That might be best. I honestly don't think there's that many distros debundling our mathjax anyway. I would still like to find a way to confirm 3.2.2 or higher before allowing the MATHJAX_DIR logic to use a local mathjax3 if at all possible.

Or like Eli suggested, we change the define to MATHJAX3_DIR and document the version minimum of 3.2.2. That way, everyone uses the correct bundled Mathjax until they pick up on the new documentation. Then Arch and others can change their build to use MATHJAX3_DIR if they so wish.

[kevinhendricks]

I hacked support for our custom-mathjax.min.js into my current macOS local build. It certainly makes everything much cleaner. No need to update or change our web-paths calculations as a there is a one to one correspondence between $tags\; in\; the\; source\; and\; injectedtags\; that\; hold\; the\; created\; svg\; completely.$

This means no hooks to run cleanups after processing.

So with our MathJax 3.2.2 custom build:

• we no longer need any config file,
• the mathjax url points to our local file so no unzipping or copying of MJ.zip pieces
• no need for startup hooks and no need for an injected clean-up script anymore
• no manipulations are needed anymore to sync from Preview to CodeView and back for text.

This means much simpler code overall.

So moving to mathjax3.2.2 for everyone by default seems to be a win for us.

I am going to work on cleaning up my hack job and once working I will push it to my personal github site for you to play around with. I will take a shot at the Windows side as well for you to check and fix before we decide how to move forward with Sigil master. How does that sound?

[kevinhendricks]

Okay, I pushed what I hope will work on all 3 platforms (but only tested on macOS) following your suggestions and using Eli's switch to using MATHJAX3_DIR for any unbundled mathjax.

I have committed everything to my personal GitHub site:

https://github.com/kevinhendricks/Sigil

I have tried to handle all 3 platforms and for both qt5 and qt6 builds but I am sure I probably messed up someplace.

It works just fine with both Qt5 and Qt6 builds on macOS but remember to enable Javascript in the Sigil Preferences. I forgot that Qt6 builds now have their own settings file and it defaulted to off, so I had to manually changed that preferences and reload the epub.

The idea is that all platforms that do not define MATHJAX3_DIR at compile time will now default to using our custom-mathjax.min.js (built from MathJax 3.2.2 source - as documented in the updated MathJax readme file).

If the MATHJAX3_DIR is set, then it will preload a config and look for statup.js in the MaxJax3_DIR and use that.

I have not tested this on Linux at all but I have a Manjaro VM running under virtual box on my mac laptop and when I get some time I will try testing it.

Attached is a test epub.

[kevinhendricks]

mathml_test.epub.zip

[kevinhendricks]

Any and all fixes to make Windows or Linux work welcomed. ;)

[dougmassay]

I'll check it out on Windows and Linux. Might not be until the weekend, though. The bundled version seems straightforward enough, but I'm not familiar enough, yet, with the config differences between versions 2 and 3 to figure out how to configure an external mathjax 3 to load the modules we need when -DMATHJAX3_DIR is in play. I'm sure we'll figure something out, though.

[kevinhendricks]

I left that fixmein the cmake files, but I handled the config setting in MainWindow,cpp and PreviewWindow.cpp so hopefully no worries about config stuff.

[dougmassay]

I had a chance to try it on Linux (Arch) using Qt5 and the bundled mathjax3.2.2 and all seemed fine.

I've noticed a couple of issues with the test epub, but I'm fairly certain they were present before.

One is the optional CSS styling test which just flat-out fails.

The other is the mandatory ltr test (70, I think). The third column definitely looks to me like a mirror image of the the second column (in both the rows). But the second row (Arabic) does not look like the second row in the image they present to verify results. I could be wrong, but I'm convinced the verification image is actually wrong!

I'll do some more tests soon.

Good work!

[kevinhendricks]

That is good news. Once we make sure it is working on Windows, I can push it to Sigil master and we can work on any issues with external mathjax 3.2.2+.

[kevinhendricks]

Okay, I used my Manjaro virtual box VM on my macOS box, and was able to use the new MATHJAX3_DIR define for Linux and was able to view the mathml_test epub properly with an unbundled external MathJax 3.2.2 version that was installed in /usr/share/mathjax.

/usr/lib/qt6/bin/qt-cmake -G "Unix Makefiles" -DUSE_QT6=1 -DMATHJAX3_DIR=/usr/share/mathjax -DCMAKE_BUILD_TYPE=Release ../Sigil

So Linux works with both bundled and unbundled MathJax 3.2.2+ as long as during the build MATHJAX3_DIR is properly pointing to whare Mathjax 3.2.2+ is located on that system.

So if the bundled custom-mathjax builds and works on Windows, we are good to go.

[dougmassay]

I could have sworn I saw a difference between bundled vs external when rendering the right-to-left test, but I could be wrong. I need to take some screen shots of both to know for sure. But other than that: yes. Everything looks good! I'll test on Windows before I spend time verifying everything is exactly the same between linux bundled vs linux debundled.

On Fri, Jun 24, 2022, 10:30 AM Kevin Hendricks @.***> wrote:

Okay, I used my Manjaro virtual box VM on my macOS box, and was able to use the new MATHJAX3_DIR define for Linux and was able to view the mathml_test epub properly with an unbundled external MathJax 3.2.2 version that was installed in /usr/share/mathjax.

/usr/lib/qt6/bin/qt-cmake -G "Unix Makefiles" -DUSE_QT6=1 -DMATHJAX3_DIR=/usr/share/mathjax -DCMAKE_BUILD_TYPE=Release ../Sigil

So Linux works with both bundled and unbundled MathJax 3.2.2+ as long as during the build MATHJAX3_DIR is properly pointing to whare Mathjax 3.2.2+ is located on that system.

So if the bundled custom-mathjax builds and works on Windows, we are good to go.

— Reply to this email directly, view it on GitHub https://github.com/Sigil-Ebook/Sigil/issues/453#issuecomment-1165633707, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACG3CXTHH4LBJU7TLU4ESW3VQXBCFANCNFSM4ITXFJIQ . You are receiving this because you commented.Message ID: @.***>

[kevinhendricks]

Neither version of Mathjax supports rtl, at least that is my understanding from their response to issue 2836 I filed. See:

In terms of the bidirectional issues, MathJax doesn't currently support 
bi-directional output (in v2 or v3). It is something we do hope to 
address in the future.

[kevinhendricks]

And like you, do not think the MathML test image with rtl subscripts it even showing the correct answer (mathematically correctly).