mathoid-treeprocessor error in svg

[paulvickers]

Been trying to get a compilation path that will render a document with math in it across html, pdf, epub, and mobi. Managed to get mathoid-treeprocessor installed after some tinkering.

Using the sample .adoc file:

= _Precompiled_ Math!

[stem#equation1]
++++
"d"(bbp,bbq) = sqrt((p_1 - q_1)^2 + (p_2 - q_2)^2)
++++

and the command:

asciidoctor  -r asciidoctor-pdf -r ./lib/mathoid-treeprocessor  -b pdf test.adoc --trace

I get the following output:

asciidoctor: WARNING: could not embed image: /Volumes/SSD Disk/git Repos/test/equation1.svg; Duplicate attribute "xlink:href"
Line: 23
Position: 7155
Last 80 unconsumed characters:

The generated svg file is not an equation but text containing an error message "This page contains the following errors..."

Running phantomjs main.js with the query string http://localhost:16000/?q=x^2&type=asciimath generates a web page in the browser saying:

This page contains the following errors:

error on line 1 at column 1252: Attribute xlink:href redefined
Below is a rendering of the page up to the first error.

The page source is:

<svg xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="-22 -915.1157009464328 932.0389244992066 953.2641499988939" style="width: 2.167ex; height: 2.167ex; vertical-align: -0.135ex; margin: 1px 0px; position: static;" xmlns="http://www.w3.org/2000/svg"><defs id="MathJax_SVG_glyphs"><path id="STIXWEBMAINI-78" stroke-width="10" d="M243 355l12 -57c70 107 107 143 151 143c24 0 41 -15 41 -37c0 -21 -14 -36 -34 -36c-19 0 -28 17 -52 17c-18 0 -54 -44 -98 -121c0 -7 2 -21 8 -45l32 -134c7 -28 16 -41 30 -41c13 0 24 10 47 40c9 12 13 18 21 28l15 -9c-58 -90 -84 -114 -122 -114 c-32 0 -47 18 -59 68l-29 119l-88 -119c-44 -59 -64 -68 -95 -68s-50 16 -50 42c0 20 14 36 34 36c9 0 19 -4 32 -11c10 -6 20 -9 26 -9c11 0 30 19 51 49l82 116l-28 124c-14 60 -21 68 -46 68c-8 0 -20 -2 -39 -7l-18 -5l-3 16l11 4c61 22 94 29 117 29 c25 0 37 -18 51 -86Z"></path><path id="STIXWEBMAIN-32" stroke-width="10" d="M474 137l-54 -137h-391v12l178 189c94 99 130 175 130 260c0 91 -54 141 -139 141c-72 0 -107 -32 -147 -130l-21 5c21 117 85 199 208 199c113 0 185 -77 185 -176c0 -79 -39 -154 -128 -248l-165 -176h234c42 0 63 11 96 67Z"></path></defs><g stroke="black" fill="black" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#STIXWEBMAINI-78" xlink:href="#STIXWEBMAINI-78"></use><use transform="scale(0.7071067811865476)" xlink:href="#STIXWEBMAIN-32" x="640" y="583" xlink:href="#STIXWEBMAIN-32"></use></g></svg>

Any ideas?

[paulvickers]

UPDATE:

If from the command line I run:

curl http://localhost:16000/?q=2^n then I get:

<svg xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 -740.4482358595916 1032.7996026178612 767.5966849120528" style="width: 2.438ex; height: 1.761ex; vertical-align: -0.135ex; margin: 1px 0px; position: static;" xmlns="http://www.w3.org/2000/svg"><defs id="MathJax_SVG_glyphs"><path id="MJMAIN-32" stroke-width="10" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path><path id="MJMATHI-6E" stroke-width="10" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"></path></defs><g stroke="black" fill="black" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xlink:href="#MJMAIN-32" xlink:href="#MJMAIN-32"></use><use transform="scale(0.7071067811865476)" xlink:href="#MJMATHI-6E" x="714" y="569" xlink:href="#MJMATHI-6E"></use></g></svg>

If I copy that into a .html file and load it into Chrome I get a properly formatted math expression on the page. But putting the same query string (http://localhost:16000/?q=2^n)into the browser address bar gives me the 'this page contains the following errors...' error.

[mojavelinux]

As it turns out, Mathoid has been completely redesigned since the mathoid-treeprocessor extension was first written. I had to do a major overhaul of the code to get it operational again. The good news is a) it works and b) it's based on a stable release of Mathoid, so we shouldn't have random breakage in the future. I think this is now a very viable extension, assuming you are okay to run it on Linux and are able to install the prerequisites.

You do need to be using Node.js 4.4.7. I recommend using nvm to manage your Node.js version. Once the nvm script is setup, you can simply run:

$ nvm install 4

or, if it's already installed:

$ nvm use 4

There are new instructions for how to setup the mathoid processor in the mathoid.rb script. See https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/mathoid-treeprocessor/mathoid.rb. It should be much simpler now. In a nutshell, you just type:

$ npm install mathoid

Mathoid no longer uses PhantomJS. Instead, it uses MathJax via an adapter that allows it to work server-side in Node.

I'm interested to hear if you're able to get it working.

Once we verify the extension is functional, there are still a lot of things we could do to improve it, such as cache the SVG so it doesn't have to be generated on each run.

[mojavelinux]

Now that mathoid has stablized and can run without PhantomJS, it's probably time to start thinking about creating a repository named asciidoctor-mathoid to complement asciidoctor-mathematical. There's no question this is going to be the more popular extension given that it doesn't have the limitations of mathematical. (It will be interesting to see if we can get it running on Windows, but I'm at least confident we can get it running under JRuby).

[paulvickers]

assuming you are okay to run it on Linux and are able to install the prerequisites.

I'm running on Mac OS but I'll see what happens.

[mojavelinux]

You'll just need to sort out how to get librsvg2. Perhaps via macports or homebrew.

[mojavelinux]

brew install librsvg

More information at https://www.npmjs.com/package/librsvg

Le 16 août 2016 6:15 AM, "Dan Allen" dan.j.allen@gmail.com a écrit :

You'll just need to sort out how to get librsvg2. Perhaps via macports or homebrew.

[paulvickers]

Yes, I needed that for the old version of mathoid. Homebrew has a version of it, so I installed that yesterday. As we speak I'm pulling down the latest version of extensions-lab. I've installed nvm and mathoid using npm. About to try the new version of mathoid-treeprocessor.

[mojavelinux]

Good. You'll definitely want to move to the new mathoid processor because the old one isn't even possible to setup anymore since the mathoid git repo it was using is now gone.

[paulvickers]

Hmmm....

• cd ~
• Install nvm: check.
• nvm install 4: check
• npm install mathoid: check

Quick check to see mathoid installed:

npm list --depth=0 2>/dev/null
/Users/paulvickers
└── mathoid@0.6.3

Mathoid installed (it's at ~/.npm/mathoid)

• npm run mathoid:

Error:

npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/node" "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/npm" "run" "mathoid"
npm ERR! node v4.4.7
npm ERR! npm  v2.15.8
npm ERR! path /Users/paulvickers/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open '/Users/paulvickers/package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/paulvickers/npm-debug.log

After copying ~/.npm/mathoid/0.6.3/package/package.json to ~, running again gives:

npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/node" "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/npm" "run" "mathoid"
npm ERR! node v4.4.7
npm ERR! npm  v2.15.8

npm ERR! missing script: mathoid
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/paulvickers/npm-debug.log

• Copy mathoid-treeprocessor folder from extensions-lab to lib folder in test project.
• cd ./lib/mathoid-treeprocessor
• sh setup.sh:

gives

npm ERR! code 1
npm WARN install Refusing to install mathoid as a dependency of itself

Hmmm.

[paulvickers]

I just can't get this working. I have uninstalled npm/nvm and reinstalled them. nvm is at ~/.nvm. I reinstalled node 4 (nvm install 4).

I went to ~ and installed mathoid which creates a folder ~/node_modules which has mathoid in it.

I copied this node_modules folder to lib/mathoid_treeprocessor in my test project. But still nothing works. Running the following:

asciidoctor -r ./lib/mathoid-treeprocessor test.adoc --trace

gives the following result:

/usr/local/Cellar/ruby/2.3.1/lib/ruby/2.3.0/open3.rb:199:in `spawn': asciidoctor: FAILED: /Volumes/SSD Disk/git Repos/test/test.adoc: Failed to load AsciiDoc document - No such file or directory - /Volumes/SSD (Errno::ENOENT)
    from /usr/local/Cellar/ruby/2.3.1/lib/ruby/2.3.0/open3.rb:199:in `popen_run'
    from /usr/local/Cellar/ruby/2.3.1/lib/ruby/2.3.0/open3.rb:147:in `popen2'
    from /Volumes/SSD Disk/git Repos/test/lib/mathoid-treeprocessor/mathoid.rb:136:in `start'
    from /Volumes/SSD Disk/git Repos/test/lib/mathoid-treeprocessor/extension.rb:11:in `process'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/document.rb:478:in `[]'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/document.rb:478:in `block in parse'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/document.rb:477:in `each'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/document.rb:477:in `parse'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor.rb:1344:in `load'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor.rb:1462:in `convert'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/cli/invoker.rb:94:in `block in invoke!'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/cli/invoker.rb:86:in `each'
    from /usr/local/gems/asciidoctor-1.5.4/lib/asciidoctor/cli/invoker.rb:86:in `invoke!'
    from /usr/local/gems/asciidoctor-1.5.4/bin/asciidoctor:14:in `<top (required)>'
    from /usr/local/bin/asciidoctor:22:in `load'
    from /usr/local/bin/asciidoctor:22:in `<main>'

The file test.adoc contains:

= _Precompiled_ Math!

[stem#equation1]
++++
"d"(bbp,bbq) = sqrt((p_1 - q_1)^2 + (p_2 - q_2)^2)
++++

and this compiles okay with asciidoctor test.adoc.

Any clues???? Would installing mathoid globally help (npm install -g mathoid)?

[paulvickers]

@mojavelinux Update: Realised that the first line of the error shows that something something (open3.rb?) is sensitive to spaces in path names. Grrrrrrrrr.

I moved the project to a disk and folder with no spaces in their names and now get:

Failed to start Mathoid server. Check that a process is not already running.
Mathoid server is not running.

I couldn't get the mathoid server to start with npm run mathoid but using the server.js file revealed that mathoid is looking for something called config.yaml. It doesn't come with that file but there are two others config.prod.yaml and config.dev.yaml. I copied the dev into config.yaml and was then able to run the server using server.js. Huzzah!

Next problem: the mathoid.rb extension is choosing to look for mathoid-config.yaml in the same folder from which asciidoctor was called. So, copying that config.yaml from ./lib/mathoid-treeprocessor/node_modules/mathoid to ./ seems to have done the trick.

Using the source:

= _Precompiled_ Math!

Here's a stem block:

[stem#equation1]
++++
"d"(bbp,bbq) = sqrt((p_1 - q_1)^2 + (p_2 - q_2)^2)
++++

Here's some inline maths: stem:[2^2=4]

Gives the following output:

NOTE:

1. The stem block comes out huge.
2. The inline stem is not being rendered.
3. It works (i.e, renders as above) with the asciidoctor-pdf backend
4. It also works with the asciidoctor-epub3 backend (as long as the spine and imagesdir are correctly formatted and set), though again, it's giving jumbo sized equations and is not rendering the inline stem.

[paulvickers]

ps, how would one incorporate mathoid-treeprocessor into a gradle.build file? I tried but am getting undefined method 'pid' for #<Thread:0xea40c9f.

[paulvickers]

pps, it seems one has to have the node_modules folder containing mathoid in the mathoid-treeprocessor folder. As this comes to about 167 MB, I was wondering if it's possible to instead link to a common location for this node? For example, on my machine mathoid is installed in ~/node_modules. Can we force it to see that instead? Sorry, but I'm new to node/npm.

UPDATE: nevermind, I saw in the mathoid.rb file that it uses either the MATHOID_HOME environment variable or the local node_modules folder, so I set up the environment variable in .bash_profile and it works.

However, I do note that it seems a bit random whether it will work or not. As often as not invoking mathoid results in a

Failed to start Mathoid server. Check that a process is not already running.
Mathoid server is not running.

error. Mathoid doesn't show up as a running process either with ps or in the Mac Activity Monitor. Is it possible to ensure it shuts down after each run?

[mojavelinux]

First, going to the top. You need to run nvm install in the mathoid-treeprocessor folder for right now. I'll clarify that. That's something we can clean up once we move to making this a formal extension. Likely we'll run npm install when the gem is installed, so that the node_modules folder ends up inside the installed gem directory. The current situation is just a requirement of us still being in the lab.

[mojavelinux]

Instructions updated.

[mojavelinux]

While trying to get it working, you may end up with the situation where the mathoid server did not properly shut down. The mathoid wrapper doesn't yet deal with this situation (in an effort just to get it working). If you get the message that it is already running, use the following:

ps a | grep server.js

Find the running server, then kill it

kill -s SIGTERM 17709

[mojavelinux]

Realised that the first line of the error shows that something something (open3.rb?) is sensitive to spaces in path names.

I did a very quick system call. I just need to quote it properly so it handles spaces.

[mojavelinux]

Spaces in filenames are now supported.

[mojavelinux]

Would installing mathoid globally help (npm install -g mathoid)?

No. At the moment the extension assumes the npm module is installed locally. I'll have to do more investigation and testing to see if we can detect it globally.

[mojavelinux]

I couldn't get the mathoid server to start with npm run mathoid but using the server.js file revealed that mathoid is looking for something called config.yaml.

Ooops! I forgot to commit the file. Doh! On the way...

[mojavelinux]

Added. Now you should be able to run:

npm run mathoid

[mojavelinux]

The stem block comes out huge.

The equation expands to fit the container by default. I now pass the width attribute from the stem block to the image block, so you can now use that attribute to customize the width.

The inline stem is not being rendered.

Yes, that is a limitation of this extension. We don't yet handle inline stem nodes.

[mojavelinux]

ps, how would one incorporate mathoid-treeprocessor into a gradle.build file?

I haven't gotten that far yet. Likely, we'll need to invoke the system command in a way that is compatible with JRuby. It's certainly possible, I just haven't looked into it.

[paulvickers]

Would installing mathoid globally help (npm install -g mathoid)?

No. At the moment the extension assumes the npm module is installed locally. I'll have to do more investigation and testing to see if we can detect it globally.

I got round this by setting the MATHOID_HOME environment so that mathoid.rb will now execute it wherever I have mathoid installed.

[paulvickers]

Added. Now you should be able to run:

npm run mathoid

Not for me:

paulvickers@PaulVickerssPro:~/test/lib/mathoid-treeprocessor $ npm run mathoid
npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/node" "/Users/paulvickers/.nvm/versions/node/v4.4.7/bin/npm" "run" "mathoid"
npm ERR! node v4.4.7
npm ERR! npm  v2.15.8
npm ERR! path /Users/paulvickers/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open '/Users/paulvickers/package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/paulvickers/test/lib/mathoid-treeprocessor/npm-debug.log

Also, when using it as a required library from Asciidoctor it's still looking for mathoid-config.yaml in the folder from where asciidoctor was invoked rather than in the mathoid-treeprocessor folder.

[mojavelinux]

I'll need to resolve the full path to the source file so it works when MATHOID_HOME is different.

Btw, I figured out how to support the inline stem macro properly, so we have a path to full STEM support.

[paulvickers]

I'll need to resolve the full path to the source file so it works when MATHOID_HOME is different.

No problem.

Btw, I figured out how to support the inline stem macro properly, so we have a path to full STEM support.

Excellent news!

[mojavelinux]

The mathoid extension should now work when MATHOID_HOME is set.

[lehoff]

I am not at all knowledgeable about how npm works, but I have managed to get a node_modules/mathoid directory in asciidoctor-extensions-lab/lib/mathoid-treeprocessor (npm install mathoid -g --prefix ./ installed it in lib/node_moduls, so I moved it up not knowing if that is okay). So far - so good.

But when I run npm run mathoid in that directory I get:

npm ERR! enoent ENOENT: no such file or directory, open '/Users/th/tools/asciidoctor-extensions-lab/lib/mathoid-treeprocessor/package.json'

And running this also fails:

$ asciidoctor -r ./lib/mathoid-treeprocessor.rb lib/mathoid-treeprocessor/sample.adoc --trace
/usr/local/var/rbenv/versions/2.2.2/lib/ruby/2.2.0/open3.rb:193:in `spawn': asciidoctor: FAILED: /Users/th/tools/asciidoctor-extensions-lab/lib/mathoid-treeprocessor/sample.adoc: Failed to load AsciiDoc document - No such file or directory - ./lib/mathoid-treeprocessor//server.js (Errno::ENOENT)
    from /usr/local/var/rbenv/versions/2.2.2/lib/ruby/2.2.0/open3.rb:193:in `popen_run'
...

Could you describe how you installed mathoid into the mathoid-treeprocessor directory and how you run it? Especially how you set MATHOID_HOME.

[mojavelinux]

I don't have time to focus on the mathoid extension, so I'm going to remove it from this repository.