patch for use with jsMath

[GoogleCodeExporter]

I need to write wikis with mathematical formulae. jsMath is a very powerful 
javascript project to accomplish this task. I have slightly modified vimwiki to 
make it compatible with jsMath. In particular, I introduced two new syntaxes:

$ blabla $    is used for inline formulae and
§ blabla §    is used for non-inline, more complex formulae.

These are recognized as code block within vim, and correctly converted to html 
using:

<SPAN CLASS="math"> bla bla </SPAN>    for inline formulae and
<DIV CLASS="math"> bla bla </DIV>      for non-inline ones.

The latter can be readily interpreted by jsMath to generate the Tex or PNG 
glyphs.

I send as an attachment a zip file that contains diff patches to be applied 
against the standard vimwiki-1-1-1 tree.

If you want to try this out, you need to install jsMath. This is easiest 
accomplished by the following procedure:
1. download the jsMath tree into the folder used for your HTML vimwiki exports:
 $ ls html/
 index.html
 wiki_1.html
 ...
 jsMath/
You might need also the PNG fonts. Refer to the jsMath website, and download 
them and create the jsMath/fonts folder.

2. edit jsMath/easy/load.js so that the root line (34) looks like this:
 root: "jsMath/",
and the autoload line (46) is:
  autoload: 0,
If you want, you can change further settings below this point, such as 
loadFiles for extensions and loadFonts for PNG fonts.

3. Now, try to write a wiki page using the $ bla $ and § bla § syntaxes for 
math formulae. Then export it to HTML, and you should see the formulae!

Original issue reported on code.google.com by iosonofa...@gmail.com on 1 May 2011 at 12:11

Attachments:

• vimwiki_jsmath.zip

[GoogleCodeExporter]

Thank you for making a patch so quickly! Most people just keep piling up 
feature requests/wild wishes in the issue tracker, and relatively few actually 
report problems. Those that submit specific changes are very rare indeed!

Have you seen http://code.google.com/p/vimwiki/issues/detail?id=147 in the 
issue tracker? When I started to use vimwiki, I first thought that some quick, 
minimal change like yours would perhaps make Vimwiki usable for those needing 
math environments. But looking at the code in various places I quickly 
abandoned this idea, seeing that the version 1.1.1 seems too brittle to absorb 
significant feature additions, and I felt I would probably spend a lot of time 
fighting all the unexpected "interactions" of various syntax elements. But 
redesigning everything from the ground up takes a lot of time, I cannot really 
claim that anybody is joining this effort, so I haven't myself done much in 
terms of coding recently... I think that some specification should be made 
first, so that people do not code away simultaneously in different direction.

OK, so when I comes to your patch: I probably won't be able to seriously try it 
in foreseeable future, although I have quite a few wiki files with marked math. 
But in case you find that you are not perfectly happy with it and decide to 
work on it more:

1.  There seem to be a strong tendency to move away from jsmath and other 
"older" systems towards MathJAX. I suspect MathJAX has currently some 
disadvantages compared to jsmath, but definitely many more advantages. If you 
could try to see that the patched wiki can be made to work with MathJAX (it can 
be configured to work with vimwiki generated files), it would be more 
future-proof. I have a feeling there may be a problem with MathJAX 1.0 if no 
math markers are passed down to HTML (that's really a problem with MathJAX, but 
one would have to deal with it).

2.  Using the section sign may be a problem for some people: I did not check, 
but I implicitly assume that for any non-ASCII character, there are going to be 
some legacy encodings (and very important ones for those who use them, e.g. 
those working in "localized" OSes) where such a character does not exist. Once 
I searched Google Code site for various character combinations, and it turned 
out that somewhat surprizingly, the LaTeX combinations \[ and \] are very, very 
rare in programming code, so I thought that those would be a pretty good pair 
of wiki markers for a block/display math. But there may be other good choices, 
although probably not as short ones.

In any case, having an asymmetric pair of markers is going to be pretty much 
necessary if you ever try to extend your "non-inline" math to be an actual 
non-inline (block) wiki environment. It is better to change earlier than later.

3.  If you think that your patch may be useable by others, it would be very 
good if you considered writing some short piece of documentation for potential 
users: how to configure their system to get it going, and perhaps also some 
honest appraisal of what people can expect: I am sure that it is not going to 
be difficult to run into some limitations. If you had no better idea, the page 
could be placed on this site's Wiki. Google wiki seems to be a strong candidate 
for the worst looking wiki on the internet, but it would be easy to point 
people to it. As I already indicated, there is not much chance someone else 
would spring up to do this (fairly important) work.

I would very much like to see a growing group of people interested in putting a 
solid math support into Vimwiki. People with both time and skill are certainly 
not easy to come by!

Original comment by tpospi...@gmail.com on 1 May 2011 at 2:46

• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

Hi, and thanks for the quick feedback.

0. I forgot to mention that you have to change your HTML header to include

    <SCRIPT SRC="jsMath/easy/load.js"></SCRIPT>

in the <HEAD> section. Otherwise, your wiki can't know the location of your 
jsmath.

1. I've switched to MathJAX 1.1 as this seems to be the more recent engine 
used. I only had to make a quick change in the HTML header and in 
autoload/vimwiki_html.vim. Now it works perfectly.

2. I chose the § (section) sign because it was short and unused (I have an 
italian keyboard). Since \[ \] are free as well, I can switch to those (the 
regexp becoming slightly more involved though).

No intention to really make it a block yet. In fact, it can be used as an 
inline syntax in the text file, and only redered as a block in the HTML. In TeX 
it is the same: you can use \[ \] environments within a line... they be only be 
*rendered* as separated.

3. Documentation is a Good Thing(TM). I'll finish my switch to MathJAX and 
write some docs on the google wiki.

(4.) I've checked out issue 147. In some sense, as far as I can tell from the 
shown images, its goal is orthogonal to mine; 147 adds math (unicode chars) to 
the TEXT file, I try to add mathematics to the HTML file. The two 
implementations could perhaps be merged to add a TXT/HTML math support for 
vimwiki.

I'll let you know when the new version is ready.

Original comment by giulia.f...@googlemail.com on 2 May 2011 at 5:23

[GoogleCodeExporter]

Ok, I am not authorized to create a new wiki page, am I? If you give me the 
permissions, I can do a wiki page for this.

Anyway, here is the new patch using MathJax. Now, the syntaxes are:

$ bla $         for inline formulae
\[ bla bla \]   for block formulae (displaymath)

The easiest way to use this patch is to put the following line in your HTML 
header, at the end of the <HEAD> section:

<script type="text/javascript" 
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorM
ML"></script>

Then, import the patch and start enjoying $$ and \[\] right away!

Original comment by giulia.f...@googlemail.com on 2 May 2011 at 6:07

Attachments:

• vimwiki_mathjax.zip

[GoogleCodeExporter]

Ad 1. That's great to hear that no modifications are needed to let MathJAX do 
the processing.

Ad 2. If you do not intend to make your "display" math a block, then the choice 
of the marker § is perhaps not too bad --- I suspect that people whose 
encodings are too different from European/US ones are scared off by systems 
that use CamelCase by default anyway. Changing to \[ and \] would almost 
certainly require to adjust the way Vimwiki check for [links] and [[links]], 
and may not be worth it if you do not mean it to become a real "display math" 
for matrices and aligned equation arrays. Besides, a really good support of 
display math would require figuring out how to indicate (via some parameter) 
that one wants an equation, align etc environments, and that is a lot more work 
(and \[ and \] then might be seem slightly misleading for LaTeX-knowledgeable). 
You can think it over, it is a minor point, compared to the following.

This issue is much more important, and I have not mention it last time: real 
block environments should be required to have their start and end markers alone 
on a line. That is the only way to make the whole system sane, lean and 
predictable. In the case of this "short display" math environment, it would 
perhaps be convenient to people to be able to enter it on one line as
  \[ f(x)=e^{-t} \]
instead of three lines. This alone would greatly reduce the work needed to make 
sure the equation is not misunderstood as a link (or vice versa), and many 
other potential problems of clashing markup. So whichever marker you decide to 
use, please adjust g:vimwiki_rxEqBl with '^\s*' and '$' (or fairly useless 
variation '\s*$') to avoid the same blunder countless wikis have done (which 
invariably leads to bugs and/or needless complexity of code, while providing no 
extra capability), e.g. http://code.google.com/p/vimwiki/issues/detail?id=205

[A much worse example is available, sadly: totally unrestricted "wild comments" 
are allowed that can start anywhere and consequently change the meaning of 
anything after, and this is a real disaster, effectively preventing any fast 
partial parsing (or anything fast, actually). While the value to the user is 
somewhere between zero and negligible: the wiki is a non-programmable document, 
there is hardly any need for "comments", let alone block comments.]

Once this "one-line" restriction is in, it is would be very easy (well, if it 
were not for the aforementioned comments "feature" and similar problems) to 
later mechanically replace the marker with some other (if the syntax is changed 
in order to allow greater generality).

Ad 3. Maxim, the project creator, will surely allow you to create wiki pages, 
or commit code if you want to (but code has to go in via the Mercurial DVCS, 
there is no way to make even small code additions "directly"). He will probably 
have to figure out which of your e-mail addresses to use, though...

For the docs: typing signs if the keyboard doesn't provide them:
 § (section sign): <CTRL-K> SE
 $ (dollar sign): <CTRL-K> DO
(only works if the encoding supports the symbol, though).

Ad 4. I do not blame you for not reading the Issue 147 at all, the appearance 
of these pages really does not invite people to read the content. But if you 
thought this was the case of "one picture tells a thousand words", those are 
completely different words than in the discussion. The discussion is on making 
a robust syntax for wiki (and an HTML generator that would not get out of sync 
with the syntax highlighting), it has nothing to do with using some Unicode 
characters to represent
command sequences on screen.

Original comment by tpospi...@gmail.com on 3 May 2011 at 6:10

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Other

[GoogleCodeExporter]

Issue 203 has been merged into this issue.

Original comment by tpospi...@gmail.com on 3 May 2011 at 6:22

[GoogleCodeExporter]

2. Wild comments are _very_ slow and cannot be allowed. I agree on not using \[ 
\] in the middle of a text line; it is possible in TeX but not very useful, 
since the maths are displayed on a different line anyway. It is confusing 
indeed.

I think that \[ \] is a different beast both from inline syntaxes (e.g. *bla*) 
and from block comments ( {{{ bla }}} ). The reason is that it is frequently 
used in two cases:

1. for single lines, such as:
\[ formula \]

2. for multiple-line formulae that will be displayed on a single line, such as:
\[
equation toooooooooooooooooooooooooooooooooooooooooooooooooo
long to be written in one line
\]

Consequently, in my view, we should use recognize both cases. In the first one, 
an "inline-like" recognition is called. In the second case, a "block 
recognition" is used, which requires \[ to be alone on the first line, and \] 
alone on the last one.

The following regexps should be conceptually right for dealing with both cases:

1. let g:vimwiki_rxEqBl = '^\s*\\\[[^\$`]\+\\\]\s*$'

Lines like this are recognized: <0+ spaces>\[<formula>\]<0+ spaces>

2. let g:vimwiki_rxEqBloStart = '\['
   let g:vimwiki_rxEqBloEnd = '\]'

which means that the lines in-between are a math block.

3. First of all, I would add some documentation. Then, I would try to commit 
via hg, it should not be a big problem. My email address is iosonofabio@..., I 
mistakenly used the other one from my GF's PC.

4. Ok, Issue 147 deals with a more fundamental and difficult task, I'll let it 
be for now (trying to start contributing with small goals).

I'll let you know when I have a patch for implementing these ideas.

Original comment by iosonofa...@gmail.com on 3 May 2011 at 7:17

[GoogleCodeExporter]

Hi, iosonofabio! (sorry, can't find your name)

Thanks for the patch!
I have granted you committer role so commit, push and make vimwiki better.

Original comment by habamax on 17 May 2011 at 6:12

[GoogleCodeExporter]

I myself don't like wild comments.
They greatly slow down html generation and I don't really see any good points 
to use it in wiki.

We should think of removing it.

Original comment by habamax on 17 May 2011 at 6:20

[GoogleCodeExporter]

And yes, the comments as they are in 1.1.1 must be eliminated as soon as 
possible.

One could reuse the % character for comments. For instance, 

%something

is for directives ("placeholders"), and

%%some comment

could be a comment (the line is to be completely ignored by vimwiki);
% would have this special meaning only as the first character on a line, 
otherwise it is treated as a percent character.

This way there is some way to put "comments" in the file for people who really 
need it, but there is no need to really introduce a new markup character for 
feature whose value is very small.

Maybe % should be used for parameters to block environment too (block code in 
particular): currently
{{{something
...
}}}
just discards "something" if it is not one of the recognized languages, which 
is going to surprize quite a lot of people; if it was required that block code 
must start with
  {{{
or with
   {{{%some language
otherwise the line starting with 
 {{{something 
is not going to trigger any block environment, such "accidents" would be 
eliminated.

A simpler alternative is to change the highlighting of the whole line starting 
with {{{ to make it clear that "something" is not considered as code text, but 
a markup.

Original comment by tpospi...@gmail.com on 18 May 2011 at 3:26

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I have added a wiki page on combining vimwiki with MathJax. The page is now 
quite complex, but it would become much simpler if we integrated the syntax 
changes in vinwiki. Basically, in that case, one should only change the HTML 
header. If you wish to include this functionality, I can prepare a mercurial 
patch.

MathJax supports a lot of mathematical environments, such as align, gather, 
multiline, etc. It should not be too hard to include those in our syntax as 
well. We could proceed by steps:
1. include real block math, opened by a line with \[, then block lines, then 
closing line with \]
2. allow for a parameters on the opening/ending lines. For instace, we could 
choose to implement
\[align
formula & 1 \\
formula & 2
\]align

That way, we need not recognize 10 different mathematical environments!

Also, I would choose the simpler variant for highlighting: the whole line with 
{{{, }}}, \[, or \] is not shown, including parameters. This lets a clear line 
before and after blocks, which is relaxing for the eye.

Original comment by iosonofa...@gmail.com on 18 May 2011 at 9:54

[GoogleCodeExporter]

You have access to the repo -- just push your changes if you think they're 
ready.

Original comment by habamax on 18 May 2011 at 10:13

[GoogleCodeExporter]

I would suggest two changes:

First, there is no need to have "align" (or whatever the parameter is) in the 
closing marker: the generators have only one way to make everything correctly 
nested (if it is possible at all). Vim syntax will do that too for regions 
automatically if they are declared nestable and without the "keepend" argument. 
Not only is it shorter to type, the generator also does not have to deal with
\[align
\[gather
\]align
\]gather
and similar.

Second, what I suggested above for block code would look even more natural to 
TeX-aware people:
\[%align
formula & 1 \\
formula & 2
\]
is somewhat closer to a valid LaTeX (in that it makes it clear the string 
"align" is not a part of the environment's content).

In any case, these "parameters" for block environments should have similar 
syntax across all the block environments, I thought that % would be a good 
choice since it already is used for start-of-a-directive (placeholder). It is a 
very lucky coincidence that it would look so natural in these math environments.

[This is totally unconnected to this issue, but the heaviest use of parameters 
occurs in tables: this will have to be addressed sooner or later, and having a 
special character to introduce a formatting parameter is going to make things 
quite a bit clearer: or else it would end up like in those wikis that use dozen 
or more different characters for various alignment or other attributes, all of 
which have a special function when near the start of a cell, and any table that 
makes a hefty use of these will look like gibberish to anyone but a few 
experts.]

I do not quite understand the remark "the whole line ... is not shown". The 
current situation is already unsatisfactory with some inline markers being 
invisible http://code.google.com/p/vimwiki/issues/detail?id=208#c0 (Problem 1), 
it would be very unfortunate if even more syntax groups got the same treatment. 
If you were thinking "there are only two block environments (of this type), 
they can be easily distinguished by their highlight group, no need to see the 
markers at all", that does not really work, does it? What people see depends on 
their color scheme, you would have to completely ignore that and hardcode the 
colors (same link, Problem 3) to make sure "everyone" can see the difference 
(where "everyone" is those with sufficiently normal color vision, not really 
everyone). For more than two environments, it will get even harder, and 
"everyone" group is going to shrink even more...

I am not talking about concealing: concealing is quite good, for inline 
environments. For block environments, it would not be a good idea, except 
perhaps for concealing the parameters. Unfortunately, concealing only the 
parameters cannot be safely done with the current vim (without a risk of nasty 
side-effects). A region's "matchgroup" cannot be subclassed, and the region's 
"contained" groups cannot be anchored to the start of a region. (Using matches 
instead of regions is not an option, unless you are sure you will never need to 
nest environments).

All the syntax markers should be shown with the same (and visible) highlight 
group, and the inline ones can be concealed (since the useful ones are so few 
(three) and the syntax highlighting can very likely distinguish them).

Back to math markers: another interesting choice of markers for block math 
would be
{{$
...
}}$
since it would sort of suggest a family of block markers. I already hinted that 
tables are pretty much going to need block markers too if they are ever going 
to grow up to tables whose cells can hold non-inline content. A variation on 
Wikimedia's markers would be
{{|
...
}}|
and it is easy later on to define more block environments if needed, without 
the syntax looking completely random (which seems to be the case of most wikis, 
sadly). 

But all this is just "the bigger picture". The most likely users of the block 
math are no doubt going to be quite fine with \[ and \].

Original comment by tpospi...@gmail.com on 18 May 2011 at 2:58

[GoogleCodeExporter]

I like idea of {{$ ... }}$ markers.

Original comment by habamax on 19 May 2011 at 7:01

[GoogleCodeExporter]

Dear all,

finally, I have found the time to work a bit on vimwiki again. I have pushed 
online a new mercurial branch with the changes needed for using mathjax with 
vimwiki. If you agree, you can merge it into the default branch. The syntax of 
mathematical formulae is the following:

1. inline math: $bla bla$
2. block math:
{{$
bla bla
}}$

Please note also the following notices:

1. no complex environment such as align, gather, etc. is supported yet (no 
placeholder feature). I can try to do it in, but it is not so urgent.

2. One needs to change the HTML template for mathjax to work. The simplest 
solution is to add the following line in the header:

<script type="text/javascript" 
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorM
ML"></script>

I will update the wiki accordingly.

Cheers,
Fabio

Original comment by iosonofa...@gmail.com on 2 Jul 2011 at 9:02

[GoogleCodeExporter]

Dear all,

finally, it is possible to use any TeX environment supported by MathJax in 
vimwiki. Please read the wiki for more information and, if you find the code 
useful, merge it into the default branch.

Original comment by iosonofa...@gmail.com on 3 Jul 2011 at 3:45

[GoogleCodeExporter]

Original comment by habamax on 22 Aug 2011 at 11:29

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Original comment by iosonofa...@gmail.com on 5 Nov 2011 at 1:11

• Changed state: Fixed