Add math support to MarkDown preview

szhorvat commented 10 years ago

This is a request to add math (LaTeX) support to the MarkDown preview.

Suggestion on how it should work:

There should be a setting to enable math support. When enabled, $...$ is for inline math, $$...$$ is for display math, and \$ is for an actual dollar sign. This is identical to how it works on StackExchange sites that have math support.

Inline math support is essential to be able to use MarkDown for taking math/research notes (i.e. adding only display math support would make this much less useful).

I am not aware of any desktop application MarkDown editors that have math support like this. (There are several web applications that do this, but all the desktop applications I tried have some limitation such as only supporting display math or being buggy.)

Note that it's not sufficient to simply add MathJax to the header of the output HTML. $...$ pairs should be parsed during conversions and MarkDown syntax should not be interpreted between these delimiters. Otherwise e.g. $x_1,dots,x_2$ won't work: the two underscores must be be interpreted as emphasis by the MD converter.

thedaniel commented 10 years ago

There should be a setting to enable math support

If there is a setting, it should be enabled by default.

szhorvat commented 10 years ago

The usual problem with enabling it by default is that this thinking usually leads to not supporting inline math with $...$ because more people want to type dollar signs than math ...

ccorcos commented 10 years ago

So, this doesn't work yet does it?

thedaniel commented 10 years ago

So, this doesn't work yet does it?

It doesn't even exist yet. Perhaps someone that really wants it will open a pull request.. I don't think anyone on the core team will get to this in the immediate future (though I do think it's a good idea and should be done at some point).

ccorcos commented 10 years ago

I see. Bummer.

ccorcos commented 10 years ago

So pandoc -s file.md --mathjax -o file.html does the trick. I'm not very familiar with developing in Atom, but this seems like a very easy feature to implement if someone wants to do it ;)

szhorvat commented 10 years ago

@ccorcos, would that be fast enough for a real-time preview which is updated as one types?

BTW adding this would make Atom the very first free desktop editor with usable math support. Existing ones are either not free, are web applications, or are simply not usable for serious math because of various seemingly small faults that stem from their programmer not understanding the needs of researchers ...

ccorcos commented 10 years ago

Definitely not quite as real time.

This will give you a good idea of speed. Refresh the page and you'll see. I don't think there will be any drawback in markdown rendering speed because mathjax seems to render after the fact...

Although, Khan Academy just wrote their own LaTeX javascript library which is supposed to be much much faster: KaTeX.

ccorcos commented 10 years ago

This is blazingly fast actually.Looks like they're using mathquill.js which is 100% css!

joyrexus commented 10 years ago

KaTeX, Khan Academy's synchronous latex renderer, looks promising! Note also the recently active discussion on github/markup#274. If Github enables support for rendering embedded latex in markdown (which at least now sounds probable), it would be nice if atom supported the same conventions.

Gosh, if atom and github had good/consistent latex support ... it would be hard to overstate what a boon this could be for scientific publishing and reproducible research.

ccorcos commented 10 years ago

good point. We should see what Github does.

Galadirith commented 10 years ago

I've actually been working on implementing this feature over the last couple of days in my mathjax branch galadirith/markdown-preview. Theres alot to do before I could think about opening a pull request, but it does work right now.

Inline math support is essential

@szhorvat It does handle both inline and displayed maths, but I haven't rigorously tested the parser yet.

Note that it's not sufficient to simply add MathJax to the header

The markdown parser will leave any LaTex alone, so there are no issues with interpretations as emphasis for example.

If anyone does want to test it out you'll need to clone MathJax into your ~/.atom. You can't use the cdn because of the csp

vikinsights commented 10 years ago

@Galadirith - I've cloned both your branch of markdown-preview and MathJax into my ~/.atom per your instruction above and i am unable to get any kind of equation to render in the prewiew window (e.g. $$ x \ne \y $$). It shows the equation as is. Have i misunderstood your instructions or should i do something else in order to test your version out?

Galadirith commented 10 years ago

@kkptNinja - I've added some instructions on how to install and use LaTex rendering LATEX.md. Please let me know if you have any problems.

Galadirith commented 10 years ago

I've kinda concluded that MathJax is probably not suitable to use in Atom (which other's have probably already realised ;D). MathJax is great at what it does, and thats typesetting static webpages.

I've switched to a new development branch galadirith/katex, which depends on KaTeX. It is more limited in the LaTeX that it can process compared to MathJax, which mostly comes down to the fact it can't really handle environments \begin{...} yet. However it is orders of magnitude faster than MathJax, and this branch is already way more functional than my MathJax branch.

If you want to try it out then I have written up some more instructions LATEX.md. Please let me know what you think and once i've written up some Jasmine specs ill think about a pull request.

szhorvat commented 10 years ago

I've tried it and here are some questions and suggestions:

Would it be possible to add \[ and \] for display math separators?
Why is an empty line required before/after the $$ separators?
Some common and standard LaTeX, such as \mathbf{x}, does not work. Is this a KaTeX limitation?
When some LaTeX can't be handled (incorrect syntax or unsupported, like \mathbf) then it's simply not rendered as math. Is it possible to add some visual indication that there's an error? For example, MathJax-enabled StackExchange sites will simply frame the incorrect equation:

Regarding points 1. and 2., these would be very valuable for compatibility with other renderers, and actual LaTeX in particular. The most common display math separator in actual LaTeX is \[ (not $$, which is discouraged). Also, in LaTeX, putting spaces before/after a display math block does change the output. An empty line indicates a paragraph break, and will influence both spacing an indentation. When I write LaTeX (not markdown), I don't usually break paragraphs at every piece of display math.

It would be very useful if text could be pasted between Atom MarkDown, StackExchange MarkDown and actual LaTeX with minimal editing. Allowing \[ ... \] separators and not requiring empty lines before/after $$ would make this much easier.

thedaniel commented 10 years ago

There's a lot of interesting stuff going on in this thread - I think another very promising / performant / complete route would be to use Lasem, but unfortunately node bindings for this do not exist. It might be worth a hack if someone interested in this has experience wrapping C libraries for node.

Galadirith commented 10 years ago

@szhorvat Thanks for trying it out :D

Would it be possible to add \[ and \] for display math separators?

I've just added support for them.

Why is an empty line required before/after the $$ separators?

I wanted to follow the behaviour of other markdown block level elements that mostly require empty lines separating them from other elements. But your right LaTeX doesn't require this, so Ive dropped that requirement. However weather you separate the block with no empty lines or a billion, it wont change the out, unlike LaTeX. The fact that LaTeX does change its output if empty lines surround a displayed equation is actually something thats always frustrated me about it, I would prefer that it did, but thats probably just me. But please do let me know your thoughts on the current behaviour :D

Some common and standard LaTeX, such as \mathbf{x}, does not work. Is this a KaTeX limitation?

Yeh I tried \mathbf{...} and a couple of other commands that dont work. It is a limitation of KaTeX. If you go to this demo then you can see a reasonably complete list of what is and isn't supported. KaTeX does need to improve, but there are a bunch of pull request on its repo, and lost of people wanting to improve it, so I do believe it won't be long before all the core common LaTeX commands are support.

When some LaTeX can't be handled (incorrect syntax or unsupported, like \mathbf) then it's simply not rendered as math. Is it possible to add some visual indication that there's an error? For example, MathJax-enabled StackExchange sites will simply frame the incorrect equation:

Done, you'll get exactly that visual cue now, with the modification that it will say 'Bad KaTeX:' instead of 'Bad LaTeX:'. I need to look into exactly what errors KaTeX throws, im not to sure its its just a general error or it cases on weather its an actual LaTeX syntax error, or just that it can't handle those particular LaTeX commands.

I think another very promising / performant / complete route would be to use Lasem

@thedaniel That looks great too. I have no experience in wrapping C libraries for node, but if I get the time ill definitely look into that. For me the first thing now would be to write some sweet Jasmine specs (which I have also never used before)

Updated instructions can be found at LATEX.md (and im really sorry for the long comment :D)

szhorvat commented 10 years ago

Wow, that was quick :smiley:

Done, you'll get exactly that visual cue now, with the modification that it will say 'Bad KaTeX:' instead of 'Bad LaTeX:'. I need to look into exactly what errors KaTeX throws, im not to sure its its just a general error or it cases on weather its an actual LaTeX syntax error, or just that it can't handle those particular LaTeX commands.

Actually I don't think it's necessary to write "Bad KaTeX". In my screenshot I wrote that by hand, StackExchange doesn't show it automatically. It only adds the frame. If you agree too, I'd prefer only the frame, no text added.

Question:

As KaTeX is being improved, do I need to re-download it to take advantage of improvements? It's not clear whether I needed to download that only for the fonts. (I'm really hoping to get \mathbf and \mathbb support from them, I use those very frequently.)

In the following days/weeks I'll try to use this for actual work (i.e. taking notes) to see what pain points come out.

szhorvat commented 10 years ago

Found a bug: the very first character in a file can't be $

Galadirith commented 10 years ago

In my screenshot I wrote that by hand, StackExchange doesn't show it automatically

Oh ha my bad :D, I haven't used StackExchanges maths so I didn't realise that. Yeh I think the frame is the right balance of not too intrusive, but you can spot where the error is if you need to.

As KaTeX is being improved, do I need to re-download it to take advantage of improvements?

So I can't say for certain. The actually parsing from LaTeX to html is done by the dependecy katex in the package.json. So if KaTeX's maintainers only update their javascript, and not their CSS, then you'll be fine with simply a apm update in my forks directory. However I really don't know what future command updates in KaTeX might also require an update to their CSS.

If it does require an update to the CSS, that will NOT mean that the preview will break, it will simply look like a particular character doesn't belong, and you will know then that the CSS needs updating.

KaTeX when used on a server (which is essentially what KaTeX's node package is designed for) assumes that you'll be serving webpages that have access to the CDN, so you wouldn't need the CSS files on the server. I might open up a pull request to have them include the CSS and fonts in the node package, its a simple fix in their package.json to make them include it. Or i'll just fork it and point my markdown-preview fork at my KaTeX fork (lots of forks to go around). In that case you would only ever need to run apm update.

Wider support would obviously be great. @thedaniel do you know how good the suppot for LaTeX commands is in Lasem.

szhorvat commented 10 years ago

Hm, I played a bit more with it. Unfortunately the limitations of KaTeX are just too many at the moment to make it usable for actual research notes ... or anything beyond elementary calculus. \mathbf, \sim, \equiv, \propto, \nabla, \Rightarrow, cases, arrays, n-th root, are just too many essentials missing.

Let's hope that either KaTeX will improve soon, or MathJax will get faster. Other than the limitations coming from KaTeX what you did here is the very best I've seen in any desktop editor.

Galadirith commented 10 years ago

Found a bug: the very first character in a file can't be $

Fixed the bug. And thanks for testing out my fork. Im sorry that its to limited right now to use in you workflow. The branch will be there and can just wait for updates to KaTeX, so I am looking forward to its future. Thanks for your kind words.

I am determined to build functional LaTeX support into markdown-preview though, I just really fraking want it, and I would use it everyday, every hour, every minute ... :D. So when I get some time next week I will definitely experiment with Lasem.

Planeshifter commented 10 years ago

@Galadirith Thanks for all your efforts, they are greatly appreciated! Writing LaTeX math in atom was something I really hoped for, very cool that it is becoming a reality. KaTeX does seem to be very limited at the moment, though. However, I also did checkout your mathjax branch and it seems to be working quite smoothly. Could you elaborate a bit on what problems you ran into? Is it getting too slow for longer documents?

Galadirith commented 10 years ago

@Planeshifter sorry I didn't get a chance to reply last night, and thanks so much for checkout out my branch :D

So the problem that I had with my MathJax branch was not actually a functional one, it was user experience, let me give you an example scenario.

I've got a page with lots of LaTeX expressions, and I change something. The markdown parser starts to process the page, and it updates the html in the preview window, but its missing all the equations. Then a few seconds later the equations pop in. I'm sure you've experienced something similar on websites using MathJax with lots LaTeX.

A consequence of the page being displayed without equations is that its physically a smaller number of pixels in hight. This can affect the scroll location, which is not based on a percentage of the page, but the number of pixels its scrolled (for example scrollTop is based on pixels not percentage).

So you may have your preview window scrolled to where you want it to be, you update your markdown source, and then your looking at the wrong location in the preview!

That issue won't affect everyone, and some people might not even care, but I personally dont like that experience.

Your question made me think about a decision I had made earlier in the week, that I would process the parsed markdown with MathJax only after I had inserted that into the pages html. I made that decision because I thought I had read MathJax's could do it any other way. Turns out I must have dreamt that (I can't find the supposed article I read it in) or it referred to and old version. Either way I realised I hadn't tested that for my self ... rookie mistake :D

I went back last night, hacked a quick test together, and realised I had made design decision on untrue info. So to cut a long story slightly less long, ive updated my branch to address this UX issue ... it fraking works :D

MathJax's is still slow, by that I mean it could take anything up to 3sec to render the preview if you've got a big markdown document. But I have some ideas that I am hopefully will dramatically improve the lag between altering the source and seeing an update in the preview window for certain use cases.

So @Planeshifter, @szhorvat and anyone else who'd like to try it out, contrary to my previous comment saying that I don't think MathJax is suitable, do use my MathJax branch, and it absolutely supports commands like \mathbf, \sim, \equiv, \propto, etc :D. Updated instructions here LATEX.md

(PS Im so sorry again for the massive comment)

szhorvat commented 10 years ago

I'm playing wit the mathjax branch now. Here's a possible bug: if putting a piece of inline math on a separate line alone (no other text) then the redered math will have quote marks around it:

Planeshifter commented 10 years ago

@Galadirith I will try to take some math notes with atom today and report any bugs that I may find. I also stumbled upon the problem found by @szhorvat, which seems to happen whenever a new line is started with inline math (the problem occurs both when the inline math stands alone as in above screenshot, but also when it is followed by other text).

Galadirith commented 10 years ago

Thanks guys :D. I've pushed a fix for the bug. Here's my todo list

[ ] Write Jasmine specs (I really appreciate all the bugs being found, but I need to make sure there is a systematic way to kill the easy ones, I don't want to keep pushing buggy code to you all, but by all means keep letting me know if you find any bugs).
[x] Work on use case performance boosts.
[x] ~~Build node wrapper for Lasem~~ UPDATE: I no longer believe this is necessary having translated markdom's Levenshtein distance algorithm.

Couple of questions:

@szhorvat where do you upload images to? I havn't been able to find a definitive answer of where to host images I plan to reference in markdown on github
To any maintainer, my code is definitely not suitable for merging yet by any means, but would it be appropriate to open a merge request so that its easy to comment on my code? (The first time I have worked on open source/ used coffescript/ hacked atom was at the beginning of last week. ive been trying to follow the contibution guidelines but I am sort of new to it all, and whats appropriate, thanks :D)

Thanks everyone

szhorvat commented 10 years ago

I was just following the instructions at the bottom of the editor and pasting images directly here:

They're hosted by GitHub. I think pasting doesn't work with all browsers but drag and drop or selecting should work.

ccorcos commented 10 years ago

One benefit of MathJaX is that it fully supports LaTeX \newcommand. For example:

\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\newcommand{\entropy}[1]{\text{H}\left[\;#1 \; \right]} % entropy
\newcommand{\expectedValue}[2]{\hollow{E}_{#2}#1} % expected value
\newcommand{\hollow}[1]{\mathbb{#1}} % hollow letter

Is this extremely useful if you're writing a substantial amount of math....

Galadirith commented 10 years ago

Hey everyone :smiley:, Im sorry I havn't updated over the last week.

I've created a new branch mathjax-markmon which is based on code from yyjhao/markmon. Please do check it out if you get a chance, its such a massive improvement over previous versions, its just stupidly more responsive for larger markdown documents with lots of equations.

markmon implements a Levenshtein distance algorithm that only updates the preview windows html with whats been changed. markdown-preview never implemented something like this I guess simply because it worked fine without it. I translated yyjhao's algorithm to coffescript (if anyone wants to look at the translated Levenshtein code without all my MathJax additions you can checkout this branch markmon).

The most important thing that this algorithm provides is the ability to only ever process LaTeX that has changed. Instructions on installing and using are still the same and can be found here LATEX.md

Planeshifter commented 9 years ago

@Galadirith Very cool, I have been working with it and think your implementation is without a doubt usable for daily work now and offers functionality I haven't seen in any other editor.

I have only found one bug so far: If a document starts with a display math equation in the second line, it is rendered in both the editor window and the markdown preview: screenshot from 2014-11-17 13 00 23

In general, do you have any suggestions/preferences on how one could help you in testing the code and report any remaining problems to you? Thanks again for your great work!

Galadirith commented 9 years ago

Thanks @Planeshifter. I couldn't reproduce the bug that rendered in both the editor and preview, however I did see that the html wasn't being tag'ed correctly for the cases when a document either started or ended with a displayed equations, so I've pushed a fix for that. Let me know if that fixes your bug. I specifically tested with a document containing:

$$
e^{i\pi} + 1 = 0
$$

I really appreciate all the bug reports. I think the easiest thing to do right now is just continue to post to this issue thread with any bugs (I check it everyday :D). Markdown snippets that cause the problem would really help, or if you want to pass me a larger document causing problems paste it to a gist and link me to it :D

Planeshifter commented 9 years ago

Yes, your change does seem to have fixed the problem, thanks!

ccorcos commented 9 years ago

Any ideas when this will be added to markdown preview?

ccorcos commented 9 years ago

Hey, I really don't know much about developing for atom. Any chance you could make this a package I can install through atom for now?

Galadirith commented 9 years ago

Any ideas when this will be added to markdown preview?

Hey @ccorcos. I really couldn't say when, or even if this will ever make it into the core markdown-preview package. Its not that it doesn't work, I suppose im biased but I think it works just as we'll as any markdown editor with LaTeX support out there, except this is completely offline which is great. The reason I say it may never be merged is because I don't think LaTeX support is high on the priority list for the core devs (I could be wrong but thats just my guess) and I have made quite a lot of non-trivial (sometimes hacky) additions to the core code that it would take some effort for one of the core team to review.

Any chance you could make this a package I can install through atom for now?

Its already a package :D. It's not available through atom's package manager, but please just follow the Installation section of my instructions which you can find here LATEX.md. Let me know if something doesn't make sense in my instructions.

Any if you find a bug please post it here and I get to work as quickly as I can to fix it. Let me know what you think :D

ccorcos commented 9 years ago

I see. It would be nice if it was available through apm but if you don't want to do that, could you provide me with some instructions on how to uninstall it too? I'm not too familiar with all this stuff.

ccorcos commented 9 years ago

Its not working -- I believe its conflicting with the existing build-in markdown preview

Galadirith commented 9 years ago

Sorry about that @ccorcos.

I believe its conflicting with the existing build-in markdown preview

When you run the command apm link or apm --dev link this will override the built-in markdown-preview package, so that shouldn't be the problem.

I've updated the instructions on how to install it to make it a little more user friendly, and also added uninstallation instructions as you requested, LATEX.md. If there are any steps you don't understand please do ask specifically about it.

Things to check

What OS are you using? Im using windows and only able to test on that, but there shouldn't be any cross platform issues that I am aware of.
Did every command run without error? I had issues with apm install because I was using a 3.x version of python, you need a 2.7.x version for it to work.
Did you simply open up atom normally (eg click on its icon) instead of running the command atom --dev? I've updated the installation instructions so that it will now work when you open atom normally. If you follow the new instructions there should be a symbolic link to markdown-preview in the folder ~/.atom/packages. (under the old instructions this symbolic link would be placed in ~/.atom/dev/packages)
If it installed correctly you should see the menu item Packages › Markdown Preview › Toggle LaTex Rendering. If you can't see that then open up DevTools with ctrl+alt+I and click on the console tab. Do you see any error messages intimating packages not loading?

I would be more than happy to help you identify why its not working. It would probably be easiest to message you directly on fb, skype, google+ etc if you like, to help you get it up and running. Otherwise just post back here.

thedaniel commented 9 years ago

I think it would actually be pretty useful to fork the package completely and add add it to APM under another name for people like @ccorcos who aren't interested in Atom development or package development directly. Then the install instructions would be as simple as "disable the built in package and install this one". Since it's just as simple as changing the package.json and readme it seems like an easy win, and easy to revert if the changes get merged upstream eventually.

In any case it's likely that at some point in the future math support will be added to this package, and it's totally plausible that it will be through @Galadirith's awesome hard work but since markdown-preview is a pretty important core package I think we'll be somewhat conservative in merging changes - lots of code review etc - and since this isn't our top priority right now it might be a while before that gets done.

Galadirith commented 9 years ago

Thanks so much for the post @thedaniel. I was hesitant to publish my fork under a new name, I thought it might be bad form, so I really appreciate your comments. I'll get right on making the necessary modifications and publishing my fork to atom.io. I'll update when thats done (and if anyone has a good idea for a name, suggestions would be appreciated, right now im thinking ladown-preview ... yeh maybe)

stephen304 commented 9 years ago

marktex-preview? Btw thanks for the amazing work - With markdown preview I was determined to take notes in all my classes in md but this just completes it so I can take calculus and discrete math notes as well.

Galadirith commented 9 years ago

Hey everyone :smile: Thanks for being patient. I've finally managed to put together a package that ive just published to the Atom Package registry. Really excited for you to try it out, its been published under the name of markdown-preview-plus (@Stephen304 thanks for your suggestion). I really hope this will make it significantly easier to get working with LaTeX in Atom.

szhorvat commented 9 years ago

That sounds great! But is the patch still considered for inclusion in the official markdown-preview? Did you get any feedback from any Atom developers?

szhorvat commented 9 years ago

Hmm, if the dependencies are ~90MB then probably not.

With this version it is not necessary to manually download MathJax into ~/.atom, right?

Galadirith commented 9 years ago

it's likely that at some point in the future math support will be added to this package, and it's totally plausible that it will be through @Galadirith

@szhorvat, @thedaniel made that comment a little while ago. So its 'considered' but its not going to make it in in the near future, or ever nessesarily, im really sorry its so up in the air.

With this version it is not necessary to manually download MathJax into ~/.atom, right?

Yep, absolutely not. All you need to do install the package like any other, and MathJax will be downloaded automatically. You basically don't need to do anything outside of Atom.

Markdown Preview actually already has ~45MB worth of dependencies, and the remaining ~45MB is MathJax.

szhorvat commented 9 years ago

I completely reset Atom by removing .atom as I saw some weird things: the disabled official markdown preview had a setting on its page called Render LaT eX (sic).

Then I installed Markdown Preview Plus and waited until it showed up in the list of packages.

However, it didn't start rendering LaTeX until I restarted Atom (no matter how many times I selected "Toggle LaTeX rendering"). Just wanted to let you know about this, as the installation instructions didn't say that a restart was necessary.

Thank you again for your work, this is a package that many of us are going to use regularly.

Galadirith commented 9 years ago

Thanks for the heads up @szhorvat. I actually had a similar thing happen when I did a test install from the registry, but it was never a problem when I was doing clean local installations. I saw some error msg related to 'incompatable native module' and a list a few modules (none of which were added by me), and the error disappeared on restarting Atom.

I don't think it was anything to do with my additions, but I think I push a note to the installation section of the README, and see if I can find any trail of this sort of error elsewhere on the interwebs.

ccorcos commented 9 years ago

works great!!!

atom / markdown-preview

Add math support to MarkDown preview #116