Add handwritten (or stylus/mouse written) to latex conversion via AI (e.g., Gemini Vision)

[williamstein]

There's a feature I've always wanted to add to CoCalc, which is the ability to take hand written math and turn it into latex. This would be handy for people writing latex or markdown, since often it's easiest to just write out math the first time instead of tediously typesetting it, especially for some people. The Gemini Vision model from Google actually seems pretty good at this now! I just tried writing a little math on some paper and it had no problem. The selected output math at the bottom of this screenshot was produced by the model:

It looks like this:

Consider $$y^2 = x^3 + \alpha x + \beta \quad \text{and} \quad \int_0^\infty \frac{dx}{x^{2+1}}$$

The exponent isn't exactly right. But it's still impressive, and we should add this.

[williamstein]

Another silly test:

And it gets

A group is:

• foo
• bar
• $sin(x^2)/\Omega_2$

So it's good at markdown with math formulas. I can't get it to actually generate latex at all (e.g., an itemize environment). But I think this is still useful.

[williamstein]

Regarding "or stylus/mouse written", we have a whiteboard in cocalc, so can use that sketchpad to draw something, then feed it to a model.