Text/Math Recognition

[OmegaRogue]

Is your feature request related to a problem? Please describe. Taking handwritten notes is useful for quickly writing down things with sketches or math without the need for a keyboard, as a replacement for taking notes on Paper, but having digital Text is useful for searching through it and other use cases. Describe the solution you'd like Having the option to select handwritten text and convert it to a text box, and the option to select formulas other math to convert it to a formatted (maybe LaTeX) math element, like Xournal++ supports with LaTeX integration.

Additional context OneNote supports OCR for mathematical formulas and equations

[PhilDevProg]

I found this: https://pyimagesearch.com/2020/08/24/ocr-handwriting-recognition-with-opencv-keras-and-tensorflow/ which describes how to do handwritten text recognition by using OpenCV, Keras & TensorFlow using Python.

[ashrude]

The process described in there would technically work, but since we'd have access to the stroke information, it could be implemented much better. Having stroke information allows for higher accuracy.

[Doublonmousse]

For the math conversion, this looks more like Task 1 of the CROHME competition https://crohme2023.ltu-ai.dev/data-tools/ This uses inkml as the file format for the stroke data and uses the stroke information ((x,y) position only).

Ideally, this would work in an incremental fashion and update the prediction for each new stroke rather than (or in addition to) only be accessible as a tool that convert a stroke selection to the math equivalent.

Conversion of a selection is only good the conversion is very good. If not you end up trying it a couple time, see that it fails to convert moderately complex equations and end up not using it (all the more so if there is no easy way to modify the converted equation afterward).

Though #788 would have to be completed before this can be tackled