FR: OCR - map OCR recognized symbols into other markdown syntax

[schniewmatz]

Is your feature request related to a problem? Please describe. It is not a problem, rather a nice idea on how to connect handwriting with other features of Obsidian:

Fist of all: The OCR tool in Excalidraw works like a charme for me! I do a lot of hand written notes in a bullet journal style and having this searchable is just splendid! Pasting content from there into my daily notes is also from the clipvboard is super useful.

While using the tool, I recognized, that a handwritten box gets recognized by the taskbone OCR as "□". And that brings me to my idea:

Describe the solution you'd like Wouldn't it be nice to define specific symbols in the OCR output to be replaced with other text? For e.g. one could define "□" to be replaced with "- [ ]", so if I insert the OCR output into my notes, I directly get lines starting with a handwritten box converted to actual tasks.
One cloud also use another symbol like e.g. an "@" sign for a specific tag e.g. "#interesting". In that way you mark content you want to come back to later, e.g. to do a little research on the topic, etc.
I guess the community comes up with many more cases for this functionality 😉 Maybe the right location to configure this would be a mapping table in the plugin's settings like:

OCR recognized symbol	Text in clipboard
□	- [ ]
@	#interesting

Additional context In Bullet Journal one uses specific symbols at the start of a line to mark the type of content, e.g. a stroke for a note, a dot for a task, a circle for a date or deadline, an exclamation mark for interesting stuff... I really like it for my handwritten notes, especially as i memorize stuff much better when writing by hand, and it makes the notebooks easily skimmable by eye. But:
I am using it now for more than a year and when i go back to my notebooks, I find stuff, that is super interesting and relevant, but I forgot about it and I can not search for it somewhere.

So I started using Obsidian a few months ago to capture this content. It sometimes works, sometimes I simply lack the time. I know that people are using Android E-Readers to use Obsidian there and I use a Wacom at my workplace to take notes. So I got the idea of using an Android E-Reader instead of my paper notebook, with Excalidraw directly in Obsidian Mobile, and using your OCR function to make it searchable and partly converting important stuff to text. Having the mapping of symbols into tasks, tags, natural language dates, etc. would then perfectly convert the handwritten bullet journal into the digital equivalent.

[zsviczian]

This is probably easier said than done... as the OCR output is not always that clean. Where would you place these actions? in the frontmatter? or somewhere else? My initial idea is a script engine script that you could customize to process the content from the clipboard...

[schniewmatz]

When I scan this:

I get this in the clipboard: □ this is a task @ this is a predefined tag

With the mapping above I would like to have the following in the clipboard instead: - [ ] this is a task

intersting this is a predefined tag

So it would be a conversion of specific symbols in the OCR output into another string.

From how i use your tool currently, I would not put it in the frontmatter and rather keep the "raw" OCR output there. I use the clipboard if I decide to further work with a handwritten note and the frontmatter to make things searchable. So I usually don't clean up the OCR output that ends up in the frontmatter.

[schniewmatz]

Let me first watch your Video of the script engine, maybe that already solves it for me 😄

[schniewmatz]

Okay, so from how the code is structured, the text is written into the clipboard at line 98: window.navigator.clipboard.writeText(text);
in src/ocr/Taskbone.ts

So for a script I could run the getTextForView() method of the Taskbone plugin:

ea.plugin.taskbone.getTextForView(view, true);

then get the text from the clipboard with:

text = window.navigator.clipboard.readText();

do some modifications and then write the text back to the clipboard with

window.navigator.clipboard.writeText(text);

When I do this in the developer console, I get:

Uncaught (in promise) DOMException: Document is not focused.

Do I do complete nonsense 😅 or does it at least make some sense?