Search for text in images

[techieshark]

Please select if your request is either something new or an enhancement

• [ ] Enhancement of an existing Feature.
• [x] Request of a new feature.

Please select the area your request applies to. (Multiple selections are Possible. You can leave blank if you're not sure.)

• [x] Workspace - VSCode workspace, vaults, Intellisense/autocomplete, Dendron settings
• [x] Lookup - Dendron's Lookup Command
• [ ] Views - Dendron Preview, Tree View, Side Panels in the UI
• [ ] Schema - Dendron Schemas
• [x] Pod - Data import from / export to Dendron
• [ ] Publish - External Site Publish
• [ ] Markdown - Markdown features, such as syntax support and features

Is your feature request related to a problem? Please describe

I sometimes have textual information which is stored inside an image, which could be a photo from a book, or a screenshot of a PDF or something found on the web for example. My problem is how to find this information.

Describe the solution you'd like

First, I'd like a way to make Dendron aware of images. This could be via pasting but preferably also by adding a folder where I store images (or PDFs), or a tool like a web clipper.

Secondly and critically, I'd like a way to search for one or more words that appear in these non-text files. Bare minimum would be being able to search for any one word but ideally also "multiple words in quotes" or using boolean logic foo +AND bar. The system would have previously (upon import) run a step to pull out strings from the image/pdf and indexed this, so the results for this search would come up quickly.

Describe alternatives you've considered

Other tools that have some image search functionality:

• Evernote image search - I've used this but I'd prefer a markdown / text editor rather than Evernote's editor
• DEVONthink "DEVONthink Pro scans paper documents and makes them searchable" [not open source]
• https://highlightsapp.net/ "Extract text from PDFs without searchable text using built-in OCR. Highlights supports a range of languages and does all processing on device." [not a zettlekasten system, just a PDF research tool]
• Hypertag (see blog post) "HyperTag uses the latest research advances from Machine Learning for vision and text to provide easy access to all your files: Search for images (jpg, png) and text documents (yes, even PDF's) content with a simple text query. Text search is powered by the awesome Sentence Transformers library. Text to image search is powered by OpenAI's CLIP model. Currently only English queries are supported." [I might end up using this; haven't tried it yet]
• Haystack (repo) an AI based search system, probably overkill for what I need right now

Additional context

n/a

[Harshita-mindfire]

Thank you @techieshark for your suggestion. Presently, something in our backlog is to support images as ![[link]] note references

[aleksey-rowan]

Linking a related issue:

• https://github.com/dendronhq/dendron/issues/1984