search

viromedia / viro

ViroReact: AR and VR using React Native
MIT License
2.31k stars 483 forks source link

Text detection -> Overlay new text on detected text (Possible?) #450

Closed HarvyJWSim closed 6 years ago

HarvyJWSim commented 6 years ago

Good morning. I'm Harvy and I'm new to ViroReact. Currently I'm working on using AR to detect text, and then overlay some useful info on the detected text. There may be 2 approaches (i think) to solve this problem:

1) Using only the libraries provided by ViroReact (Is it possible? As I noticed much of the use case is focused on image instead of text. Which part of API should I refer to?)

2) Integrate additional text detection libraries to ViroReact (Is it possible? Same problem exists as well where much text detection libraries are focused on still image instead of video (AR).)

Thank you very much and I really appreciate a suggestion for this. :)

dam00n commented 6 years ago

Hi @HarvyJWSim,

The Viro platform uses ARKit and ARCore for image recognition and neither supports text recognition. Here is a thread on options for iOS -> https://stackoverflow.com/questions/45497972/does-arkit-support-text-recognition

There are a couple OCR libraries for React Native available but we have not integrated with any directly. Here is one example -> https://www.npmjs.com/package/react-native-tesseract-ocr

HarvyJWSim commented 6 years ago

Hi @dam00n ,

Really thanks for the tip provided. I understood that currently I need a text recogniser (eg MLKit Text Recognition / Vision Text Detection) to detect text in real time.

Sorry another question: If I do want to integrate certain text recogniser into ViroReact, I will need to create a certain "marker" component (similar to ViroARImageMarker) and then add this component as the child of ViroARScene, is my strategy to counter this problem correct?

Thank you very much and I appreciate the advice given. Have a great day!

dam00n commented 6 years ago

I think it depends on your UI. You could take a screenshot of the text and pass that image to the text recognizer and display the result "on the glass/2D screen space". This would not require new components.

HarvyJWSim commented 6 years ago

Hi @dam00n ,

Thanks so much for the tip! I will try my best to work on it! :)

Have a great day!