search

immersive-web / proposals

Initial proposals for future Immersive Web work (see README)
95 stars 11 forks source link

Optical Character Recognition #64

Open AdamSobieski opened 3 years ago

AdamSobieski commented 3 years ago

Introduction

I would like to propose that XR-device-based optical character recognition be considered as an important XR scenario.

Optical Character Recognition

With XR devices and their sensors, users could scan text and mathematics content, from papers, chalkboards, dry-erase boards, and other surfaces.

Multimodality

While using XR devices and their sensors to scan text and mathematics content, users could read the content aloud to enhance the accuracy of scans. This could also be interoperable with eye tracking.

Interactivity

Multimodal dialogue systems could interact with users to ensure that contents are thoroughly, properly, and accurately scanned.

physikerwelt commented 3 years ago

Check out https://mathpix.com/ @mathpix . I can imagine their approach would work in XR scenarios.

cabanier commented 3 years ago

Do you know of any devices that are planning on having support for this? If not, it's too early to take this up in the group.

AdamSobieski commented 3 years ago

@physikerwelt , thank you for the link to Mathpix. That is an impressive technology. Some interesting XR computer algebra system user interfaces can be envisioned.

@cabanier , while any XR device with external cameras and/or microphones is relevant, I do not know of a specific XR hardware vendor with interest at this time. Software vendors in the XR business collaboration space may have an interest in STEM collaboration scenarios but I haven't contacted any.

Please let me know if this specific proposal issue would be better to present again at another time.