Image output

[amyjko]

What's the problem?

There are many reasons why creators might want to include images in their content. But we have to do this delicately, as images are inherently inaccessible (to blind creators, often to color blind creators), and greatly increase bandwidth and storage

What's the design idea?

One possible way to design accessible, multilingual use of images would be to permit them, but require descriptions, and to pull out the text of images automatically. We could then add images as a type of output, like Phrases, with all the same features, but showing an img tag instead of text.

Design specification

Proposal:

Features

• Automated Image Description Generator:

  • Automatically generates concise and accurate alt-text for images using AI technology. This text is essential for screen readers and aids in making content accessible to visually impaired users.
  • Users can review and modify the generated descriptions to ensure accuracy and relevance of the output image.
  • Example: NVDA screen reader - Image description: https://gist.github.com/ShubhamJain7/7986b9056376c9a70e744e0ab3100b85

• Multilingual Support:

  • Automated Image Description Generator should support multiple languages, allowing alt-text to be automatically generated in the user’s preferred language.
  • This feature enhances the accessibility of content for non-English speaking users. Provide a more comfortable platform for all users from different language backgrounds.

• Image Text Extraction:

  • Automatically extracts text embedded in the images by using Optical Character Recognition technology.
  • Extracted text is presented alongside the image as searchable, selectable, and translatable text. This will provide a more comprehensive description of the image and can be translated into different languages also.
  • Example: OCR with Google AI https://cloud.google.com/use-cases/ocr

Design Specifications

• Image Upload Interface: Simple drag-and-drop area with options for manual alt-text entry and language selection.
• Image Review Panel: A side panel where users can see auto-generated alt-text and extracted text, with options to edit or approve.
• Accessibility Simulation Tools: Visual tools that simulate different visual impairments on the uploaded images to provide immediate feedback on accessibility.

[ShufaW]

Features

• Automated Image Description Generator:

  • Automatically generates concise and accurate alt-text for images using AI technology. This text is essential for screen readers and aids in making content accessible to visually impaired users.
  • Users can review and modify the generated descriptions to ensure accuracy and relevance of the output image.
  • Example: NVDA screen reader - Image description: https://gist.github.com/ShubhamJain7/7986b9056376c9a70e744e0ab3100b85

• Multilingual Support:

  • Automated Image Description Generator should support multiple languages, allowing alt-text to be automatically generated in the user’s preferred language.
  • This feature enhances the accessibility of content for non-English speaking users. Provide a more comfortable platform for all users from different language backgrounds.

• Image Text Extraction:

  • Automatically extracts text embedded in the images by using Optical Character Recognition technology.
  • Extracted text is presented alongside the image as searchable, selectable, and translatable text. This will provide a more comprehensive description of the image and can be translated into different languages also.
  • Example: OCR with Google AI https://cloud.google.com/use-cases/ocr

Design Specifications

• Image Upload Interface: Simple drag-and-drop area with options for manual alt-text entry and language selection.
• Image Review Panel: A side panel where users can see auto-generated alt-text and extracted text, with options to edit or approve.
• Accessibility Simulation Tools: Visual tools that simulate different visual impairments on the uploaded images to provide immediate feedback on accessibility.

[amyjko]

Thanks @ShufaW! Is this a design proposal? Are you asking for feedback? Per the design wiki, proposals go in the issue body, not a comment, and when you're ready for design feedback, you need to tag a maintainer. Let me know if you want me to assign this issue to you, so you have edit access.

[ShufaW]

@amyjko This is my proposal for the Image output. Can you please review the proposal? Thanks.

[amyjko]

Thanks for your proposal! A few comments:

• Automated image descriptions and translations are often of very low quality. How will this design ensure that users don't just accept low quality descriptions?
• It's not clear what would be done with the extracted text. Are you proposing that it would be part of some data structure somewhere? What is the use case for extracting it?
• How would the image interface with the existing output APIs? Would the images be used as backgrounds, as output content? How would they interact with the physics engine and other features on the platform?

[ShufaW]

@amyjko Hi Amy, Here the improvement for the first two question.

• Improving the Quality of Automated Image Descriptions

  • Implement a system where initial AI-generated descriptions and translations are reviewed and validated by a team of trained volunteers or professional editors before being finalized. Also, the volunteers could also give the feedback and the improvement advice to the AI generated description.
  • Create a platform where users can contribute to the review process. Incentivize participation with a gamified system that rewards users for accurate reviews.

• Use Case for Extracted Text

  • After OCD technology extract the text, the extracted text should be read to the users. Such as "This photo describes xxx and includes the words xxx."

Still doing some research on the API.

[amyjko]

Thanks for the ideas @ShufaW! I have many design questions about these ideas:

• Who would train the volunteers?
• How would the project afford professional editors?
• What would the qualifications be for the professional editors?
• What would the motivation be for creators to review the generated image descriptions of other creator's projects?
• For the extracted text, is the idea that the image description and extracted text would be combined into a single image description? How would they be combined?

[joonpiter]

@ShufaW, will you go forward with this issue?

[ShufaW]

Yes I will. But if anyone else want to focus on this issue I am okay with it.