Voice-Based Form Filling Component for Flutter to Enhance Digital Access to Public Services

[Ramkrishna-egov]

Description

In light of the digital transformation of public services, this project endeavours to create a UI component tailored for Flutter applications. The main objective is to address the challenge faced by many citizens in completing digital forms by developing a voice-based form-filling solution. This component, when integrated into a Flutter form, will automatically extract existing form elements and conversationally engage users to facilitate form completion. By offering a generic voice-based form-filling capability, the project aims to significantly enhance the user experience of Flutter-based apps. Given the increasing popularity of Flutter for building mobile applications used by citizens and frontline workers, such innovation has the potential to streamline interactions with digital services and improve accessibility for all users.

Goals

Design and develop a voice-based form-filling component specifically tailored for Flutter applications on DIGIT. Create a conversational interface that engages users to facilitate form completion using voice commands. The interface should allow the user to hear as well as see the voice prompts and recorded responses.

Expected Outcome

• Design and build UI elements to activate voice based form filling
• Implement logic to extract existing form elements from Flutter forms and dynamically generate conversational prompts for users.
• Design and build of Interface where users can see written versions of the voice prompts and recorded responses. Interface should provide feedback to users during the form-filling process, including confirmation messages, error notifications, and prompts for clarification.
• Design and build of a backend service or API that can process speech input and convert it into text. This could involve utilizing existing speech recognition APIs (must be open source)
• Integration/build of LLM to understand and interpret user’s input -Acceess to Open AI or alternate LLM can be provided
• Thorough testing and debugging to ensure reliability, accuracy, and performance of the component across different devices and platforms.
• Optimization of the frontend code for efficiency, minimizing resource usage and maximizing responsiveness during voice interactions.

Acceptance Criteria

Voice-Based Form-Filling Component Development:

• The voice-based form-filling component must be designed and developed
• The component should work on Flutter applications on DIGIT.
• The component should seamlessly integrate into existing Flutter forms, allowing users to interact with forms using voice commands.
• Users should be able to activate the voice-based form-filling feature through a designated UI element.

Conversational Interface:

• The component must feature a conversational interface that engages users during the form-filling process.
• Users should be provided with clear and concise voice prompts to guide them through the form completion process.
• The interface must allow users to both hear and see the voice prompts and recorded responses.

Form Element Extraction and Dynamic Prompt Generation:

• The component should automatically extract existing form elements from Flutter forms.
• Dynamic conversational prompts must be generated based on the extracted form elements to facilitate user interaction.

User Interface Design:

• A user interface must be designed and built to display written versions of the voice prompts and recorded responses.
• The interface should provide feedback to users during the form-filling process, including confirmation messages, error notifications, and prompts for clarification.

Backend Speech Processing Service:

• A backend service or API must be implemented to process speech input and convert it into text.
• The backend service/API should utilize open-source speech recognition APIs to achieve speech-to-text conversion.

Integration with NLP Service:

• Integration with an NLP service or API is required to understand and interpret user input.
• The NLP service/API must be capable of analyzing user utterances and extracting relevant intents and entities.

Thorough Testing and Debugging:

• Comprehensive testing must be conducted to ensure the reliability, accuracy, and performance of the voice-based form-filling component.
• Testing should cover different devices, platforms, and scenarios to identify and address any issues or bugs.

Frontend Optimization:

• The frontend code must be optimized for efficiency to minimize resource usage and maximize responsiveness during voice interactions.
• Measures should be taken to ensure smooth performance across various devices and screen sizes.

Out of Scope:

• The component should be able to identify the selected language of the user and provide voice prompts accordingly

Implementation Details

• Libraries/3rd Party tools may be used but must be open-source
• DIGIT uses Java Spring Boot in the backend and Flutter for the frontend for mobile applications
• Staging environment of DIGIT will be provided and an existing form from a product may be used.

Mockups / Wireframes

• To be created by the selected contributor with support from DIGIT product manager

Product Name DIGIT

Project Name Voice-Based Form Filling Component for Flutter

Organization Name: eGovernments Foundation

Domain Public Services

Tech Skills Needed: Flutter, Java Spring Boot

Mentor(s) Ramkrishna Sahoo

Complexity High

Category Feature

Sub Category API, Frontend, Backend, LLM

[EuclidStellar]

Weekly Learnings & Updates

Week 1

1. Research and Development on Digit Component: Explored the digit_components package to understand its functionality and how it can be leveraged for form fields. Identified the key components that could be integrated with voice commands.

2. Library Exploration: Investigated libraries such as speech_to_text and flutter_tts for integrating voice commands. Evaluated the pros and cons of different libraries to determine the best fit for the project.

3. UI Components for Voice Commands: Created basic UI components to handle voice commands and form submission. Developed initial prototypes to test the integration of voice commands with form fields.

Week 2

1. Voice Integration on Dummy UI: Implemented basic voice integration on a dummy UI to test the feasibility. Ensured the app can listen for voice commands and update the UI accordingly.

2. TalkBack Feature Exploration: Experimented with TalkBack as an alternative to text-to-speech for providing audio feedback to users. Compared the user experience and effectiveness between TalkBack and flutter_tts.

3. Semantic Widget Integration: Integrated semantic widgets to improve accessibility and provide better screen reader support. Ensured that the app is accessible to users with visual impairments.

Week 3

1. Flutter TTS and Speech-to-Text Integration: Successfully integrated flutter_tts and speech_to_text packages into the project. Implemented voice command functionality to fill form fields.

2. DigitTextField from Digit Components: Incorporated DigitTextField from the digit_components package to manage form fields. Created UI components to handle voice commands and form submission.

Week 4

1. Form Validation and Snack Bar Notifications: Implemented form validation to ensure user input meets required criteria. Provided user feedback through a snackbar notification upon successful form submission.

2. Reactive Text Form Field: Used the Reactive Text Form field in code along with form control. Transitioned DigitTextField to DigitTextFormField.

3. Digit Components with Voice Commands: Added more Digit Components enabled with voice commands such as DigitReactiveDropDown.

Week 5

1. Flutter Material 3 Widget Implementation: Implemented voice command functionality on Flutter Material 3 widgets. Transitioned this functionality to DigitReactiveDropDown.

2. Voice Command in Digit Component (DigitCheckBoxTile): Added voice command functionality to DigitCheckBoxTile (reactive checkbox). Used "agreed" as a keyword to mark the tick in the checkbox.

Week 6

1. Tested Multiple Open Source Model Tested multiple open-source models for speech recognition and benchmarked them accordingly GitHub Repo of benchmark of the models : https://github.com/EuclidStellar/Model-Bechmarks-for-stt

2. Documented the Complete Research on the Model Notion link of Documentation : https://www.notion.so/LLM-for-speech-recognisition-08eec4116c01495c94cd6ec612eafbac

[Ramkrishna-egov]

Weekly Goals

Week 1

• [x] Research and development over digit component
• [x] Look for libraries that can help us
• [x] Create UI components to handle voice commands and form submission.

Week 2

• [x] Research on methods to achieve voice integration on dummy UI
• [x] Try talk back feature
• [x] Semantic widget integration

Week 3

• [x] Flutter TTS and speech-to-text package integration
• [x] Add voice command functionality to fill form fields
• [x] Use DigitTextField from digit_components package
• [x] Create UI components to handle voice commands and form submission

Week 4

• [x] Implement form validation and snack bar notifications
• [x] Use the Reactive Text Form field in code along with form control
• [x] Transition DigitTextField to DigitTextFormField
• [x] Add more Digit Components enabled with voice commands such as DigitReactiveDropDown

Week 5

• [x] First, implement it on the Flutter Material 3 widget and then transition it to DigitReactiveDropDown
• [x] Add voice command in Digit Component (DigitCheckBoxTile) (reactive checkbox)
• [x] Use "agreed" as a keyword to mark the tick in the checkbox

Week 6

• [x] Research and Choose the Best LLM Model Compare Different LLM Models:

  • Evaluate models like OpenAI's GPT-4, GPT-3.5, and other alternatives.

    Consider Factors:

  • Look at accuracy, response time, cost, ease of integration, and available features.

    Select the Best Model:

  • Choose the model that best fits your needs based on the evaluation.

• [x] Documentation on the finalized approach.

  Week 7

• [x] Create an analysis of Research in a summarized manner

• [x] Make PPT to showcase the work to the technical council

• [x] Add a dummy apk of current version using speech_to_text in PPT

[EuclidStellar]

Model Benchmarking research repo of Google Colab : https://github.com/EuclidStellar/Model-Bechmarks-for-stt