As a foundational part of the system, the Data Collection Module is responsible for continuously scanning the internet for unanswered questions, particularly focusing on questions posted on Twitter that are marked with a question mark “?”. The goal is to capture these questions in real-time and prepare them for processing and subsequent answering through an LLM (Language Learning Model).
Acceptance Criteria
[ ] Module can authenticate with Twitter API.
[ ] Module can perform continuous searches for tweets containing a '?' symbol.
[ ] Module filters and discards non-question tweets or spam.
[ ] Module prioritizes questions based on a predefined set of criteria (e.g., recency, engagement).
[ ] Module stores collected questions in a structured format for the LLM to process.
[ ] Module respects API rate limits and implements efficient querying to minimize costs.
[ ] Module includes error handling to manage search interruptions or API changes.
[ ] Collected data is periodically audited for quality and relevancy.
sequenceDiagram
participant TwitterAPI as Twitter API
participant DataModule as Data Collection Module
DataModule->>TwitterAPI: Authenticate
loop Search for questions
DataModule->>TwitterAPI: Request tweets with '?' symbol
TwitterAPI-->>DataModule: Stream of tweets
DataModule->>DataModule: Filter and store questions
end
Data Collection Module Specification
As a foundational part of the system, the Data Collection Module is responsible for continuously scanning the internet for unanswered questions, particularly focusing on questions posted on Twitter that are marked with a question mark “?”. The goal is to capture these questions in real-time and prepare them for processing and subsequent answering through an LLM (Language Learning Model).
Acceptance Criteria