Implement Telegram Message Scraping

[nolanjacobson]

Summary: The purpose of this feature request is to implement functionality for scraping messages from specified Telegram channels.

Motivation: Scraping messages from Telegram channels is crucial for client integrations. This feature will enable us to collect valuable data that can be used to run sentiment analysis.

Detailed Explanation: The feature should leverage the Telegram Bot API to retrieve messages from a given public Telegram channel. It must comply with Telegram's API usage policies and rate limits to ensure uninterrupted service and legal compliance.

Proposed Solution:

• Utilize the Telegram Bot API, ensuring the bot is added to the channel with the necessary permissions to read messages.
• Design a rate limit handling strategy to avoid exceeding Telegram's API request thresholds.
• Securely store the scraped messages in a structured format within a database or file system, facilitating easy retrieval for analysis or processing.

Considerations:

• Ensure the bot has appropriate permissions set by the channel administrators to access the messages.
• Develop comprehensive error handling to cope with potential API changes, network issues, or unexpected service disruptions.
• Prioritize user privacy and adhere to data protection laws, being transparent about the data collection process and its purpose to all channel participants.

Related issues:

• https://github.com/masa-finance/masa-oracle/issues/358

• 357

Acceptance Criteria:

• [ ] The scraping feature must consistently and accurately collect both new and historical messages from the specified Telegram channel.
• [ ] The implementation must not violate Telegram's rate limiting constraints or API usage policies.
• [ ] The data storage solution should be secure, organized, and allow for efficient data retrieval for subsequent use cases.
• [ ] compliance when scraping messages on telegram (TBD)
• [ ] We have a way for a user to invite the bot in a Telegram channel to scrape
• [ ] users should be able to do search queries and sentiment analysis.

[nolanjacobson]

Feature Request

Summary: The purpose of this feature request is to implement functionality for scraping messages from specified Telegram channels.

Motivation: Scraping messages from Telegram channels is crucial for client integrations. This feature will enable us to collect valuable data that can be used to run sentiment analysis.

Detailed Explanation: The feature should leverage the Telegram Bot API to retrieve messages from a given public Telegram channel. It must comply with Telegram's API usage policies and rate limits to ensure uninterrupted service and legal compliance.

Proposed Solution:

• Utilize the Telegram Bot API, ensuring the bot is added to the channel with the necessary permissions to read messages.
• Design a rate limit handling strategy to avoid exceeding Telegram's API request thresholds.
• Securely store the scraped messages in a structured format within a database or file system, facilitating easy retrieval for analysis or processing.

Considerations:

• Ensure the bot has appropriate permissions set by the channel administrators to access the messages.
• Develop comprehensive error handling to cope with potential API changes, network issues, or unexpected service disruptions.
• Prioritize user privacy and adhere to data protection laws, being transparent about the data collection process and its purpose to all channel participants.

Acceptance Criteria:

• The scraping feature must consistently and accurately collect both new and historical messages from the specified Telegram channel.
• The implementation must not violate Telegram's rate limiting constraints or API usage policies.
• The data storage solution should be secure, organized, and allow for efficient data retrieval for subsequent use cases.

[nolanjacobson]

@lacyg4 @giovaroma @teslashibe

[nolanjacobson]

Sentiment Analsyis API

[giovaroma]

Action items after reviewing with Nolan and Lacy:

[ ] @nolanjacobson double on compliance when scraping messages on telegram. [ ] TBD how user will invite the bot to be added to Telegram channel. [ ] Add users able to do search queries and sentiment analysis.

[H34D]

@nolanjacobson can you please create follow up tickets?

[mudler]

This is blocked by #357

[mudler]

this was already discussed , so I am removing the triage label, and the card is now split in several tickets (the first in the row is #357)