nolanjacobson commented 3 months ago

Implement Discord Message Scraping with Distributed Worker Model

Summary

We need to implement a feature that allows consumer nodes to request message data from a specific channel within a Discord server. This feature will utilize our distributed worker model, where the request is broadcast to all nodes, and the node that is a member of the targeted server will fulfill the request.

Details

The primary objective (P0) is to establish functionality for scraping messages from a specified channel. Future enhancements (post-P0) may include the ability to request data from an entire server rather than a specific channel.

Requirements:

Implement a distributed work model using the workers package.
Allow consumer nodes to send a request for channel data from a Discord server.
The request should be fulfilled by a node that is part of the target server.
Introduce an endpoint to list all available servers/guilds that can be queried, filtering nodes where discord_scraper is set to true.

Proposed Endpoint

GET /api/v1/discord/guilds/all: Returns a list of servers/guilds available for scraping by querying nodes with discord_scraper set to true.

Tasks

[ ] Define a new worker type for Discord scraping in the workers package.
[ ] Create a Pub/Sub topic for Discord scraping requests.
[ ] Implement a handler that processes scraping requests and distributes them to worker nodes.
[ ] Develop logic for worker nodes to determine if they can fulfill a request based on server membership.
[ ] Add an endpoint to list available servers/guilds for scraping.
[ ] Ensure proper error handling and logging for the scraping process.
[ ] Write unit tests to cover new functionality.
[ ] Update documentation to reflect the new feature and endpoint.

Out of Scope

Scraping data from an entire server (this will be considered for future iterations).
OAuth

Acceptance Criteria

A consumer node can request message data from a specific channel, and the request is fulfilled by an eligible worker node.
An endpoint is available to list servers/guilds that can be scraped, based on the nodes' capabilities.

Notes

This feature must comply with Discord's API rate limits and terms of service.
Ensure that all data handling follows privacy and security best practices.

nolanjacobson commented 3 months ago

@lacyg4 @giovaroma @teslashibe

Luka-Loncar commented 3 months ago

hey @nolanjacobson should this be in Distributed work epic?

nolanjacobson commented 3 months ago

removed the epic, my bad @Luka-Loncar

mudler commented 3 months ago

@nolanjacobson any news on this?

teslashibe commented 3 months ago

@nolanjacobson what is the status here matey?

nolanjacobson commented 3 months ago

@mudler @teslashibe - done once I push docs/loom video this morning sers.

nolanjacobson commented 3 months ago

docs are done - now just going to do a loom demo :)

mudler commented 2 months ago

@nolanjacobson where are docs? can you point me where those have been added? I guess we are good to close this one if docs are in place then.

mudler commented 2 months ago

moving this back in progress if you are still working on it @nolanjacobson

nolanjacobson commented 2 months ago

Hi - yes the docs are here: @mudler - https://developers.masa.ai/docs/masa-oracle/oracle-node/discord-data, https://developers.masa.ai/docs/masa-oracle/worker-node/discord-worker

masa-finance / masa-oracle