spike(data storage): Indexed Twitter data storage

[Luka-Loncar]

Problem Statement

Validators are currently indexing Twitter data periodically, but have nowhere to store this valuable information. This indexed data, while not generated by external user queries, is crucial for maintaining an active and responsive network. We need a solution to efficiently store and manage this data for future use and analysis.

There is a potential security risk on how we are currently exposing validator endpoints this is covered here: https://github.com/masa-finance/masa-bittensor/issues/243

We have been pushing data we have scraped to HF with an upload script: https://github.com/masa-finance/awesome-masa/tree/feature/huggingface/hugging_face

Considerations

• [ ] Race condition: each miner responds with unique times
• [ ] Unique tweets: a miner with unique tweets might have more value
• [ ] Do we want to ultimately only score miners if they submit unique tweets or for the first 3 tweets do we accept those? Some form of consensus here?
• [ ] Tweet metadata and the information that is associated with the tweets, trend analysis, and market specific data on where the information is being requested.
• [ ] tweetMetadata has an edge to a tweet object one tweet can have many relationships to tweetMetadata
• [ ] Batch all transactions and incoming payloads into a single file for each data stream; events, tweets
• [ ] Log transaction failure and measure data loss
• [ ] Authentication out of scope in v1

[Luka-Loncar]

Changing into Spike ticket - we need follow-up tasks after it is done. @teslashibe @grantdfoster @hide-on-bush-x @5u6r054

[Luka-Loncar]

We need follow up ticket here in order to be able to close out this spike ticket:

@theMultitude @grantdfoster

[Luka-Loncar]

Linked to feature from this PR https://github.com/masa-finance/masa-protocol-api/pull/4