We want to have all the scrapers that are available in the masa-oracle repository as part of jobs available to execute in the tee-worker enclave. This have to take into account of configuring the scraper required settings (such as login credentials, where to store cookies, and so on so forth)
Seems the most battle-tested and used is basically twitter. The web is pretty unsophisticated code which needs improvement, discord looks functional but reddit and telegram are not supported for now.
Acceptance criteria
[ ] We have migrated the twitter scraping code to the tee-worker repository
[x] We have migrated the web scraping code to the tee-worker repository
Is your feature request related to a problem? Please describe. As we have PoC of a tee-worker that runs in an enclave, we want to enhance the scrapers available in the tee-worker ( https://github.com/masa-finance/tee-worker/tree/wip/internal/jobs )
Describe the solution you'd like
We want to have all the scrapers that are available in the masa-oracle repository as part of jobs available to execute in the tee-worker enclave. This have to take into account of configuring the scraper required settings (such as login credentials, where to store cookies, and so on so forth)
Describe alternatives you've considered N/A
Additional context
https://github.com/masa-finance/product-issues/issues/87
Seems the most battle-tested and used is basically twitter. The web is pretty unsophisticated code which needs improvement, discord looks functional but reddit and telegram are not supported for now.
Acceptance criteria