MVP-ready website scraping

mudler commented 2 weeks ago

Currently, the oracle can scrape websites and collect a sentiment analysis result automatically (from LLMs) #166 #163 #167 #187.

We want to make sure this feature is mvp-ready before launching testnet, and asses the quality of it before release.

It seems the /data/web endpoint is buggy - see the notes below.

Acceptance item

[ ] Make sure that when depth is set to 0, it defaults actually to 1.
[ ] QA
[ ] Identify bugs and open follow-ups if needed, or if chores, fix the bugs found along the way

By default, depth is set to 0, and the scraping request never returns (depth=0 seems to be invalid)

mudler commented 1 week ago

@nolanjacobson are you currently working on this?

nolanjacobson commented 1 week ago

@mudler - yes

H34D commented 1 week ago

@nolanjacobson i assigned you to this task now

mudler commented 1 week ago

@nolanjacobson can you please reference the issues when opening up PRs? thank you!

@nolanjacobson is left now QA and documentation, right?

nolanjacobson commented 1 week ago

@mudler - correct.