Build a data pipeline that scrapes problems from Reddit

KenjiPcx / IdeaGen

0 stars 0 forks source link

Open KenjiPcx opened 6 days ago

KenjiPcx commented 6 days ago

Similar to gummysearch

Steps:

Fetch reddit threads using snoowrap
Enrich data using an agent
- For ex: the openai agent can guess whats the root cause
Output schema:
- problem name,
- tags: money talks, solution request (the ones in the video)
- post url
Store it in a json file for now

MishalRizvi commented 6 days ago

Working on this.

MishalRizvi commented 4 days ago

@KenjiPcx removed node_modules

MishalRizvi commented 4 days ago

Command to fetch Reddits: node src/scripts/fetchRedditPosts.js Command to run scraping: node src/scripts/redditScraper.js