Open jacebenson opened 5 months ago
This crawler will read the sitemap, rss feeds, and manually crawl the pages of a site given, convert them to markdown and for each page, create a webpage record in postgres. Additionally, it will put this data into pinecone
This crawler will read the sitemap, rss feeds, and manually crawl the pages of a site given, convert them to markdown and for each page, create a webpage record in postgres. Additionally, it will put this data into pinecone