search

Berea-CS-Courses / capstone-project-schweinsbergs

capstone-project-schweinsbergs created by GitHub Classroom
0 stars 0 forks source link

Addition of Web Crawler #11

Closed schweinsbergs closed 3 years ago

schweinsbergs commented 3 years ago

Webscraper needs to have some sort of feed-in from URLs to actually do its job.

I could hand-feed the webscraper, but that would be tedious and impractical.

A web crawler should be able to grab a bunch of website URLs for me and store them somewhere that my program can read from for the webscraper.

jazzij commented 3 years ago

Is this what you are planning to use the sitemap for?

jazzij commented 3 years ago

Please also link this to the code that you're committing. (In your commit description, add the commit number. For example: "#11 Added excel spreadsheet"