search

Hyperion101010 / naukri-scraper

A scraper using python requests library , that can scrape over 4,00,000 job profiles over naukri.com .Also it creates excel files automatically from scraped data facilitating data analysis .
GNU General Public License v3.0
13 stars 9 forks source link

Add timing paramters to the scraper cli. #5

Open Hyperion101010 opened 4 years ago

Hyperion101010 commented 4 years ago

We can add ability to customise the fetching speed of URL's to maximise the number of pages fetched in given time. It's a simple fix so open for new contributors.

ritwikchakraborty123 commented 4 years ago

Sir I want to contribute

Hyperion101010 commented 4 years ago

Sure @ritwikchakraborty123 you can first start with one issue and go for subsequent ones after finishing. First to start contributing, clone my repo on your local PC and fork it on your github instance.

I will suggest to read the scraper file and understand where you can make changes to add timings to the program. Lookup the implementation and let me know how you think we shall progress. For now i am thinking as a paramter name 't' to be supplied that tells the parser to wait to 't' time while scraping URL's. Let me know!, I am here to help :)

Steps for PR https://opensource.com/article/19/7/create-pull-request-github

-Shivam

Hyperion101010 commented 4 years ago

I have plenty of ideas we can work on so I can open plenty of issues during this Hacktoberfest. Let's make scraper more useful for fellow programmers.