search

PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

Failed to scrape jobs for MonsterScraperUKEng #138

Closed benb0jangles closed 2 years ago

benb0jangles commented 3 years ago

I'm getting an error: Terminal Command: funnel load -s settings.yaml [2021-03-21 01:21:25,777] [INFO] JobFunnel: Scraping local providers with: ['IndeedScraperUKEng', 'MonsterScraperUKEng'] [2021-03-21 01:21:26,638] [ERROR] JobFunnel: Failed to scrape jobs for IndeedScraperUKEng [2021-03-21 01:21:26,639] [INFO] MonsterScraperUKEng: No get() or set() will be done for Job attrs: ['REMOTENESS'] [2021-03-21 01:21:28,454] [ERROR] JobFunnel: Failed to scrape jobs for MonsterScraperUKEng [2021-03-21 01:21:28,454] [INFO] JobFunnel: Completed all scraping, found 0 new jobs. [2021-03-21 01:21:28,475] [INFO] JobFunnel: Done. View your current jobs in demo_job_search_results/demo_search.csv

Using: Mac OS 10.12.6

PaulMcInnis commented 3 years ago

Thanks for opening an issue, can you confirm that you are able to obtain results (non advertisement results) for the search you are performing on the monster website?

Additionally, if you are open to sharing your search URLs from logs it would be very helpful to identify what the issue is.

We current have CI for the UK Indeed scraper but it only performs a basic search.

PaulMcInnis commented 3 years ago

I think we have just merged a PR that resolves this, feel free to try again on current master

benb0jangles commented 3 years ago

pip uninstall JobFunnel pip install git+https://github.com/PaulMcInnis/JobFunnel.git funnel load -s my_settings.yaml

[2021-04-02 10:24:34,081] [INFO] JobFunnel: Scraping local providers with: ['IndeedScraperUKEng', 'MonsterScraperUKEng'] [2021-04-02 10:24:35,154] [INFO] IndeedScraperUKEng: Found 3 pages of search results for query=HTML [2021-04-02 10:24:36,456] [INFO] IndeedScraperUKEng: Scraped 126 job listings from search results pages 100%|##################################################################################################| 126/126 [03:09<00:00, 1.51s/it] [2021-04-02 10:27:46,274] [INFO] MonsterScraperUKEng: No get() or set() will be done for Job attrs: ['REMOTENESS'] [2021-04-02 10:27:48,070] [ERROR] JobFunnel: Failed to scrape jobs for MonsterScraperUKEng Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/bin/funnel", line 11, in load_entry_point('JobFunnel==3.0.2', 'console_scripts', 'funnel')() File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jobfunnel/main.py", line 28, in main job_funnel.run() File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jobfunnel/backend/jobfunnel.py", line 114, in run scraped_jobs_dict = self.scrape() File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jobfunnel/backend/jobfunnel.py", line 244, in scrape self._check_for_inter_scraper_validity( File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jobfunnel/backend/jobfunnel.py", line 220, in _check_for_inter_scraper_validity raise ValueError( ValueError: Inter-scraper key-id duplicate! ead06c46e9940d3d

Mac OS 10.12.6

PaulMcInnis commented 2 years ago

Duplicate #148