Handle program differently in case of an exception

PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.

MIT License

1.78k stars 210 forks source link

Handle program differently in case of an exception #120

Closed markkvdb closed 2 years ago

markkvdb commented 3 years ago

Is your feature request related to a problem? Please describe.

JobFunnel crashes when an exception occurs during scraping because a page cannot be reached due to a CAPTCHA or network error. However, sometimes hundreds of job listings have been scraped so far.

Describe the solution you'd like

Save job listings that have been scraped so far instead of exiting the program with an exception. This could also be implemented for a KeyboardInterrupt.

PaulMcInnis commented 3 years ago

:+1: We should definitely have better exception handling, Currently we capture a limited number of issues.

thebigG commented 3 years ago

Just wanted to add a quick note:this will also fix the issues we are having in the CI pipeline right now. Indeed has been doing this a lot with CAPTCHA lately.

PaulMcInnis commented 2 years ago

closing due to inactivity