Studentbrad/scrape speed

PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.

MIT License

1.85k stars 215 forks source link

Studentbrad/scrape speed #28

Closed studentbrad closed 5 years ago

studentbrad commented 5 years ago

Implemented multithreading to improve scrape speed. Surprisingly this improves speed on Linux and Windows.

fixes #18

itSeez commented 5 years ago

why does pycharm need to call python run.py? can't it just do funnel -s settings.yaml?

studentbrad commented 5 years ago

@itSeez I tried that too. The reason why is because pycharm insists on calling python first. If you run the debugger, because pycharm uses a python script for debugging it must call python <debug script> <your script>. So instead I do python <debug script> run.py -s settings.yaml

studentbrad commented 5 years ago

I'm stupid. I didn't realize pycharm had a module option for exactly this reason. I removed run.py and updated the pycharm document.

itSeez commented 5 years ago

don't forget the version number :P