ArshKA / LinkedIn-Job-Scraper

LinkedIn scraper to retrieve and store a live stream of job postings
93 stars 28 forks source link

Fixed missing industry and skill name, other minor improvements #2

Closed AndhikaWB closed 11 months ago

AndhikaWB commented 11 months ago

I also wrote the revisions in create_db.py and database_scripts.py so job_skills and job_industries tables will have a new column containing the full industry/skill name, though I left them commented since I'm not sure if that's what you want.

ArshKA commented 11 months ago

Checked it out, everything looks good!

AndhikaWB commented 11 months ago

@ArshKA the resuming part is quite important, when debugging what's wrong with industry/skill name, I opened the db during sleep period and stopped fixing/modifying the db when it resume.

I noticed that you removed the search_retriever.py resume indicator on the latest commit, it may be useful to prevent conflict write since I'm not sure how SQLite handle concurrent writing.

ArshKA commented 11 months ago

Just re-added it