링크드인 크롤링 코드화

[S0rrow]

Sign in/out

• 채용공고 접근 URL이 로그인 여부에 따라서 달라짐

• 세션(로그인)이 생성되었을 때, 얻을 수 있는 정보가 달라짐

  • 기술 스택
  • 회사규모

  Selenium > Chrome Driver error

Exception managing chrome: error sending request for url (https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json)

when this problem arise, I think we have to revert with two ways

1. Debug this problem and Solve it
2. Don't use Selenium.

Solving this Problem, Just Server that executing selenium Con't connect network, So this is the reason that this problem arise

Sign in/out

• 채용공고 접근 URL이 로그인 여부에 따라서 달라짐

• 세션(로그인)이 생성되었을 때, 얻을 수 있는 정보가 달라짐

  • 기술 스택
  • 회사규모

Selenium > Chrome Driver error

Exception managing chrome: error sending request for url (https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json)

when this problem arise, I think we have to revert with two ways

1. Debug this problem and Solve it
2. Don't use Selenium.

Solving this Problem, Just Server that executing selenium Con't connect network, So this is the reason that this problem arise
I think that It's Okey We'll keep going

Finally data schema is bellow

{'title': 'English Teacher',
 'wr': 'On-site Full-time',
 'etc': 'Skills: English Teaching, Teaching English as a Foreign Language, +5 more;Applicant review time is typically 4 days;Learn more;See how you compare to over 100 other applicants. Try Premium for ₩0',
 'tech_stacks': 'Business English;English Teaching;English as a Second Language (ESL);Language Teaching;Teaching;Teaching English as a Foreign Language;Teaching English as a Second Language',
 'geo': 'Seoul, South Korea ',
 'apt': 100,
 'content': "We are seeking a highly motivated and dedicated English Language Teacher who is eager to embark on an exciting career abroad. The ideal candidate will possess a genuine passion for teaching English and the ability to inspire students to reach their full potential.\n\nIf you are passionate about teaching English and eager to embark on a new and exciting career abroad, we encourage you to apply for this position.\n\nWe offer competitive compensation packages, professional development opportunities, and a supportive work environment that values diversity, inclusivity, and excellence in education.\n\nWe also encourage freshers to apply.\n\nRequirements:\nHolds a 4-year bachelor's degree\nHas a clean national criminal record\nPass a health and drug test\n\nDetails:\nStarting Date: Early September., 2024\nTeaching Date: Mon thru Fri\nSchool Hour: 8:30 Am ~ 5:30 Pm\nStudent Level: Kindy + Elementary\nSalary: 2.4 Million Won ($1,700+) (Negotiable)\nVacation days : 11 days plus red days.\nHousing: One room furnished or Housing Allowance\nBenefit: Airfare, Severance Pay, Health Insurance, National Pension\n\nWe are committed to supporting all applicants throughout the application process and providing help as needed.\n\nESL Teaching is proud to be an Equal Opportunity Employer. We do not discriminate based on race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics.",
 'get_date': '2024-08-02_16:20:1722583221',
 'company': 'ESL Teaching Jobs',
 'jf': 'Education and Training',
 'industry': 'Human Resources Services',
 'reqr': 'associate degree;60 month',
 'dateposted': '2024-08-01T22:57:14.000Z'}

Process

require Job_id

Selenium

1. data extract in linkedin(sign in)
2. data extract in linkedin(sign out)

RESTful

1.data extrack in linkedin(sign out)

[Abel3005]

Issue: LinkedIN puzzle arise -> limit apttempts -> alert admin user

[S0rrow]

linkedin을 guest 계정으로 크롤링하는 jobspy라는 이름의 외부 패키지를 적용해 구현 완료.