crawling 리팩토링

[heehehe]

• [x] driver 함수별 인스턴스 생성
• [x] pycharm 기반 crawling.py reformat
• [x] jobplanet - get_recruit_content_info / postprocess로 분리
• [x] method별로 결과 임시 저장되도록
  • [x] get_url_list: {사이트명}.url_list.json
  • [x] get_recruit_content_info: {사이트명}.content_info.json
  • [x] postprocess: {사이트명}.result.jsonl
• [x] 각 method별로 실행시킬 수 있도록
  • 크롤링하는 부분이 오래걸려서 단계별로 결과 임시저장해둘 수 있도록
• [x] (get_url_list / get_recruit_content_info) 중단된 경우 중단된 이후부터도 진행될 수 있도록

---

• 현재 method 종류
  • get_url_list: 채용공고 url 목록 불러오기 (직무 카테고리별로 분류해서 저장)
  • get_recruit_content_info: 채용공고 url별 page source 불러오기
  • postprocess: 채용공고 url page source별 정보 추출

[heehehe]

jobplanet 실행 오류 🥲

$ python3 script/crawling.py -s jobplanet -d ./data -m get_url_list
INFO:root:[INFO] Set instance of crawling
INFO:root:[INFO] Get recruit content info
Traceback (most recent call last):
  File "script/crawling.py", line 795, in <module>
    main(args)
  File "script/crawling.py", line 782, in main
    method()
  File "script/crawling.py", line 398, in get_url_list
    job_find_window(job_filter)
  File "script/crawling.py", line 360, in job_find_window
    driver.find_element(By.CLASS_NAME, "jply_btn_sm.inner_text.jf_b2").click()
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 741, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
    self.error_handler.check_response(response)
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: ; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception

[dbsgh3344]

jobplanet 실행 오류 🥲

$ python3 script/crawling.py -s jobplanet -d ./data -m get_url_list
INFO:root:[INFO] Set instance of crawling
INFO:root:[INFO] Get recruit content info
Traceback (most recent call last):
  File "script/crawling.py", line 795, in <module>
    main(args)
  File "script/crawling.py", line 782, in main
    method()
  File "script/crawling.py", line 398, in get_url_list
    job_find_window(job_filter)
  File "script/crawling.py", line 360, in job_find_window
    driver.find_element(By.CLASS_NAME, "jply_btn_sm.inner_text.jf_b2").click()
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 741, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
    self.error_handler.check_response(response)
  File "/Users/user/Library/Python/3.8/lib/python/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: ; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception

같은 이슈인진 확실하지 않지만 제가 실행했을 때 발생하던 오류 수정했습니다! 원인 : popup창이 떠서 selenium에서 element들을 캐치하지 못했던 이슈 해결 : 웹사이트 크롤링 시작 시 뜨는 popup창을 먼저 제거하고 크롤링 시작하도록 수정