version history - Githubissues

seoyeong200 / Book-data-Pipeline

책 줄거리를 기반으로 비슷한 책을 추천해주는 서비스를 위한 데이터 파이프라인입니다. 도커 환경에서 파이프라인을 구성하여 스파크를 사용해 배치 데이터를 처리합니다.

2 stars 0 forks source link

version history #5

Closed seoyeong200 closed 5 months ago

seoyeong200 commented 6 months ago

related to scraping

aws default python version = 3.9.16
local python version = 3.9.6
selenium = 4.16.0
- pip3 install selenium -t python/lib/python3.9/site-packages
  - 4.10^ version : executable_path was removed
  - have to use service arg instead ref
aws lambda, layer runtime = 3.9

# download chromedriver
curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
# download headless-chromium
curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-41/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip