Web-crawler

Web crawler made with Python.
For both static and dynamic crawling.

Virtual environment setup

Create venv

py -m venv env

Run venv

.\env\Scripts\activate.ps1

If there is unrestricted access problems,

Set-ExecutionPolicy Unrestricted -Scope Process

Used libraries

• requests
• beautifulsoup4
• selenium
• streamlit
• pyperclip

Installing libraries

py -m pip install [library-name]

Included files

• Static
  Includes code that statically crawls stock name, prices and change from Naver and Google.
  Includes code that statically crawls current exchange rate from Naver.
  Simple GUI is made with Streamlit.

• Dynamic
  Uses Selenium for to dynamically crawl Chrome.
  If necessary, download Chromedriver that supports your Chrome version. https://chromedriver.chromium.org/downloads
  Please note that using Chromedriver for Selenium has been deprecated, so the step above is no longer necessary.

Install Docker, then run

docker run -p 4444:4444 selenium/standalone-chrome

Static UI local development

py -m streamlit run static/gui.py

The UI should be on http://localhost:8501/