Crawler

Web crawler for parsing data from http://140.112.166.97/power/fn2/dataq.aspx

Environment

Please run the following command first.

# Install tools
sudo ./install_packages.sh

Demo

You can try the following command:

# Basic parser and database initialization.
chmod +x ./demo.sh
./demo.sh

Crawl data in background

You can try the following command:

nohup python3 -u -m src.startCrawler > [LOG_FILENAME] &