Scrapy spider to crawl and fetch information about real estate information
This project uses pipenv for package/dependency and virtualenv management, to learn more about how to use pipenv
Requirements:
brew install pipenv
pipenv --two
pipenv lock
pipenv install
Redis cluster for local dev (Celery):
brew install redis
brew services start redis
Postgres cluster for local dev (Backend database):
brew install postgres
brew services start postgres
run the following command from root directory
cd services/craigslist
scrapy crawl craigbot_all -o craigslist_result.csv
cd services
export POSTGRES_DB_URI="postgres://postgres@localhost:5432/helios"; python bots.py
export FLASK_ENV=development
export POSTGRES_DB_URI="postgres://postgres@localhost:5432/helios"
FLASK_APP=app.py flask run --debugger
# to trigger the bot, run
curl http://localhost:5000
export POSTGRES_DB_URI="postgres://postgres@localhost:5432/helios"
python app.py
# connect to database
psql postgres -U postgres
\c helios
# execute create_initial_schema.sql script
# To test postgres database CRUD operations
cd services/postgres
export POSTGRES_DB_URI="postgres://postgres@localhost:5432/helios"
python postgres.py
helios system is set up to be in continous deployment to Heroku platform at https://dashboard.heroku.com/apps/beast-helios tracking master
branch
# deploy to Heroku, just check in/ merge into master branch
# restart app
heroku restart
# set/unset environment variables
heroku config:set <env_var>
heroku config:unset <env_var>
# check logs
heroku logs --tail
Note Commits are associated with GPG signing key