buzzn/core

buzzn/core is the central server-side application of buzzn. It contains the business logic, storage and API for our client/frontend applications.

buzzn/core
Table of contents
Application Architecture
Bin-Scripts
Useful rake tasks
Beekeeper import
- How to run it
How to deploy
How to set up a development environment
Misc admin info (may be outdated)
Further links & documentation

Application Architecture

See docs/application_architecture.md.

Bin-Scripts

bin/console - starts application with pry session
bin/server - starts puma server
bin/rake - runs rake with application bundler context
bin/rspec - runs rspec with the current bundler context
bin/reset_test_db - runs some rake tasks to reset test DB
bin/example_data - runs some rake tasks to setup and seed DB for running local server

you can add ./bin to your PATH to simplely use console, server, rake or rspec.

Useful rake tasks

Discovergy Credentials

To wire up some datasource of some registers with the discovergy API you need add the credentials in .env.local

DISCOVERGY_LOGIN= DISCOVERGY_PASSWORD=

Overview

task name	description	source
`rake db:empty`	Resets the database without dropping the DB. Useful to reset DB when it has open connections.	Buzzn custom
`rake db:seed:example_data`	Load an example localpool into the DB. It does not prepare/empty the DB, run `rake db:empty` beforehands if required.	Buzzn custom
`rake db:dump:transfer`	Transfer relevant data from buzzn_core_development to DATABASE_DUMP_NAME	Buzzn custom
`rake deploy:staging`	Deploy the application to staging.	Buzzn custom
`rake deploy:production`	Deploy the application to production and create a new release tag.	Buzzn custom
`rake heroku:update_db:staging`	Import the beekeeper dump locally and push it to staging.	Buzzn custom
`rake heroku:pull_db:staging`	Fetch a postgres dump from staging to DATABASE_DUMP_NAME, for use with `db:dump:transfer`
`rake heroku:update_db:production`	Import the beekeeper dump locally and push it to production.	Buzzn custom
`rake heroku:pull_db:production`	Fetch a postgres dump from production to DATABASE_DUMP_NAME, for use with `db:dump:transfer`

Common testing workflow - after checking out a remote branch

checkout the branch
run reset_test_db once to ensure DB is prepared
run rspec

Common testing workflow - to run one test file

run reset_test_db once to ensure DB is prepared
run rspec path/to/spec_file

Loading setup and example data ("seeds")

Our application has two kinds of data that we can pre-load ("seed"), setup and example data.

Setup data is essential for any deployment of our application to work. Among others, it loads the buzzn organization into the database (which has hard-coded references in the code as Organization.buzzn). It can be loaded by running rake db:seed:setup_data on the shell.

Example data contains an exemplary localpool, as well as contracts and their users, meters etc.. This data is completely optional and should not be loaded into the production system. We use it for demos or testing where we don't have real user-generated data. Load it into the database by running rake db:seed:example_data on the shell.

Use the list of example users to log in. Login is the email, the password always is Example123.

Note that creating a user account with the buzzn operator role (super user) is not part of the example data. Use the separate rake tasks db:seed:buzzn_operators or db:seed:buzzn_operator to create one.

Important: both rake tasks do not empty the database before running, so when there already is data in the system, there may be conflicts, causing the task to abort. So if you know what you are doing, run rake db:empty first, to completely delete all data from the database.

Beekeeper import

How to run it

make sure your .env.local has the correct, safe DEFAULT_ACCOUNT_PASSWORD
load setup and example data into the local core DB (we need the user accounts)
- rake db:empty db:seed:example_data
get the latest beekeeper MySQL dumps ("minipool_..." and "buzzndb_2017-11-17_TT.zip")
convert/load them into separate DBs in local postgres server
- rake beekeeper:sql:mysql2postgres FILE=/absolute/path/to/minipooldb_2017-11-17_TT.zip
- rake beekeeper:sql:mysql2postgres FILE=/absolute/path/to/buzzndb_2017-11-17_TT.zip
- make sure no table has any errors, (check summary table at the end)
convert and import that data into the core DB:
- rake beekeeper:import
optionally upload the local core DB into the Heroku DB
- heroku pg:reset
- heroku pg:push buzzn_development DATABASE_URL

How to deploy

We're running on Heroku, so you can deploy from Heroku's web interface if you want. To do it from the command line:

Maintenance Mode

to switch maintenance mode off/on execute the rake tasks on heroku for staging

`heroku run rake maintenance:on -a buzzn-core-staging`
`heroku run rake maintenance:on -a buzzn-core-staging`

or for production

`heroku run rake maintenance:on --remote production`
`heroku run rake maintenance:off --remote production`

this is important when the DB structure changes and the code and DB have mismatch for some time during deployment.

Deploy staging

Staging is deployed automatically for every green CI build on master. To do it manually: rake deploy:staging.

Deploy production

Run rake deploy:production. This pushes the current branch to Heroku and sets a release tag on the git repo.

Note on the previous, docker-based system and deployment: the Dockerfiles and related code have been removed, use this git tag to get them back. The same tag is set in the console app.

How to set up a development environment

Setup Ruby (using rbenv)

See https://github.com/sstephenson/rbenv#installation
The required version number is found in file .ruby-version

Install required software

imagemagick, postgresql, redis
for postgresql important on linux: 
  Edit the pg_hba.conf file /etc/postgresql/9.5/main/pg_hba.conf
  Change all authentication methods to "trust" and restart Server. 

use homebrew on a Mac

Set up the project

Grab the source git clone https://github.com/buzzn/core.git

Get dependencies

cd core
gem install bundler
bundle install

Create a .env.development

# Please keep the variables ordered alphabetically.
# This file is for overrides of the .env file when running in development.
ASSET_HOST=http://localhost:3000
DEFAULT_ACCOUNT_PASSWORD=Example123
DISPLAY_URL=http://localhost:2999
SESSION_INACTIVITY_TIMEOUT=31536000 # 1 yearPOSTGRES_HOST=localhost
POSTGRES_USER=postgres
POSTGRES_PASSWORD=secret
POSTGRES_BASE=postgres://$POSTGRES_USER@$POSTGRES_HOST
DATABASE_URL=$POSTGRES_BASE/buzzn_developmentREDIS_HOST=localhost
REDIS_CACHE_URL=redis://$REDIS_HOST:6379/0
REDIS_SIDEKIQ_URL=redis://$REDIS_HOST:6379/1

MAIL_BACKEND=stdout
DISCOVERGY_LOGIN=YOUR_DISCOVERGY_LOGIN_NAME
DISCOVERGY_PASSWORD=YOUR_DISCOVERGY_PASSWORD

Start postgres and and create a database and a user according to the entries of the previous .env.development

createdb -U postgres -h localhost buzzn_development
pg_restore -h localhost -d buzzn_development -U postgres ./some_data.dump # Import some data if you already have any

Start redis

Start server in development mode

bin/server

Reset and start test environment

RACK_ENV=test rake db:reset
guard

Set up rubocop

We run rubocop in CI to ensure consistent coding style and prevent error-prone syntax. Let your editor check the rules as well so you don't have to rely on CI.

Here's how to integrate it into editors:

for SublimeText: https://packagecontrol.io/packages/SublimeLinter-rubocop
for Atom: https://atom.io/packages/linter-rubocop
for Emacs: https://github.com/bbatsov/rubocop-emacs

To run rubocop from the CLI, simply type rubocop.

How to auto-corrrect rule offenses

Rubocop is smart enough to auto-corrrect most offenses by running rubocop -a

How to disable checking a rule locally

For a section of code:

# rubocop:disable RuleName
some = Exception(to, the, rule)
# rubocop:enable RuleName

How to run rubocop before every push automatically

Install the overcommit gem; it'll then use the .overcommit.yml file that's already checked in to run rubocop before pushing.

Misc admin info (may be outdated)

Start sidekiq message queue

redis-server
remark: probably necessary to reinit database (rake db:init) to let sidekiq run properly
bundle exec sidekiq -r ./config/sidekiq.rb

The (very useful) Sidekiq Admin interface currently isn't set up. See Sidekiq's documentation on standalone installation for how to enable it.

Stop sidekiq

rake sidekiq:kill

Find missing Indexes

lol_dba db:find_indexes

Run analysis of security vulnerabilities

brakeman

Further links & documentation

buzzn / core

readme