ODN Backend
A REST API for the Open Data Network. Available at api.opendatanetwork.com.
Documentation
API documentation is available on Apiary.
Development
The ODN backend is built using Node.js.
After cloning the repository and downloading node,
simply run npm install from within the project directory
to install all dependencies.
The ODN backend uses syntactic features of ES6 that are only available in
recent versions of Node, so it may fail to run with older versions.
Make sure that you use the Node version specified in package.json (currently 6.2.2).
You can check your node version using node --version.
Memcached
The ODN backend uses memcached to cache responses from the Socrata backend and store map sessions.
Installation with Homebrew
Memcached is available over homebrew.
To install it, use brew install memcached.
There are many options for running memcached including the most direct: /usr/local/opt/memcached/bin/memcached
To see them all, use brew info memcached
Flushing the Cache
Run ./flush-memcache.sh to flush the development cache.
Server
Use npm run server to start the development
server at localhost:3001.
It will automatically restart when the source is changed.
Start a local memcached server to get maps to work.
Tests
REST API tests are written using Chakram
and run with Mocha.
They are available in the test directory.
Run tests using npm test.
Sometimes, running tests will trigger a webserver restart which
will then cause many tests to fail.
If this happens, start the server using node app.js.
Test Coverage
Test coverage reports for master are available on
Code Climate.
To view test coverage while in development: start the server, run the tests, and then go to localhost:3001/coverage to see the reports.
Deployment
The ODN backend is hosted on Heroku. There are two environments: production and staging.
To deploy to an environment, push to the corresponding branch on github.
For production, push to master and for staging, push to staging.
There should be no need to manually deploy to Heroku,
but if you do, make sure that master stays in sync with what is
on Heroku.
Check LastPass for Heroku access.
MemJS
The ODN uses the Heroku MemJS add-on
for memcached.
The cache is flushed on each deploy to avoid synchronization conflicts.
Make sure to the NODE_ENV environmental variable to production
on every Heroku dyno so that the cache knows to flush itself.
Integration Tests
Integration tests are run to check each deployment using Travis CI. These tests must pass for the deployment to succeed.
Running Tests before Committing
Since all tests must pass for a deployment to succeed,
it is a good idea to run unit tests locally before pushing to GitHub.
The pre-commit.sh script will make sure that all unit tests succeed before
every commit. To install it, run:
ln -s -f ../../pre-commit.sh .git/hooks/pre-commitLoad Tests
The load-test.sh script can be used to put load on the service.
It randomly generates thousands of valid URLs and then uses
Siege to test them.
First, install Siege. On a mac, just use brew install siege.
Then, run the load-test.sh script with a domain.
For example, test against localhost, use
./load-test.sh localhost:3001This will start a small load test. To change the parameters of the test,
edit load-test.sh. Please do not use this script on production.
Also, remember that the main bottleneck of the ODN API is the Socrata platform,
and that a load test of the ODN is a load test of the platform.
Don't take it down.
Logging
All Heroku logs are forwarded to Sumo Logic.
This includes basic information for each request,
as well as detailed stack traces for all exceptions and 500s.
Search with _source=odn_api_heroku to see all of the logs.
Use the Sumo dashboard for an overview of how the service is performing. For access to this dashboard, use the Socrata Sumo account.
Monitoring
There are several Pingdom alerts that monitor the service:
Pingdom will alert Chris, Deep, Lane, and Tosh if the API is down. Once apps are built around the ODN API, alerts will be sent to on call.
Adding Data to the ODN
See /data for information on adding data to the ODN.