ESDR is an open source data repository intended for storing and retrieving time series environmental data. Basically, if the data has a timestamp and a value, ESDR can store it and provide ways to retrieve it quickly and securely. Data is stored in a custom, open source datastore which provides extremely fast data inserts and fetches, making for fast and responsive visualizations. The ESDR web site (esdr.cmucreatelab.org) provides a REST API interface to the datastore, making it easy to read/write data. Metadata is stored in a MySQL database.
ESDR is pronounced like the female name, Esther.
https://github.com/CMU-CREATE-Lab/esdr-explorer
If you're familiar with Xively's API, you'll see a lot of parallels with ESDR. Our intention is to use ESDR as the data repository not only for our own products and visualizations, but also for anyone else who wants a place to store data and have tools to easily visualize it.
First, some terminology: ESDR has clients, users, products, devices, feeds, channels, and tiles. Understanding how these entities relate will give a good understanding of how the data and metadata are structured and how the system works.
Again, the data samples themselves are all stored in the datastore. Data for the above entities is stored in a MySQL database. The big win with the datastore is that it works with billions of samples, doing time aggregation upon insert (and yet inserts are still fast), storing the data at a number of different summarization levels. Thus, it can return a summary of a year's worth (or more!) of data just as quickly as, say, five minutes worth. No summarization computation is required when fetching tiles, so visualizations remain responsive and fast at any zoom level.
We don't yet do spatiotemporal aggregation, but it's on the TODO list.
Please see the HOW TO document for more details on how to use ESDR.
Install the module dependencies:
npm install
Install the BodyTrack Datastore by doing the following
Fetch the BodyTrack Datastore. In your terminal window, set your working directory to the root of the ESDR repository and do the following:
git clone https://github.com/BodyTrack/datastore.git
Follow the BodyTrack Datastore's build and install instructions.
Install MySQL if necessary. ESDR was tested with and assumes MySQL 5.6/5.7/8.0 (there are known issues with 5.5).
Do the following to create the development MySQL database and user:
The syntax differs depending upon the version of MySQL you are using.
If you are using MySQL 5.7:
CREATE DATABASE IF NOT EXISTS esdr_dev;
GRANT ALL PRIVILEGES ON esdr_dev.* To 'esdr_dev'@'localhost' IDENTIFIED BY 'password';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_dev.* TO 'esdr_dev'@'localhost';
If you are using MySQL 8+:
CREATE DATABASE IF NOT EXISTS esdr_dev;
CREATE USER 'esdr_dev'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON esdr_dev.* TO 'esdr_dev'@'localhost';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_dev.* TO 'esdr_dev'@'localhost';
If you choose to change the password, make sure it matches the password in config-dev.json
.
If you want to be able to run the tests, do the following to create the test database and user:
The syntax differs depending upon the version of MySQL you are using.
If you are using MySQL 5.7:
CREATE DATABASE IF NOT EXISTS esdr_test;
GRANT ALL PRIVILEGES ON esdr_test.* To 'esdr_test'@'localhost' IDENTIFIED BY 'password';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_test.* TO 'esdr_test'@'localhost';
If you are using MySQL 8+:
CREATE DATABASE IF NOT EXISTS esdr_test;
CREATE USER 'esdr_test'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON esdr_test.* TO 'esdr_test'@'localhost';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_test.* TO 'esdr_test'@'localhost';
If you choose to change the password, make sure it matches the password in config-test.json
.
If running in production, do the following:
Create the config-prod.json
and mail-config-prod.json
files. Just copy from the other configs, but you need only include the parts that differ from config.js
.
Do the following to create the production database and user:
The syntax differs depending upon the version of MySQL you are using.
If you are using MySQL 5.7:
CREATE DATABASE IF NOT EXISTS esdr_prod;
GRANT ALL PRIVILEGES ON esdr_prod.* To 'esdr_prod'@'localhost' IDENTIFIED BY 'USE_A_GOOD_PASSWORD_HERE';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_prod.* TO 'esdr_prod'@'localhost';
If you are using MySQL 8+:
CREATE DATABASE IF NOT EXISTS esdr_prod;
CREATE USER 'esdr_prod'@'localhost' IDENTIFIED BY 'USE_A_GOOD_PASSWORD_HERE';
GRANT ALL PRIVILEGES ON esdr_prod.* TO 'esdr_prod'@'localhost';
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE ON esdr_prod.* TO 'esdr_prod'@'localhost';
Again, make sure the user and password you specify matches those in config-prod.json
.
Make sure the datastore data directory defined in the config file exists.
The NODE_ENV
environment variable may be specified when running, and must be one of dev
, development
, test
, prod
, or production
. Defaults to dev
if unspecified.
To run the server in development mode, do any of the following:
npm start
NODE_ENV=dev npm start
NODE_ENV=development npm start
To run the server in test mode, do:
NODE_ENV=test npm start
To run the server in production mode, do either of the following:
NODE_ENV=prod npm start
NODE_ENV=production npm start
To generate the CSS from the SCSS template, do:
npm run-script gen-css
To compile the handlebars templates, do:
npm run-script gen-handlebars