Environmental Networked Sensing (ENTS) backend is part of the ENTS hardware and software ecosystem for outdoor sensor networks. It's an open source data ingestion and visualization service that parses data from the hardware nodes and presents it in an easy-to-use web interface. Users can dynamically generate interactive plots, live monitor their sensors, or download data for offline processing. Our live version, DirtViz, is available at https://dirtviz.jlab.ucsc.edu/.
Dependency |
---|
Python |
Node |
Docker |
A local version of the ENTS backend can be started using docker-compose.yml
. This will build the local images and start the required services in the background, including the database.
docker compose up --build -d
At this point the backend is accessible at http://localhost:3000/, but will likely show a blank page in your web browser and throw an error. This is due to the database being empty, therefore there is no data to display.
Alembic is used to manage database migrations. Alembic is a python package and acts as a complement to sqlalchemy which is used to query the database. The following will upgrade the database to the most recent version and should be done any time the database schema changes.
NOTE: It is recommended that a virtual environment is setup and ALL dependencies are installed via
pip install -r requirements.txt
. If you are unsure what this means, read this.
A migration script is provided in this repository that abstracts the migration process.
# To check out usage run
./migrate.sh -h
Real life example data is provided and can be imported with the following. The data was collected at UC Santa Cruz by jlab.
python ./import_example_data.py
Now some graphs should appear on the website and look like the following.
ENTS backend's client is built with React and is located under the frontend
folder. ENTS backend's API is built with Flask and located under the backend
folder. Check out the frontend readme and the backend readme for more information.
To compile for with development configurations (eg. hot reload and logs), in docker-compose.yml
set target: development
. To test containers in with production configuration use target: production
.
For bugs refer to bug_template.md. For other issues, create an new issue in this repository.
For documentation on the backend, refer to backend readme
To start contributing to the ENTS backend, please read CONTRIBUTING.md
Here's a list of good first issues to get yourself familiar with the ENTS backend. Comment in the issue to pick it up, and feel free to ask any questions!
This projects makes use of alembic to handle database migrations and flask-migrate as an extension to make alembic operations avaliable through the Flask cli. It is recommended to have a understanding of the package first before attempting to modify the database schema. Due to the way that alembic handles package imports, the config file needs to be specified while running from the root project folder. For example the following will autogenerate new migrations from the latest revision of the database.
The script migrate.sh takes in a "-m \<msg>" for generating a new migration and by itself runs "alembic upgrade head".
NOTE: Autogeneration of migrations requires a running version of the database. Refer above to see how to create and connect to a local version of the database
./migrate.sh -m "migration message here"
Sometimes the database breaks and causes errors. Usually deleting the docker volume postgresqldata
causing the database to be recreated fixes the issue. The following does exactly that and reapplies the migrations to the cleaned database.
docker compose down
docker volume rm dirtviz_postgresqldata
docker compose up --build -d
./migrate.sh -u
For this error, it either means that you've deleted a revision corresponding to the id located in ./backend/api/migrations/versions
or that if it's during the deployment process, the alembic version in the db (under the alembic version table) is mismatched. Double check to see if the revision history is the same for both deployment and locally.
There exists csv importers that can be used to populate the database. Python utilities currently exist to import RocketLogger and TEROS data. These are available as modules under the backend folder. More information on used can be found by running the modules with the --help
flag.
python -m backend.api.database.utils.import_cell_data