GeoSensorWebLab / data-transloader

Download and convert weather data for SensorThings API
GNU General Public License v3.0
3 stars 1 forks source link
etl

Arctic Sensor Web Data Transloader

Build Status

This tool downloads weather station data from multiple sources and uploads it to an OGC SensorThings API service. This makes the data available using an open and interoperable API.

Setup Instructions

This tool requires Ruby 2.6 or newer. Once Ruby is installed, you will need to install Bundler to install the library dependencies.

$ cd data-transloader
$ gem install bundler
$ bundle install

This tool should be runnable on Linux, MacOS, and Windows.

Docker

The tool can also be ran inside a Docker container, without having to install Ruby at all. See the Docker Instructions for details.

Usage

This tool is meant to be scriptable using cron, a scheduling daemon, to automate imports on a schedule. This is done by passing in command line arguments to the tool that handle the sensor metadata and observation download/upload.

The tool is separated into downloading/uploading metadata and downloading/uploading observation data. This is because metadata is not often updated from the source, and the metadata may need manual correction by the data transloader user. The observation data is constantly updated, so it needs to be efficient and avoid checking/creating entities for metadata every time it is run.

Currently supported weather station sources:

Instructions by Data Provider

Different source data providers have moderately different implementations and usage. Please see the detailed documentation for an explanation of the source-to-SensorThings API mappings.

Development Instructions

This tool can be modified and extended by editing its source files. In this environment, you must tell Bundler to also install the "test" group gems:

$ bundle install --with test

Authors

James Badger (james@jamesbadger.ca)

License

GNU General Public License version 3