This app allows you to convert data into an HSDS-compliant datapackage.
The Human Services Data Specification (HSDS) is a data model that describes health and human services.
This transformer currently transforms data into HSDS version 1.1.
Lots of people have health and human services data, but it's all in different structures and formats. Many groups want to share their data, or use data that's shared with them, but the lack of standardized data or an easy way to standardize the data presents a major barrier.
Open Referral provides a Human Services Data Specification that groups can use to share data via a common format. The HSDS Transformer tool (this project) enables a group or person to transform data into an HSDS-compliant data package, so that it can then be combined with other data or used in any number of applications.
The real-world case that's motivating this project is Illinois Legal Aid Online (ILAO), an organization that helps individuals in Illinois find legal help or resources online. ILAO has a resource directory of community services data, and they'd like to update this data to be formatted in compliance with HSDS version 1.1. They also are looking to incorporate other data sets in disparate formats and structures into their own database in order to better serve Illinois residents with more extensive and more accurate information.
This project uses ILAO's resource database in the form of three CSVs as the test case. The data was pulled in January 2019, with some records not having been updated since 2017, and therefore some of the data may not be accurate or up-to-date. The data is located in spec/fixtures/input
.
First, double check whether your data is already HSDS-compliant with the OpenReferral Validator.
If it is, voila!
If it's not, proceed.
Note: Your source data must live in a single directory and be stored as CSVs.
Make sure you have bundler
installed, and then run:
gem install hsds_transformer
or add hsds_transformer
to your Gemfile.
Then you will be able to use the Transformer as documented below in "Transforming using the Ruby library". Instead of setting the ROOT_PATH
env as the HSDS Transformer project directory, however, you will need to set this env variable as the path to the directory your source data is stored in.
cd
into the root of the hsds_transformer directory..env
. Copy the contents of .env.example
into the new .env
file and update Users/your_user/dev/hsds_transformer
to be the correct path to this directory on your local environment (you can run pwd
in terminal to find this out).bundle install
If you're familiar with Ruby and you want to use this tool in the command line, you can open an IRB shell and require the library, and begin transforming data:
ROOT_PATH
env variable to in step 3 above).irb
(or pry
- up to you!)require "./lib/hsds_transformer"
HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")
tmp
directory for your newly created HSDS files!You can also pass two additional arguments to the .run
command: include_custom
and zip_output
. The output is by default not zipped, but if you want it to be, you can pass true
as the value of this argument:
HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/", zip_output: true)
If your input data includes non-HSDS fields you want to see in the output files as well, you can pass true
for include_custom
.
HsdsTransformer::Runner.run(input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/", include_custom: true)
If you don't want to use this as a Ruby library, you can use it as an HTTP API. Start the API from the root of the project directory:
rackup -p 4567
Make a POST request with params: input_path, mapping, include_custom. Each of thse should contain a path to a CSV file and a mapping file (for mapping).
E.g.
curl -X POST -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform
The response will be a zip file of the transformed data. You can also pass add -F "include_custom=true"
if your input data has custom non-HSDS columns you wish to include.
The API then streams a zip file back with the properly transformed data. The zip is also saved locally on the API server (maybe your local env) at data.zip
in the root directory
Before anything else, make sure you have docker installed and running.
First, build the image locally:
docker build . -t transformer_api
Now, run it:
docker run -p 4567:4567 transformer_api
You should now be able to interact with the API at http://localhost:4567
.
The BaseTransformer maps data from the input directory to compliant HSDS datapackage and CSVs using the mapping.yaml, and it requires a pretty one-to-one and straightforward mapping. You may need additional cleanup, parsing, or mapping, such as parsing out schedule text. If so, you can create a custom transformer and specify it when running the script or using the API. Check out the lib/hsds_transformer/custom
directory for examples.
lib/hsds_transformer/custom
following the naming conventions already there.HsdsTransformer::Runner
class.HsdsTransformer::Runner.run(custom_transformer: "Open211MiamiTransformer", input_dir: "/path/to/input/", mapping: "/path/to/mapping.yaml", output_dir: "/path/to/output/")
or when making a request to the API:
curl -X POST -F "custom_transformer=Open211MiamiTransformer" -F "input_path=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/input" -F "mapping=/Users/gwalchmai/Dev/hsds_transformer/spec/fixtures/mapping.yaml" http://localhost:4567/transform
You can find examples of data and mappings in the examples
directory.
Open Referral Playground App
Open Referral Drupal
Open Referral Wordpress
Open Referral Laravel Services
Open Referral Validator
Open Referral datapackage code
Open Referral Sample Data
Open Referral Gem
Open Referral OpenAPI Specification