Use case: ECHO data

[Frijol]

Hi! Not sure where you're collecting use cases, so feel free to move this, but wanted to share:

I've been working with the ECHO Exporter zip file from EPA's ECHO (Enforcement and Compliance History Online) portal (Data Downloads page), which contains:

• a giant csv of the data
• echo_exporter_columns_07242019.xlsx gives column descriptions and a key for the larger echo_data_exporter

I'd really like to pin it to IPFS through Qri (ideally on the Data Together node) but am running into a couple of challenges:

1. How to add both the csv of data and the xlsx containing the custom metadata to the Qri repo (it appears to accept only one file for upload
2. How to use the auto-pulling feature for something that comes in a zip file (there's no way to link directly to the CSV that's inside the zip file).

ECHO only shows you a 12-quarter history, so every quarter we lose a quarter off the back!

[ramfox]

Hey, this sounds super awesome and you can 100% do this using transforms.

You would need to create a transform script that: 1) downloads the zip file 2) unzips the file (using the zipfile package) 3) parse the xlsx file (using the xlsx package) 4) translate the column headers and descriptions into JSON schema format 5) add that schema to the dataset 6) add that body to the dataset

Then you can set up the dataset to update every quarter so it auto updates for you.

This sounds like a really good use case for Qri and for transforms in particular. Our documentation on some of this is a bit spotty, so if you want to find a time to pair on this lmk!!!

[Frijol]

Thanks! Really want to try this but stuck on database transition to the new Qri I think 😭 – let me know when to try again on account creation?

[ramfox]

Hi! read your post on discord and sent over a fix, let me know if that works. If not, dm me on discord and we will fix it together: https://discordapp.com/channels/497052800778502152/580806684088336395