Download model data - Githubissues

danielfdsilva commented 5 years ago

WB feedback

Is there any way to download the data/documentation of the API?

bpstewar commented 5 years ago

We are looking specifically for tools to download the geospatial data presented on the platform: we can access a pdf describing the map, but right now there is no way to download GIS data or the results

vgeorge commented 5 years ago

We propose to perform the following changes to address this issue:

Rename "Export" button to "Download";
Add a toggle to this button, with the following options:
- Current view as PDF: the existing PDF report
- Full Scenario Data: the csv file for the current scenario, available in S3 (eg. mw-1-0_0_0_0_0_0.csv.zip)
- Country Clusters: the shapefile for cluster geometries of the current country

@bpstewar @alekordESA please let us know if we can move forward with this approach. If that is the case, permissions on /GEP folder in WB S3 bucket must be set to public, as the files are not available from outside at the moment:

https://s3.amazonaws.com/wbg-geography01/GEP/models/mw-1/explorer/mw-1-0_0_0_0_0_0.csv.zip

bpstewar commented 5 years ago

I would prefer the second two options direct to a page on EnergyData.info, rather than provide direct access to the data - the files are too big for most users to access, so we need to provide a bunch of contextual information

vgeorge commented 5 years ago

@alekordESA @bpstewar The download button was implemented as discussed last meeting. The live website now shows a "dropup" listing the three download options (pdf/geo/raw) with the the last two disabled for future implementation. Please review and lets us know if it is ok.

vgeorge commented 5 years ago

For the official launch, the buttons should link to the model page at https://energydata.info. Remaining tasks:

[ ] Provide country page links at energy.info @bpstewar
[ ] Include links to cluster and model data in model config @akorkovelos @vgeorge
[ ] Provide public links for the final S3 bucket to be added to country pages @EricSoroos

vgeorge commented 5 years ago

I've updated the comment above with the remaing tasks for this issue. Please let me know if you have any comments.

cc @bpstewar @akorkovelos @EricSoroos

vgeorge commented 5 years ago

This was wrongly closed, we still need to populate the URL links at the model files. Will add them when energy.info pages are ready.

I believe that instead of offering the download of bulk model files, which are expensive, the model page at energy.info could offer several zipfiles, one for each scenario. An user would be able to download a scenario directly and, if there is need to download all of them, he/she could reach the consortium via e-mail and request. @bpstewar, do you think this strategy could work?

bpstewar commented 4 years ago

The model files are now available on energydata.info (although still private for right now). What do you need to link the download link to the energydata.info pages?

EricSoroos commented 4 years ago

Each of the countries has a dataset page with links to the 4 files that we're serving:

It's probably best to link once to this page in the dropdown.

This is a list of the datasets that are currently on the platform.

gep-data-links.xlsx

vgeorge commented 4 years ago

Download links are set in model config files like this:

https://github.com/global-electrification-platform/data-service/blob/5925b296c9bac650684c756dbf9211d0081f9455/fixtures/mw-1/mw-1.yml#L11-L13

To make them available at the front-end, all model configs should be updated with the links and reingested.

cc @EricSoroos

bpstewar commented 4 years ago

Thanks, Vitor, is that something you will do?

vgeorge commented 4 years ago

@bpstewar, @EricSoroos suggested and I created a repository to host model configuration files. We thought that it would be good to put these files under version control to have a better log of changes. If you prefer the previous approach, please let us know.

Regarding the download links, my understanding is that Eric would add them to files once the download URLs exist and perform the reingest on the production database.

EricSoroos commented 4 years ago

@vgeorge There are 4 resources in the datasets, including the additional contextual resources that are important for the interpretation of the contents of the other items.

There should be one link to the dataset in the application, rather than the two links to individual resources that are there.

EricSoroos commented 4 years ago

I've sent a PR for the single link

global-electrification-platform / explorer

Download model data #133