[data] wpdxdata: an R package for the Water Point Data Exchange data

[larnsce]

Context

• Highly valuable data with great reuse potential
• Updated daily as single CSV
• https://www.waterpointdata.org/

Question

• Where can CSV be accessed?

[larnsce]

Email response:

• I would recommend using the frictionlessdata library in R: https://github.com/frictionlessdata/frictionless-r
• And use it to open our datapackage file: http://catalog.waterpointdata.org/wpdx_enhanced/datapackage.json
• Which contains a link to the CSV file but also contains a schema for the different columns in the file.

@mianzg: Do you have enough to get going with this package?

I suggest we call it wpdx. This package has great potential to be published on CRAN and a package with that name doesn't currently exist: https://cran.r-project.org/web/packages/available_packages_by_name.html

[mianzg]

@larnsce I checked the resources. This dataset set-up would be quite different from our current practice.

The data is pretty tidy but the size is huge (~800,000 rows in basic), and it's stored and updated remotely. I also tried basic functions of frictionlessdta for enhanced data (half of basic version). Again, to load full data might be hard if internet or hardware is not good enough.

Here we need to write more R functions to load and interact with the data, some initial considerations are:

• use lazy loading
• prompt users with their hardware capability, and retrieve subset or full data
• etc...

@mbannert If you got experience with large size data like this, let's talk!

Nevertheless, I will go ahead and create a repo to initiate

[larnsce]

We have a PostgreSQL database that we could add the data to and then call based on a query from the user.

[mianzg]

Consider to work on this for openwashdata hackathon