Wishlist

[lapidus]

Background

Making the DDF standard accessible and useful to people outside the internal, existing community, starts with state-of-the-art documentation and guides.

Goals

The DDF Github organization aims to:

• Build an authoritative website / resource collection for DDF so that other organizations find the project trustworthy and vibrant enough to "bet on" this standard and technology
• Make it possible for any data engineer/data scientist to produce a valid DDF dataset
• Introduce DDF in an accessible language so that almost anyone can understand its applications and scope (incl. its comparative advantages to other data standards)
• Collect all resources in one place
• Make it easier to contribute to the DDF community (incl. new tools, guides, bug fixes etc.)

What exists today?

Websites https://open-numbers.github.io/ddf.html https://ddf-utils.readthedocs.io/en/latest/ https://github.com/Gapminder/ddf-validation

Data model (30 pages) https://docs.google.com/document/d/1Cd2kEH5w3SRJYaDcu-M4dU5SY8No84T3g-QlNSW6pIE/edit#heading=h.5h1e33vzhdlu

CSV format (31 pages) https://docs.google.com/document/d/1aynARjsrSgOKsO1dEqboTqANRD1O9u7J_xmxy8m5jW8/edit#heading=h.1nakjy92hz6r

DDF Query format (18 pages) https://docs.google.com/document/d/1olFm-XXjWxQ4LrTCfM42an6LbjbIgnt__V1DZxSmnuQ/edit

Wishlist

This issue aims to produce a first wishlist for a better ecosystem to navigate and understand the DDF standard and it's applications. When the wishlist has been further developed it will be split into an MVP (minimum viable product) – consisting of features that will immediately increase the impact of DDF – and another list of "Add-on functionality" that are considered more as "nice-to-haves".

Current draft:

• [ ] Non-technical introduction to DDF
• [ ] Lots of hands-on examples, starting with simple and then expanding to more advanced, real use cases
• [ ] Comparison to other standards (how does it relate to SDMX, Datapackages, StatDCAT-AP etc.)
• [ ] Chapter-based documentation of DDF (loosely based on current Google Docs content)
• [ ] Interactive playgrounds to test correct/incorrect CSVs and what they generate (with interactive feedback)
• [ ] Explanations of tooling with link to Github (validators, utils etc.)
• [ ] Description of workflow: How to start a DDF project. Best practices etc.

Suggested Tech:

• [ ] Version-controlled static website for the docs
• [ ] This Github org. for all repos dealing with DDF (website, validators, utils etc.)

Please contribute ideas and links below. I will updated the above text to reflect your ideas.

[jheeffer]

Here's current public DDF documentation: https://drive.google.com/drive/folders/1IflpeZ-03ltqiujDTKN6duQkxjo7CrlO?usp=sharing

Including: A simpler, shorter, introduction to DDFcsv A few points of comparison against SDMX DDFcsv datapackage specification Overview of ideas/problems to solve for DDF 2

These can be used as starting points for the DDF (documentation) site we're envisioning.

[semio]

Hey! I wrote a doc about how we do etl a while ago: ETL Tasks in Gapminder Which includes recommendations for etl project structure and tools manage a project. This might be useful too