Better definition for DataDictionary

[chachasikes]

I'm researching the use and formatting of open data dictionaries.

The Project Open Data schema defines a dataDictionary as: "URL to the data dictionary for the dataset or API. Note that documentation other than a data dictionary can be referenced using Related Documents as shown in the expanded fields."

It could be helpful to provide a definition of what a data dictionary is actually supposed to be, and contribute some best practices towards formatting data dictionaries. Schemas help people put information out there in formats that are more useful, and I would like to see data dictionaries improve in the next couple of years.

• In my survey of 10 data dictionaries, I found at least one mistaken file on data.gov (for the SSA Baby Names data dictionary.) I will keep track of other errors I find as I look at the data.
• The formats of the dictionaries vary in an extreme way and are very complicated to read: PDFs, search engines, excel files and word docs.
• It would be nice to see some efforts at the communication aspects of the data dictionaries be improved. Gov.UK has some style guidelines on plain-language writing which might help anyone who is creating new data dictionaries or updating older ones.

(My initial survey)

At its simplest, a data dictionary can be

• a list of the field names and their definitions (see City of Chicago Data Dictionary).
• It's helpful if the field names also come with a table for the attributes for each field, since field values often use reference codes.

I looked at Socrata (looking at the raw JSON data output), and they currently have the following fields associated with their fields:

• id, name (machine readable name), fieldName (display name), dataTypeName (checkbox, text, integer, etc), description, comments, renderTypeName (for machine display of field content), cachedContents (a summary)

---

From Wikipedia: http://en.wikipedia.org/wiki/Data_dictionary A data dictionary, or metadata repository, as defined in the IBM Dictionary of Computing, is a "centralized repository of information about data such as meaning, relationships to other data, origin, usage, and format."[1] The term can have one of several closely related meanings pertaining to databases and database management systems (DBMS):

• A document describing a database or collection of databases
• An integral component of a DBMS that is required to determine its structure
• A piece of middleware that extends or supplants the native data dictionary of a DBMS

---

My Notes on Data Dictionaries

I very much appreciate your schema and look forward to getting to know it better. If you have any responses or feedback, I would like to hear it, or even help out with this issue.

• Chach Sikes

p.s. I also think a schema for JSON compatible data dictionary content would be helpful, but am researching that and will follow up with a separate issue if I find anything useful.

[philipashlock]

Thanks for the feedback! On your last point on a schema for data dictionary content, we've already included some guidance on using established schema standards for machine readable API documentation in our API guidance but we haven't done this for any other kind of datasets. For simple tabular data like CSVs, it looks like the JSON based Data Package specification, specifically the JSON Table Schema, has potential (that page also links to related work). It looks like that spec work is now being formalized as part of the W3C Model for Tabular Data and Metadata on the Web.

For something more complex like XML or for anything that's just free form HTML, it would be good to have an ideal way of formatting and presenting that (using an XSL if needed) with a basic HTML5/CSS template.

[rebeccawilliams]

Closing this issue, but I have ported over the notes to #465