assist with DE data documentation

[damonmcc]

DE issue: https://github.com/NYCPlanning/data-engineering/issues/944

DE would like automate the generation of data documentation.

current state

The Data Engineering (DE) and GIS teams make and distribute data. In most cases, documentation is made and distributed along with the data and must change to reflect changes in the data.

That documentation is currently manually created and modified. The current forms of documentation which distributed with data are:

• excel files
• PDFs
• text files

The Data Catalog in the DE wiki has links to the destinations these data products and their documentation are distributed to.

desired state

DE would like to minimize or eliminate the manual modifying of data documentation. In, general, DE would like to:

1. create templates of data documentation
2. use the new data product metadata to populate templates

This is an opportunity to standardize and improve the design of data documentation.

Some documentation has strict formatting rules (e.g. data dictionaries in the Attachments of on Open Data pages). But we can determine the form and content of all other documentation.

[jessicashanshanhuang]

initial questions/thoughts:

• what md flavors are currently being used? i'm assuming default is GFM (github flavor). > from kickoff meeting, sounds like there is a custom flavor in the works.

intro to metadata

[damonmcc]

what md flavors are currently being used? i'm assuming default is GFM (github flavor).

we currently use GFM for all our markdown files, but that's just because we only make markdown files to be viewed in github lol

[jessicashanshanhuang]

Follow up from Matt:

I met with Amanda yesterday and she confirmed design fellow should focus on design of a new DCP Readme file. Data Engineering and engineering fellow should focus on developing code to distribute metadata from YAML files to the various metadata end points; DCP Readme, OTI required Data Dictionary, ESRI metadata.

I’m following up with templates and examples of the different metadata, Readme and Data Dictionaries used here at DCP.

Templates (see this SharePoint ​Folder icon folder) – if fellows do not have access please reach out and I’ll send copies directly.

Open Data Dictionary (Individual dataset) Open Data Dictionary (Collection) DCP Readme ArcGIS metadata

Examples (standard):

ArcGIS Metadata (print out) – Zoning Districts DCP Readme - COLP Open Data Dictionary (Individual) – Housing Database: Project Level Files (downloads .xlsx file) Open Data Dictionary (Collection) – Capital Projects Database (downloads .xlsx file)

Examples (ad hoc / non-standard)

PLUTO Data Dictionary PLUTO Readme PLUTO Change File Facilities Readme Facilities Data Dictionary – added tabs (downloads .xlsx file) Zoning Tax Lot Database Data Dictionary/Readme

You can explore all our data and metadata offerings on the BYTES of the BIG APPLE site.

[jessicashanshanhuang]

@damonmcc do we have a specific timeline of when we should get the initial research phase done by?

[jessicashanshanhuang]

Observations:

• updates could be phrased into bullet point format, easier to read at a glance (noted when i looked at the tag portion of COLP readme)
• any default answers for certain portions we must include or phrase a certain way like:

  Use limitations The City of New York makes no representation as to the accuracy of the information or its suitability for any purposes. The City disclaims any liability for errors that may be contained herein.

• seems like there's some inconsistencies between the readme template and COLP readme, some portion omitted although they do not have optional tag
• using a table for the data dict portion, especially for mapping the values section

Questions:

• are all current readmes in pdf format? is there any examples in md format? it would be nice to have an example in md, we could try experimenting and editing an existing one to see what changes would help improve readability
• seems like all current metadata from bytes of the big apple link to a pdf, moving forward will we be linking to a readme in a repo?
• will most readmes be read through github (on the site) or will users most likely read it by downloading md?
• any phrasing that is required for readmes (i.e. use limitations) that need to abide to certain rules?

[jessicashanshanhuang]

check-in 07/24 notes:

• format: pdfs for mvp, open to other options for post-mvp
• legal info: what's required
• collab with heng for html style guide (currently using jinja)
• alex will get back on what fields (pretty open) and legality associated with metadata

[jessicashanshanhuang]

broke down the two major deliverables we will be working on into these issues:

• https://github.com/NYCPlanning/design/issues/54
• https://github.com/NYCPlanning/design/issues/55

[jessicashanshanhuang]

Metadata PDF Heng produced from PLUTO

Questions/Observations:

• Are most data sets usually as large as PLUTO?
• not as large, pluto is one of the largest
• How often is the data in these sets updated? I noticed that some descriptions are left blank so I was wondering if those will be updated in the future and in general if other components change often.
• not as often, not even each year
• Heng mentioned that the content is highly nested - "if you look at the sample output pdf, sometimes the hierarchies/indents are hard to follow, especially when we have a long list of things"
• Heng also mentioned that we can take out categories like destination info and package info
• Depending on where data source is from, there's differences on how things are described in similar categories (i.e. some values descriptions have more information than others or is described in various formats)
• I guess relating to the observation above, how is data prioritize in terms of rank? (which one goes first or is at the top of the stack, what dictates the order?)
• try to preserve the order of columns, generally: first is dataset then columns

Ideas:

• I think it would be good to create a wireframe in Figma to outline branding things like headers and then for each category, pull one-two points and mock how they would look
• Would be good to see how accurate dev mode would be in terms of css
• if we include a header and footer in the pdf, i think it would be good to only have header on first page and repeating footer for the rest

hey @damonmcc, @wndyli here are my thoughts/questions for the check-in, if you could relay those that would be great!

[wndyli]

going off of jessica's comment regarding the pluto metadata pdf:

questions/observations:

• is everything regarding the formatting up for change? such as font, sizing, colors, etc. or what's the flexibility we have when designing?
• links are unclear that they're links
• would de prefer formatting to be relatively simple and similar throughout the pdf or more decorative to provide distinctions between sections?
• is there high variance between categories for different metadata? like would the categories remain mostly the same across different projects or be commonly different?

ideas:

• i like jessica's idea of including a header and footer!
• we could approach the design by taking each category and mocking up what it would look like on figma
• utilizing updated design system to provide more "design" for each category so that it's easier to read (but depends on the answer for the first question)
• including a table of contents at the beginning of the pdf