IDC Guidance Updates and Discussion

[rebeccawilliams]

Below are the OMB August 2015 IDC requirements for Open Data:

Open Data Progress, Use, and Impact

Status: Updated (bold indicates updated requirements)

• Agencies will continue to be assessed on their Enterprise Data Inventory (EDI), Public Data Listing (PDL), Customer Feedback Process, and Data Publication Process. Additional information is available here: https://community.max.gov/x/tBWSK.
• OMB greatly appreciates agencies’ cooperation in responding to OMB’s guidance to publish their EDI, with FOIA exemption-related redactions as necessary, as a downloadable dataset in their PDL. We believe this further makes the U.S. the most transparent government in the world. To further enhance this transparency, effective May 31, 2015, agencies are required to do the following:
  • Include all “non-public” data assets in their PDL, in addition to the “public” and “restricted public” data assets that have long been required. If necessary, agencies shall coordinate with their Freedom of Information Act (FOIA) offices to redact specific metadata text only if the text is subject to a FOIA exemption. Agencies shall use the instructions provided by OMB at https://project-open-data.cio.gov/redactions/ to conduct these redactions. Essentially, this reframes the PDL to be a public version of the full EDI. Note: Because the only difference between an agency’s full EDI and the agency’s PDL will be the existence of any needed redactions, agencies no longer need to submit an EDI to OMB unless their PDL contains any redactions. Non-redacted EDIs, if applicable, should be submitted to the appropriate agency folder here: https://community.max.gov/x/8YamK.
  • If a metadata record in an agency’s PDL contains one or more redactions pursuant to a FOIA exemption, the agency shall explain the need for the redaction in the “rights” metadata field. This is in addition to the existing requirement to use the "rights" field to explain the non-public access level. Please see https://project-open-data.cio.gov/redactions/ for instructions and examples. These explanations should not themselves contain redactions.
• Agencies are already required by the Open Data Policy to have a process to engage with customers to help facilitate and prioritize data release. Each agency shall ensure that their customer engagement process includes a transparent two-way feedback mechanism. A link to an e-mail address is not sufficient. For examples of sufficient tools, please see best practices identified on the Project Open Data Dashboard (http://labs.data.gov/dashboard/), as denoted with a green star. Data.gov is developing a “Help Desk” tool that will help fulfill this requirement. As Held Desk development progresses, updates will be provided on the Open Data Listserv.
• To ensure the legal reuse of federal open data is conspicuous agencies must include an explanation for all datasets that do not include a Public Domain URL in the license field by filling out the rights field. See license and non-license (i.e. Public Domain) URL examples and more details at: https://project-open-data.cio.gov/open-licenses/
• Each agency shall ensure their EDI and PDL contains all of the agency’s public, restricted public, and non-public APIs. Agencies shall use the guidance provided at http://18f.github.io/API-All-the-X/pages/apis_in_data_catalogs (with more details at https://project-open-data.cio.gov/v1.1/api/) to help ensure their APIs are included in these catalogs and that each API is identifiable by the use of the format field set to "API" in the metadata record's distribution object.
• Each agency shall enrich their EDI and PDL by ensuring all data assets include the individual datasets within by using the identifier and isPartof fields. See examples and more details at: https://project-open-data.cio.gov/v1.1/collections/

  Questions on Use and Impact

Provide at least 5 quarterly examples of open maturity based on public feedback, and include the following information.

Requirement	Description
Dataset Name	The name of the dataset
URL	The URL for the dataset
Date of Feedback	Date public feedback on dataset was received
Public Feedback Received	What public feedback was received? Public feedback examples include: FOIA requests, two-way public feedback mechanism requests, Data.gov Help Desk requests, emails, news reports, academic papers, and citizen or advocacy blogposts.
Improvements Made	What improvements were made to the dataset to make it more open? Examples include: machine-readable formats, real-time delivery, worldwide public domain dedication, bulk downloads, free access to previously charged for data, public access to previously Restricted Public data, and data quality improvements.
Provide quarterly summary information from your two-way public feedback mechanisms.	Provide qualitative responses to each of the four questions: 1. What are some of the primary uses of your agency’s data? 2. What have been the primary channels for the users to learn about your data? (e.g. email, agency-sponsored events such as datapaloozas and hackathons, non-agency sponsored events, etc.) 3. What are suggestions from key users on improving the usability of your data? 4. What are suggestions from key uses on additional agency data resources to release?

Question on the Enterprise Data Inventory

Question	Options
To what extent is your agency’s Enterprise Data Inventory (EDI) complete?	Select one of the following options:To a very great extent, To a great extent, To some extent, To a very little extent

What best practices exist for the updated requirements? Is anything unclear about the cited guidance at:

• https://project-open-data.cio.gov/open-licenses/
• https://project-open-data.cio.gov/v1.1/collections/

[rebeccawilliams]

Noting November 31, 2015 IDC Guidance is the same. Only addition is a catch-all comments field to help explain discrepancies.

[rebeccawilliams]

We should add this to the Project Open Data site itself. cc @justgrimes

[justgrimes]

@rebeccawilliams yes, I agree that this should be on the website and not an issue; should we include previous versions or only the most recent IDC guidance?

[rebeccawilliams]

Fixed with #543.