research question: mapping pilot DATA Act element names to v .5 of the DATA Act standard

For story #148 (confirm flat file mappings to draft of the DATA Act standard in XBRL form), we're producing a flat file-->XBRL crosswalk for each of the four files being used in this pilot: https://github.com/18F/data-act-pilot/tree/master/mappings

Summary

This is summarized list of findings. Scroll down for more details.

We were unable to map the following fields in the pilot .csvs to the data standard:
- FederalPrimeAward
- OfficeName and OfficeCode (both awarding and funding)
How should we handle fields in the data standard that have a different origin and/or meaning depending on award type?
- fain and piid for awardID
- ReasonforModification and TypeOfAction for typeOfAction
- TypeOfContractPricing, idvType, ContractAwardType, and AssistanceType for typeOfTransactionCode
Is highly compensated officer data expected for all award types or just contracts? The current data standard defines these fields as relevant to all awards, not just procurement data.
The format of recipient address was inconsistent between the pilot files and the data standard.
Details

This exercise raises the following research questions.

Note: This issue isn't meant to address discrepancies we can code around until the element names are synced up. For those, we coded in a temporary crosswalk.

Award ID

The award flat file template used in this pilot specifies different data element names for an award identifier:

Award ID (components of the piid) for contracts
Award number (fain) for assistance awards

The .xsd files (i.e., schema docs) that accompany the XBRL format specify one data element called awardID, regardless of award type. This is consistent with the finalized definitions of Award Identification Number.

Current Action for Sprint 9: Map fain to awardID, since we're only using grant data

Suggestions:

revise the award flat file template to align with the Data Exchange Standard Draft (v0.5) (i.e., the file will contain one awardID field that represents both contracts and financial assistance)
add a validation that checks the award id to make sure its format is consistent with the type of award
Other Award Type Differences

Similarly, there are a few other elements in the pilot award flat file template that are applicable to a specific award type but ultimately map to the same element in the data standard.

ReasonforModification (contract) and TypeOfAction (financial assistance) map to typeOfAction
TypeOfContractPricing (contract), idvType (contract), ContractAwardType (contract), and AssistanceType (financial assistance) map to typeOfTransactionCode

Current Action for Sprint 9: Map grant-related fields to the award-type-agnostic element names in the data exchange standard

Suggestions:

revise the award flat file template to align with the Data Exchange Standard Draft (v0.5) (i.e., the file will contain one, standardized field that represents all award types)
provide explicit guidance about how agencies should supply data for these fields, given type of award and other relevant information
add supporting validation rules, if appropriate
typeOfTransactionCode for Contracts

The Element: Award Type, Action Type, Record Type whitepaper lists several contract-related fields as components of transaction type (i.e., typeOfTransactionCode in the data standard draft).

The text is a bit confusing. I understand that Infinite Delivery Vehicle and Contract Award Type values are mutually exclusive. How does Type of Contract Pricing fits into typeOfTransactionCode?

Current Action for Sprint 9: None needed - we're not using contract data

Suggestions:

provide explicit guidance about how agencies should supply typeOfTransactionCode for contracts
add supporting validation rules, if appropriate
FederalPrimeAward

The Element: Award Type, Action Type, Record Type whitepaper describes a prime award as:

the highest level of award for which funding will be summarized and for which more detailed information will be made available

Current action for sprint 9: Ignore FederalPrimeAward when converting to XBRL unless we can identify a target element.

Question: How do FederalPrimeAward values (grant, loan, cooperative agreement, other, and contract) map to the data standard?

Awarding and Funding Office

The data standard draft (specifically, da-gen-2015-06-29.xsd) defines agencyOfficeComplexType. However, I don't see it referenced in any of the award .xsd files (da-award-content-2015-06-29.xsd, da-finassist-content-2015-06-29.xsd, or da-procurement-content-2015-06-29.xsd).

Current action for sprint 9: Awarding and funding office information will not be converted to XBRL unless we can identify the target elements.

Question: Does the current data standard draft include the following elements? If not, will they be included in a future revision?

Awarding Office Name/Code
Funding Office Name/Code
Address Inconsistencies

Noticed a few inconsistencies between gen:addressComplexType as defined in the data standard and recipient address information in the pilot's award flat file template:

The flat file lists two address lines, and the data standard contains one
The flat file lists two zip code-like fields (RecipientLegalEntityZip, RecipientLegalEntityZip+4).

Current action for sprint 9:

Pick a zip code field (preferably the one with the best data) to convert to XBRL
Convert the first line of recipient address to XBRL; ignore second line

Suggestions:

Make the number of address lines consistent in the flat file spec and the draft data standard
Remove one of the zip code fields from the flat file spec
Highly Compensated Officer

Need clarification about how this info is handled in the draft data standard.

Current action for sprint 9: None, since we're working with grants data.

Questions:

Is highly compensated officer data to contract data only? If so, should it live in da-procurement-content instead of da-award-content?
In da-award-content-2015-06-29.xsd, there is an element named highlyCompensatedOfficer, which is a complex type consisting of officer first, last, and middle initial and the compensation amount. However, the name and amount fields are also listed as separate elements. What is the intended use?

18F / data-act-pilot