search

Nonprofit-Open-Data-Collective / irs-efile-master-concordance-file

The Master Concordance File defines standards and provides documentation necessary to build structured databases from the IRS E-File XML files posted on AWS.
https://nonprofit-open-data-collective.github.io/irs-efile-master-concordance-file/
40 stars 6 forks source link

Needs to be added. Was: Variables introduced in 2016 #15

Open jsfenfen opened 6 years ago

jsfenfen commented 6 years ago

I think the current version of the concordance leaves out elements introduced in 2016v3.0, including the following. That may be deliberate--but at some point I thought this had been discussed and was going to include this stuff?

/ReturnHeader/FilingSecurityInformation/IPAddress/IPv4AddressTxt /ReturnHeader/FilingSecurityInformation/IPAddress/IPv6AddressTxt /ReturnHeader/FilingSecurityInformation/IPDt /ReturnHeader/FilingSecurityInformation/IPTm /ReturnHeader/FilingSecurityInformation/IPTimezoneCd /ReturnHeader/FilingSecurityInformation/FederalOriginalSubmissionId /ReturnHeader/FilingSecurityInformation/FederalOriginalSubmissionIdDt /ReturnHeader/FilingSecurityInformation/FilingLicenseTypeCd /ReturnHeader/FilingSecurityInformation/AtSubmissionCreationDeviceId /ReturnHeader/FilingSecurityInformation/AtSubmissionFilingDeviceId

/IRS990ScheduleA/AgriculturalResearchOrgInd /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CollegeUniversityName/BusinessNameLine1Txt /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CollegeUniversityName/BusinessNameLine2Txt /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CityNm /IRS990ScheduleA/AgriculturalNameAndAddressGrp/StateAbbreviationCd /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CountryCd

/IRS990ScheduleA/DistributionAllocationsGrp/ExcessDistributionCyovYr3Amt /IRS990ScheduleA/DistributionAllocationsGrp/ExcessFromYear4Amt

/IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/PriorCHNAImpactInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/NotifiedFAPCopyBillDisplayInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/FAPTranslatedInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/PermitDeferDenyRqrPaymentInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/EngageDeferDenyRqrPaymentInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ProvidedWrittenNoticeInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/MadeEffortOrallyNotifyInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ProcessedFAPApplicationInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/MadePresumptiveEligDetermInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/LookBackMedicareInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/LookBackMedicarePrivateInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/LookBackMedicaidMedcrPrvtInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ProspectiveMedicareMedicaidInd

borenstein commented 6 years ago

We need a protocol for updating the concordance. That should be part of our powwow.

On Oct 26, 2017 11:47 PM, "Jacob Fenton" notifications@github.com wrote:

I think the current version of the concordance leaves out elements introduced in 2016v3.0, including the following. That may be deliberate--but at some point I thought this had been discussed and was going to include this stuff?

/ReturnHeader/FilingSecurityInformation/IPAddress/IPv4AddressTxt /ReturnHeader/FilingSecurityInformation/IPAddress/IPv6AddressTxt /ReturnHeader/FilingSecurityInformation/IPDt /ReturnHeader/FilingSecurityInformation/IPTm /ReturnHeader/FilingSecurityInformation/IPTimezoneCd /ReturnHeader/FilingSecurityInformation/FederalOriginalSubmissionId /ReturnHeader/FilingSecurityInformation/FederalOriginalSubmissionIdDt /ReturnHeader/FilingSecurityInformation/FilingLicenseTypeCd /ReturnHeader/FilingSecurityInformation/AtSubmissionCreationDeviceId /ReturnHeader/FilingSecurityInformation/AtSubmissionFilingDeviceId

/IRS990ScheduleA/AgriculturalResearchOrgInd /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CollegeUniversityName/ BusinessNameLine1Txt /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CollegeUniversityName/ BusinessNameLine2Txt /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CityNm /IRS990ScheduleA/AgriculturalNameAndAddressGrp/StateAbbreviationCd /IRS990ScheduleA/AgriculturalNameAndAddressGrp/CountryCd

/IRS990ScheduleA/DistributionAllocationsGrp/ExcessDistributionCyovYr3Amt /IRS990ScheduleA/DistributionAllocationsGrp/ExcessFromYear4Amt

/IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/PriorCHNAImpactInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ NotifiedFAPCopyBillDisplayInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/FAPTranslatedInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ PermitDeferDenyRqrPaymentInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ EngageDeferDenyRqrPaymentInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ProvidedWrittenNoticeInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/MadeEffortOrallyNotifyInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ProcessedFAPApplicationInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ MadePresumptiveEligDetermInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/LookBackMedicareInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/LookBackMedicarePrivateInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ LookBackMedicaidMedcrPrvtInd /IRS990ScheduleH/HospitalFcltyPoliciesPrctcGrp/ ProspectiveMedicareMedicaidInd

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Nonprofit-Open-Data-Collective/irs-efile-master-concordance-file/issues/15, or mute the thread https://github.com/notifications/unsubscribe-auth/AEPgn50yLMf3me0pn_5vxV-RW2b_eugMks5swVJjgaJpZM4QIgtO .

jsfenfen commented 6 years ago

I thought about updating the concordance with these--and can do so eventually--but what's missing is the variable name generation code. It would be awesome if two people added the same xpath they got back the same automatically generated variable name...

borenstein commented 6 years ago

I don't think that's possible, because multiple xpaths map to the same variable. That's why the first validatathon was a manual process--we were making up arbitrary variables.

On Oct 27, 2017 11:46 AM, "Jacob Fenton" notifications@github.com wrote:

I thought about updating the concordance with these--and can do so eventually--but what's missing is the variable name generation code. It would be awesome if two people added the same xpath they got back the same automatically generated variable name...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Nonprofit-Open-Data-Collective/irs-efile-master-concordance-file/issues/15#issuecomment-340008976, or mute the thread https://github.com/notifications/unsubscribe-auth/AEPgn_Oy_nEZuljwbviR9uDznRh0Y9Y4ks5swfq_gaJpZM4QIgtO .

atartanian commented 6 years ago

This is an issue I'm running into as well. My company is looking to keep an up to date database of the filed 990s and the lack of support for 2016 and 2017 schemas mean's I'll have to do this manually. Is there any plan for how to keep this concordance up to date or alternatively some documentation on how to add new rows to the concordance document? Since I'll be doing the work anyway, I'd be interested in contributing my schema updates back to this project, but will definitely need some guidance about how to best do that.

atartanian commented 6 years ago

regarding the variable name generation question, wouldn't it be most sane to generate the variable name from the earliest occurrence (aka the field name that said piece of data appeared under in the schema with the lowest year and version number)? That would allow anyone to have a clear rule to add new variables without having to consult with everyone here beforehand.

jsfenfen commented 6 years ago

@atartanian: 2016 and 2017 support is coming!

regarding the variable name generation question, wouldn't it be most sane to generate the variable name from the earliest occurrence

Yeah that's an approach I take in my version of the metadata here: https://github.com/jsfenfen/990-xml-metadata/ .

atartanian commented 6 years ago

@jsfenfen any idea how far off that is? I'm currently weighing if I need to write a mapping myself for the fields i'm interested in or if I can wait for an official update to this project.

lecy commented 6 years ago

@atartanian would love to hear more about your project.

We are focused on the most useful fields on the base 990 (990 and 990-EZ) for the moment, with a few misc schedules. We should have a significant update in a couple of weeks, including more consistent and readable variable names.

atartanian commented 6 years ago

@lecy my company provides philanthropy information to non-profits who are looking for donors. one of the ways we do that is to connect a given philanthropic gift to the non-profit who received it and provide additional information about said organization. we use the 990 data to do that. I'm looking to automate the process of updating the 990 data into our database and this concordance is saving me from having to write my own. We're hoping to keep our 990 dataset up to date in at least a quarterly cadence, so I'm looking to see how frequently this concordance will be updated as well as how I can participate to help on that front.

Can you share which fields from the 2016 and 2017 schemas you are planning to include for the upcoming update?

jsfenfen commented 6 years ago

@atartanian believe this data is complete through 2016v3.0, see more here: https://github.com/jsfenfen/990-xml-reader/ ; That project standardizes 2013 forward and will be crosswalked to Jesse's concordance variables when they're out. All variables in all lettered schedules from schema year 2013 through 2016v3 are standardized, I should have 2017 soon.

atartanian commented 6 years ago

@jsfenfen @lecy ok great that sounds like it's worth me putting this on the back burner for a little bit to avoid reinventing the wheel. thanks!

atartanian commented 6 years ago

@jsfenfen @lecy hey folks, just looping back to this and wanted to check to see if the post-2016v3 schemas have made it into the concordance?

jsfenfen commented 6 years ago

Hey @atartanian the 2017 schema changes are up here: https://github.com/jsfenfen/990-xml-metadata/ . The changes seem pretty minimal, btw, and as of May IRS had yet to release any xml in 2017 schemas. Not sure when they'll hit the concordance, @lecy would know best.

lecy commented 6 years ago

Hi @atartanian, I will be pushing some changes to the concordance later this week that impact some variable names, mapping, and meta-data on the main 990-PC and 990-EZ forms (not the 990-PF or Schedules).