Generate initial flat file from PolicyEngine Enhanced CPS

[nikhilwoodruff]

As the first task in the project, we need to flatten PolicyEngine's Enhanced CPS (2023) file and enable it to be used in Tax-Calculator.

[martinholmer]

@nikhilwoodruff said in issue #2:

As the first task in the project, we need to flatten PolicyEngine's Enhanced CPS (2023) file and enable it to be used in Tax-Calculator.

Why not 2021?

That is the year for which the most recent IRS-SOI targeting statistics are available, as I mentioned in this issue 5 comment.

[nikhilwoodruff]

@nikhilwoodruff said in issue #2:

As the first task in the project, we need to flatten PolicyEngine's Enhanced CPS (2023) file and enable it to be used in Tax-Calculator.

Why not 2021?

That is the year for which the most recent IRS-SOI targeting statistics are available, as I mentioned in this issue 5 comment.

Can't we extrapolate the targets to 2023? If we're going to extrapolate the microdata to 2023 anyway...

[martinholmer]

@nikhilwoodruff said:

Can't we extrapolate the targets to 2023? If we're going to extrapolate the microdata to 2023 anyway...

Doesn't make any sense to me.

Only a few days ago we were planning to develop a 2015 data set from the 2015 PUF and the 2015 CPS, and then evaluate that 2015 data set against a number of 2015 validation targets. All of this was for the same year and involved no extrapolation of any kind.

I don't see how the extrapolation of the validation targets (like an historical JCT EITC tax expenditure estimate) can be done in any sensible way. Any way of doing it will introduce reasons why the 2023 data set generates tax expenditure estimates that differ from the extrapolated tax expenditure estimate that have nothing to do with the quality of the constructed data set.

[nikhilwoodruff]

Ok, thanks for the clarification @martinholmer. Though- how were we planning to validate 2023 anyway? If tax expenditures statistics are uninformative for years after their data year, even if we did start at 2021, surely we wouldn't have helped much to prove the validity of a 2023 dataset?

[martinholmer]

@nikhilwoodruff asked in issue #2:

how were we planning to validate 2023 anyway?

We weren't planning on doing that in Phase 1, Phase 2, or Phase 3. Why? Because people are just now (until April 15th) filing their 2023 income tax returns. So, there will be no IRS-SOI statistics on 2023 returns for a couple of years. In a couple of years, we will be able to "validate 2023".

[nikhilwoodruff]

@martinholmer I guess I'm just struggling to understand the difference in value between:

• Validating 2021 data and logic against 2021 external evidence, then making some assumption about how much that validity probably carries through to 2023

• Validating 2023 data and logic against 2021 external evidence (with some assumption about how it'd likely change up to 2023)

They both seem to require the same level of assumption/uncertainty on our part and the same quality guarantee on the final 2023 data, but the former adds extra work in making 2021 data in addition to the other years, which won't be used for policy analysis.

[martinholmer]

@nikhilwoodruff, OK. Go ahead with your plan of creating a flattened file from the 2023 PEUS PUF-enhanced CPS data. We will not be able to compare 2023 aggregate tax liabilities with IRS-SOI data (the most recent of which is 2021), but we can compare 2023 aggregate tax liabilities from the 2023 flattened file with recent CBO projections.

[donboyd5]

@nikhilwoodruff @martinholmer I was away yesterday and most of today.

Here's how I see this:

1. Validation against published historical data is very important but (1) we don't have to do it in Phase 1, and (2) we can't easily do it if the PE file is 2023 and we don't have published stats for 2023. So we should just not do it in Phase 1. (Can we forecast 2021 stats to 2023 and use that to help us target 2023? Yes. Could that improve the quality of a 2023 file? Yes. Should we do that in Phase 1? No. Our goal is not to develop the best-possible PE file for 2023 in Phase 1. Our goal is to develop a deliverable-meeting PE file for 2023 for Phase 1, while developing tools and methods that will help us develop the best possible Tax-Calculator input file that we have time to develop in Phases 2 and 3.)

2. Analysis of future-year file quality in Phases 2 and 3 will be important, but perhaps we should not (?) use the term validation for this? We're not validating against known numbers, we're looking for information that makes us investigate and improve quality. For example, we can hit extrapolated targets. I suppose that's validation but it's only part of checking quality and realistically, we should get that right. The bigger question for future years is comparing liability under current law and potential reforms against presumed-high-quality alternative estimates to look for clues about where we might have problems (although other estimates may be the ones that have problems), use those clues to help us figure out whether we did something wrong. If we did, we fix it. If we didn't, we have a record of our analysis that will help if anyone ever asks about it.

3. The key in Phase 1 is to develop a deliverable-meeting Phase 1 PE file, and tools and methods that will help us in Phases 2 and 3.

Agree/disagree?

[martinholmer]

@donboyd5, I agree with what you said here in issue #2.

Do you have any ideas about an alternative to "validation"?

[donboyd5]

@martinholmer The best word I have come up with so far for what we want do regarding data constructed for future years is "examine" the data with an eye toward finding potential problems. "Validation" clearly is wrong because we don't know truth against which to compare the data. "Evaluation" seems wrong for the same reason. "Analyze" seems too open-ended. "Examine" to me implies a more targeted and limited look at the data - focusing on specific things (e.g., how different are our estimates of current law from estimates of CBO and JCT?, how different are our estimates of selected reforms from CBO and JCT?).

I asked ChatGPT "What's the difference in meaning between examine and analyze?" and here are key parts of its response (FWIW):

While both terms involve close inspection, analysis is typically more in-depth than examination. Analysis seeks to understand the how and why, going beyond surface-level inspection to uncover underlying structures or meanings. ... Examination is often the preliminary step that can lead to analysis. You might examine something to get a comprehensive view of its state or components, but you analyze it to understand its nature, relationships, or functions more deeply.

I think @martinholmer is developing tools for examination. What we learn from those tools may lead @nikhilwoodruff and @donboyd5 to do further analysis. For example, we may learn that our data produces higher estimates of cost for certain kinds of reforms than JCT estimates. That might lead @nikhilwoodruff and @donboyd5 to conduct analysis to look into what could be causing that: Do we have more more of one kind of income or filers than JCT? Do we have a different distribution of income or # of taxpayers (if that information is available)? We may not have a way to quantify this - we might even discuss on the phone or by email with JCT, and we might ask them to provide more information about their data. After that, we might conclude that our differences are sensible and we'll stand with what we have, or we might decide to make changes.

So, I suggest that, with regard to data for future years, what @martinholmer is doing is developing tools for examination that may lead to deeper analysis.

[martinholmer]

@donboyd5, Thanks for you analysis of alternative to the work validation. Henceforth, I will refer to my activities as "data examination".

[nikhilwoodruff]

All agreed on the above from me here.