donboyd5
commented
5 years ago The third of these approaches is quite interesting but I'm not sure how to operationalize it unless we can define what makes for a good set of weights, beyond satisfying targets, because there will be a huge number of sets of weights that can satisfy targets.
If we can operationalize it, we can start to examine it empirically early on, using data files @MaxGhenis has already created - a problem that is large enough to be interesting, and small enough to keep us from worrying about computer resources. If we work out an analytic approach, we can easily apply it to more-sophisticated synthesized files, and compare to other approaches.
But let's see if we can operationalize it.
Let's assume we have 2 data files:
- "actual," which is the true, nonreleasable file (we could use @MaxGhenis's 5% holdout dataset, test.csv, for this)
- "syn," which is the synthesized releasable file (we could use the synthpop-synthesized file, synth_synthpop.csv, for this)
In round numbers, both files have 22k observations and 60 variables (including a weight).
We want to compare file-quality measures such as weighted wages or AGI against the values in actual (presumed to be correct), by income range, for 3 approaches to the weights:
-
use the synthesized weights in syn, which we know have some problems
-
use the synthesized weights in syn, adjusted as @donboyd5 suggested, obtained by hitting a set of targets while minimizing how far the new weights move from the synthesized weights (implicitly, this assumes that there is something good about the synthesized weights, and moving too far from them is bad)
-
throw out the synthesized weights in syn, and choose weights that minimize an objective function based on deviations from the targeted values, as @MaxGhenis suggests. I think this is based on the idea that approach 2 might be artificially and unnecessarily anchoring us to weights that simply may not be very good. Maybe we are better off constructing weights from scratch that will achieve all targets and not be anchored to original or synthesized weights.
We know from experience that version 2 virtually always will be easily solvable even with equality constraints for hundreds of targets. That is, there are many - perhaps thousands - of sets of weights that will satisfy the constraints exactly. (We're finding 22k unknown variables that satisfy a few hundred constraints - many different sets of variables can do this.) Approach 2 chooses the set of weights that are closest to the synthesized weights.
Now on to approach 3: any set of weights that hits all of the constraints exactly will minimize an objective function based solely on distance between targets and weighted values on the file, so how do we know which one to choose?
To make it more concrete, suppose we want to satisfy 9 targets constructed from the actual file:
- number of weighted returns in AGI range $0 - $50k
- number of weighted returns in AGI range >$50k - $100k
- number of weighted returns in AGI range > $100k 4-6. total weighted wages in each of those ranges 7-9. total weighted itemized deductions in each of those ranges
In approach 2, we would choose 22k new weights that satisfy these 9 constraints and also minimize an objective function that penalizes change from synthesized weights. One simple objective function we could use to decide which constraint-satisfying set of weights is best is minimizing the sum over 22k records of the squared difference from 1 of the ratio of new weights to synthesized weights. Formally, this is:
objective = sum(x[i] - 1)^2
where each x[i] is the ratio of the new weight to the synthesized weight on a record.
There would be 9 constraints, as defined above.
This is a nonlinear program rather than an LP but the idea is essentially the same as what @MaxGhenis discussed. In practice we might use something slightly more complex.
In approach 3, what would we do? We might set the objective function up as the sum of squared differences between each constraint's calculated value and its target value -- something like:
choose new weights w to minimize the sum of the following 9 squared differences, where each i indexes into the 22k records:
objective = {sum(w[i] (AGI[i] in 0-$50k)) - target1}^2 + {sum(w[i] (AGI[i] in >$50k - $100k)) - target2}^2 + {sum(w[i] (AGI[i] in >$100k)) - target3}^2 + {sum(w[i] wages[i] (AGI[i] in 0-$50k)) - target4}^2 + {sum(w[i] wages[i] (AGI[i] in >$50k - $100k)) - target5}^2 + {sum(w[i] wages[i] (AGI[i] in >$100k)) - target6}^2 + {sum(w[i] itemded[i] (AGI[i] in 0-$50k)) - target7}^2 + {sum(w[i] itemded[i] (AGI[i] in >$50k - $100k)) - target8}^2 + {sum(w[i] itemded[i] * (AGI[i] in >$100k)) - target9}^2
(There are obviously scale issues in defining this objective function. We might scale the calculations so that each constraint is in [0, 1], or in some other way, but that's a next step after we figure out how to set up the problem.)
As I understand how @MaxGhenis put this forward, there wouldn't be any constraints - in approach 3, we would just minimize this objective function.
Obviously the objective function is minimized when all constraints are exactly satisfied, and so the optimal solution to approach 2 (based on its definition of optimality) would also minimize the objective function in approach 3. But many other sets of new weights would satisfy it, too - for example, any of the constraint-satisfying solutions that the NLP solver may have iterated through before it found the solution that minimized the objective function in approach 2. If the goal is to select weights that are better than those in approach 2 in some way, then either we need to add some sorts of constraints (I am not sure what) or somehow add a measure of "good" weights to the objective function. Is there some better definition of a good weight, other than one that is close to the synthesized weight? Would we rather have equal weights for all records, or something else? Even if we were to have hundreds of targets rather than 9 (and we probably would), we would almost certainly end up in this situation.
@MaxGhenis, can you elaborate on how we would implement this approach? I think we would need to somehow define what good weights would be, and incorporate them in the objective function. (And after doing that, it is not clear to me why it would be better to have the constraints incorporated into the objective than set out separately as constraints - the former requires us to deal with scale issues and possibly assign relative importance to constraints, and the latter does not.)
MaxGhenis
feenberg
All CDF comparisons we've looked at so far conflate two factors:
So far we've been synthesizing weight as any other feature; @donboyd5 has been making it one of the first in the synthpuf sequence, while I've been making it the last in sequential random forests. As the most important feature, it may deserve special treatment.
Per the PUF handbook:
We have a few options:
There may be others. IMO we should consider separating this problem from the problem of record synthesis, which should be evaluated on record-level similarity.