Treatment intensity model

[vibhuti6]

Hi all,

I've been thinking about the treatment intensity model that we consider. I posted three potential specifications here and my understanding of their differences.

I think the correct model should be Model 1 in there, but I might be missing something. Could you please review these notes sometime, and let me know your thoughts or comments? Thanks!

[vob2]

Thanks, Vibhuti! I am convinced by your argument that Model 1 is the way to go. @JNing0 @harishk05 what do you think?

[JNing0]

Thanks, Vibhuti! Here are my two cents.

• The basic idea is to have treatment as a continuous variable, hence the name "treatment intensity." From this perspective, the rhos should not include all government projects. It should include small-business projects only. -- An important benefit of having the "treatment intensity" is that it provides convincing evidence that the delay is not caused by other policies that are not considered in the model, as the intensity of such policies is unlikely to be completely aligned with that of QuickPay. If we use the "business reliance" metric, then we lose this interpretation. -- Actually I am not sure how to interpret the results with the "business reliance" metric. What does the interaction between treatment and business reliance tell us?

• Having the intensity or the reliance in the baseline has an overfitting problem. The reason is that it is a firm-level + time-level metric. So its effect should already to absorbed in the firm and time fixed effects. Why do we still have it in the baseline?

• So I think we should use the model I proposed earlier, not Model 1 or 2 or 3. We should use the treatment intensity that considers only small businesses, not the business reliance. Rho appears only in the interaction term with treatment.

[vob2]

Good discussion, all. If Jie has 2 cents, I have 1 cent. Here it is.

If I read Vibhuti's model 1 correctly, this is a DDD model (http://www.eief.it/files/2011/10/slides_6_diffindiffs.pdf) with all possible variables and interactions present, except those absorbed by fixed effects (e.g., Post gets absorbed by time fixed effects). If so, it is saying that a reaction to the policy (expedited payment) depend on the reliance on gov. This accounts for the fact that there are difference between firms based on reliance on gov and there are time trends in how this reliance changes. If so, DDD is just giving us a more robust estimate on the effect of treatment and I think it gives us ability to argue that we are seeing effect of quickpay.

If there was another law coincidental with QuickPay but uncorrelated with firms' reliance on gov (which is what we want to say that continuous variable helps us to eliminate effects from another law, right), then this other law should not be correlated with the reliance variable, regardless if this is for small or large firms. I am not sure but I do not see why there is a problem with interpreting results.

Good point on not overfitting. Reliance \rho does change in time and firm, would it be absorbed in firm + time FEs? If it does then we remove terms beta_3 and possibly beta_4. But I am not sure it gets absorbed.

Jie, I noticed your model there are no firm FE? Should there be?

[JNing0]

@vob2 Yes, there should be firm FE in that model. The key points I want to make with that model are:

• The use of treatment intensity
• The absence of treatment intensity in the baseline

I don't think Model 1 is DDD. DDD requires another country where QuickPay is never implemented. (Contracts within the private sector in the US are likely to be indirectly affected by QuickPay.) Model 1 is a continuous version of our contract financing model. If you replace the continuous reliance variable with a dummy variable that indicates whether the reliance is high or low, you get the contract financing model. (Note that the CF dummy in our contract financing model is a project-level variable.)

I see two issues here. One is about overfitting, whether we should have the reliance or intensity in the baseline. The other is modeling choice: whether we should use reliance or treatment intensity.

For the first one, my opinion is that it does get absorbed, as both firm and time are FEs. That is probably why I have not seen it.

For the second one, I think treatment intensity makes more sense and is what we need to achieve the robustness effect discussed above. The reason is that the treatment intensity is fully aligned with QuickPay but government reliance is not. We could have firm A and firm B with the same government reliance metric, but firm A has much more weight in the small businesses than firm B. If there was a law that applies to all government projects, then the government reliance metric cannot tease it not. On the other hand, the treatment intensity can as it is tailored to fit the specificities of QuickPay which applies only to small business. So firm A has significantly higher treatment intensity than firm B.

Continue with the example I gave above, one would expect QuickPay to have a higher impact on firm A than firm B. To capture this, again we should use treatment intensity, because firm A and firm B will be exactly the same with the reliance metric.

We can definitely use the government reliance as a dimension in our analysis. But we need to first develop theories/hypothesis about it. Our theories so far do not provide predictions on the effect of reliance on government. In the portfolio model, for example, the government project is affected by QuickPay. I don't think we have discussed the version where the portfolio has private customer, government projects that are affected by QuickPay, government projects that are not affected by QuickPay.

For the reasons above, I am not sure how to interpret the results using government reliance.

[vibhuti6]

HI Jie and Vlad, thanks very much for your helpful inputs.

Jie, I just realized that I misread the model that you proposed. I was considering the sum of all obligations in the numerator whereas it should only be the obligations from small projects. My bad! I will update the results with this definition, and think some more about how its interpretation differs from the business reliance metric.

But as such, my understanding is that we still need to have the treatment intensity/business reliance metric in the baseline. It is not absorbed by time and firm fixed effects. The reason is the following. Time fixed effects control for changes over time that are constant across other factors. Likewise, firm fixed effects control for changes across firms that are constant across time.

The metric for treatment intensity/business reliance, however, varies across both dimensions: firms and time. So the same firm will take different values for \rho at different points in time. And at any given point in time, different firms will have different values of rho. This is why I think we need to have this metric in the baseline, and not just in the interaction.

[JNing0]

Great. We are converging.

@vibhuti6 Please, use the following slightly modified version of the treatment intensity. I just realized that with the original definition, a large business project in the control group may has positive treatment intensity if its owner firm also has small business projects. The change below makes sure that only treated projects have nonzero treatment intensity.

With rho_it defined as above, it may have a collinearity problem with the Treat_i dummy as all non-small business projects have rho_it=0 and Treat_i=0 so the correlation between the two covariates is likely to be high. For this reason and for simplicity, let's simply replace Treat_i dummy with the intensity rho_it, like we did at the very beginning.

Vibhuti, could you please run this model above? Thank you!

[vibhuti6]

Thanks Jie, I will get back to you on this.

[vob2]

Great! Nice to have equations. Fewer chances for misunderstandings. Obviously, project i belongs to firm j

[vibhuti6]

Hi all,

Here are the results from the treatment intensity model based on our discussion today: https://github.com/QuickPay-Operational-Performance/Data-and-code/wiki/Results_Treatment_Intensity_2009_2012

I will add them to the paper once we finalize the specification. Thanks!

[vibhuti6]

Also, just an FYI: I have merged the changes from my branch into the master branch of the paper repository. Thanks.

[JNing0]

Thank you, Vibhuti! What's the difference between (1) and (3)? What's the difference between (2) and (4)? It seems that the sample size is different?

[vibhuti6]

Hi Jie, in columns (3) and (4), I removed the observations that were treated but had a value of rho <= 0. Because if we didn't have any data quality issues, rho for treated observations would always be positive. Ideally it should also be less than one but I didn't truncate those values. Sorry I forgot to mention this in the main text. It is currently only stated in the table notes but I will update the page.

[JNing0]

I see. sorry I missed the notes in the table.

If we are dropping the negative values, then we should probably also drop the values with rho>=1. Could you please also run the model with the truncated data? Also, it would be nice if we can have a visual of rho, e.g., a histogram or a boxplot.

[vibhuti6]

Thanks, Jie -- I have updated the page with the new regressions (columns 5 and 6) and the box plot. Please see here: https://github.com/QuickPay-Operational-Performance/Data-and-code/wiki/Results_Treatment_Intensity_2009_2012

[vob2]

Hmm, I guess intensity variable did not work out. That's too bad. Now we will have to work harder to eliminate the possibility that there is no other event at the same time that could have affected small and large firms differently and their project delays.

[harishk05]

Interesting, thanks.

[JNing0]

Thank you, Vibhuti.

From the boxplot, more than 25% of the rho values is greater than one. So we lost lots of information by dropping them, and by dropping the observation with negative rho values. This reduces our statistical power and increases the estimated error. So it is not surprising that we do not have statistical significant estimates.

On the other hand, if we keep the observations with rho >>1 as they are, then they will have extreme effects on our estimates.

Here is a thought. The absolute value of rho is not a good proxy for treatment intensity. But I think there is information in the ranking of the rho values. For example, a high value of rho probably means a higher treatment intensity than a low value of rho. So I propose that we take the rank order from the computed rho values and use that as the treatment intensity.

Specifically, use the notation that rho_ft is the treatment intensity of firm f at time t. Let Nt denote the total number of firms in the treatment group at time t. For each t, we sort all Nt values of rho_ft in increasing order. Set the lowest value of rho_ft the treatment intensity 1/Nt, the second lowest with treatment intensity 2/Nt, ..., the highest value with treatment intensity 1. So essentially we find the empirical distribution of rho_ft.

This approach allows us to use all the data, eliminate the extreme effects of extremely high rho values, and keep the rank order of rho.

This is the continuous version of the tercile idea. So to push the treatment intensity analysis forward, we can

• run the model with the empirical distribution of rho_ft. In this case, the treatment intensity is continuous.
• run the model with a categorical variable based on rho_ft, e.g., high/low treatment intensity, or high/med/low treatment intensity. In this case, the treatment intensity is discrete.

[vibhuti6]

Thanks Jie, I will get back to you on this.

[JNing0]

Hi all, I have some thoughts about modifying the treatment intensity model. Please see details here.

[vob2]

Thank you very much, Jie, for such a detailed and clear description!

If I understood our discussion on Wednesday and this note correctly, because treatment variables (at least in the way the three estimation alternatives are described) are no longer time dependent, the concern about interpreting treatment coefficients does not apply anymore and we can read beta_2 as the effect of the QuickPay law effect.

As far as which one of the three estimation alternatives to use, the trade off seems to be between relying on the most recent (pre-law) data vs smoothing out possible spurious fluctuations in that data (not necessarily because of data errors in the dataset, but because government can give and then take away obligations). My gut feel is that using either the total over the 4 quarters before the law or the average would work better. But this is just a gut feel, and no convincing reason. Between these two, I would choose the one that matches Sales dimension. If Sales are per year, then total FAO is better. If Sales are per quarter, then average FAO is better.

A robustness test is to try the last quarter vs 4-quaters definitions and see if this makes difference in the results.

Question: By the same logic, should we consider averaging delays over 4 quarters before and after the law?

I completely agree that we need to decide what to do when data is missing for all 4 quarters. One strategy is to fill this data with the average for the quarters for which it is available. Another is treat missing data as zeros. Hard to tell which approach is better. Again, my gut feel is that using the average for missing data is better. Hopefully, this does not affect the results regardless.

Rank ordering seems like a good idea if our data looks non-elliptical. It might be a good idea to visualize data of \rho vs Delays to see if they look elliptical.

Question: Instead of doing rank ordering, how about applying transformations to \rho (e.g., using \ln(1+\rho))?

Question: Should we do rank ordering on Delays as well? This way we are estimating Spearman correlation.

If using rank ordering, I would use Model 2 because Model 1 is a special case when the number of brackets = N.

Question: When using Discrete model, should be withholding one category?

[harishk05]

Thank you Jie and Vlad for this detailed and helpful discussion.

[JNing0]

Hi Vlad, please see below for my thoughts on your questions.

Question: By the same logic, should we consider averaging delays over 4 quarters before and after the law? I don't feel strongly about averaging delays. An appeal of using the annual FAO rather than the FAO in one quarter is to cover an entire fiscal year so that our metric is less subject to the government's over-commitment and withdrawal throughout a fiscal year. In addition, the sales data is in years as well.

The measurement of delay does not appear to have such a problem. Let Dt denote the delay observed in time t. Then Dt is the projected delay in project completion at time t. (In our analysis, we set t on a quarterly basis. So we simply take a snapshot at the end of each quarter and look at the projected delay at that time.) So the variation in Dt itself is informative in how a firm views its project quality, measured by completion time, in different periods. I feel that it is better to use the Dt without aggregation and control for time fixed effects. This approach also gives us more statistical power as we have a larger data set.

Question: Instead of doing rank ordering, how about applying transformations to \rho (e.g., using \ln(1+\rho))? This is a good point. We can try as robustness checks for our results. A nice thing about using the empirical distribution is that it connects nicely with the discrete version where we divide the treatment intensity into K subsets. Also, it might be easier to interpret than using ln(1+\rho).

Question: Should we do rank ordering on Delays as well? This way we are estimating Spearman correlation. I am not so sure about rank ordering delays... The reason for rank ordering the rho values is that a significant portion (more than 25%) of the values is higher than one when we used a quarterly FAO, possibly due to government over-committing to an account. So rank ordering is a way to mitigate the effect of extreme values while preserving the relative magnitude of the rho values. (This may be less of a problem if we use annual FAO.)

Delays are firm projections, which do not have such a problem. Also, I think the numeric value of delay is actually informative. It allows us to quantify the effect of QuickPay in extending the delays.

If using rank ordering, I would use Model 2 because Model 1 is a special case when the number of brackets = N. This is not a question but I want to clarify that Model 1 is not a special case of Model 2. This is because the coefficient beta_2 is a scalar in Model 1 and is a K-dim vector in Model 2.

Question: When using Discrete model, should be withholding one category? We need to have K terms in the summation. Think about the contract financing case. The CF is a dummy that takes two values, 0 and 1. And in the summation we have two terms and we estimate two coefficients.

[vob2]

Sounds good.

[JNing0]

An unresolved issue is the interpretation of the model proposed above using \rho, which is the ratio between the sum of federal action obligation and a firm's sales. I have been calling it the "treatment intensity," but now I feel that it really measures how the treatment effect is moderated by the firm's entire business portfolio. So the model tests the portfolio theory, which is a reason for the delay under QuickPay for small businesses.

An underlying assumption of this portfolio theory is that the firm centralizes the management of all its projects. While this might be plausible for small businesses, it may not be so plausible for large businesses, where different multi-billion projects are managed individually. We need to keep this in mind where doing the portfolio analysis on the 2014-QuickPay.

I need to think more for a good measure of "treatment intensity."