decode "small" and "large" business

[JNing0]

As discussed here, right now our treatment effect, QuickPay, varies based on whether the contract is labeled as a "small" or "large" business by a government officer. It would be nice if we can open these black boxes and replace "small" and "large" labels with objective company measures such as sales, industry, PSC codes etc.

One way I can think of is to consider the treatment effect on contracts that are awarded to firms that are sometimes labeled as "large" and sometimes as "small". After controlling for other characteristics, we should be able to simply state that the "effect of QuickPay is XXX" and drop the opaque qualifiers "small" and "large" businesses. Any other thoughts?

Because the QuickPay is rolled out at different time points, the time effect may always be there. Any idea of taking it out?

[vob2]

It looks like there are several issues here. I will comment separately.

1. Treatment Ideally we would like to know that a contract (or a supplier) received a payment sooner by X number of days than control suppliers. However, we do not observe when payments actually happen.

We do know that the law applies to small firms or large firms. So we use those as "intention to treat." Question is what is used when applying QuickPay in practice: determination of a grant officer or objective characteristics? My guess is the determination of grant officer.

Let's say a grant officer decides that Boeing is small business qualifying for QuickPay. We do not want to say that Boeing is not treated because objective measures point that Boeing is large company.

[vob2]

2. QuickPay rolled out at different points of time. We should be able to handle this with generalized version of DiD where treatment happens over time, as long as we have a control that never gets treated, right? I thought we had this discussion earlier and I learned from Jie and Vibhuti why this is so

[JNing0]

@vob2 Thanks for the comment.

I am not questioning whether QuickPay is actually implemented. My understanding is that we rely on the Barrat and Nanda paper for evidence on the actual implementation of QuickPay.

I am concerned with the interpretation of our results. Our current result says that on average, implementation of QuickPay

• increases the project delay of small businesses by x days with st. dev. _sigmax.
• increases the project delay of large businesses by y days with st. dev. _sigmay.
• x and y are statistically significantly different. That is, _x + 3 sigma_x < y - 3 sigmay.

This result naturally leads to questions. Why does QuickPay have different effects on small and large businesses? How do "small businesses" differ from "large businesses"? I don't have an answer to these questions. Intuitively, if the treatment has different effects, then the two treatment groups must be different. But how and why, especially given that we have controlled for a variety of firm and project characteristics such as size, industry, PSC code in our analysis?

I don't think (I may be wrong) that we can attribute the different treatment effects for small and large business to the fact that small and large businesses are treated at different times. I think we have controlled for time fixed-effect in the model. If so, then everything related to time should have been teased out.

To see why this can be problematic, suppose firm Z has two projects B and S. Firm Z is characterized as "big business" for project B and "small business" for project S. Then our result states that QuickPay affects the behavior of the same firm, firm Z, differently, on the two projects. How do we rationalize this result?

Ideally, we should be able to make statements like "QuickPay on average leads to a delay of x days for a business that is financially constrained and y days for a business that is not financially constrained." We are not there yet. But we have not been careful with the controls we use. I hope that once we control more for project and firm specifics, the treatment effects in 2011 and 2014 are not statistically significantly different.

[vob2]

Thanks, Jie (@JNing0 ).

First, I love the intuitive notation B and S.

Second, I interpreted your original comment as saying that you want to change the treatment variable. That will not work. But if you are saying we should add controls and use other proxies for being financially constrained, then my comment does not apply. Sure we should have controls. But let's make sure we are talking about the same apples. Are you worried about Treatment or about Financial Constrains.

For financial constraints, we must have something not perfectly correlated with firms being Large and Small, because if we already use Large and Small as Treated Group indicator. That is why I hope that other variables, where officers indicate that business is disadvantaged and receives financing can be used to indicate the presence of financial constraints. These seem not be perfectly correlated with Large and Small designation.

Third, your question "why does QuickPay have different effect on Big and Small businesses" is difficult. When Quickpay is first applied to Small businesses, we use Big ones as control. When QuickPay later also applies to Big businesses, it is already in force for Small businesses, so parity is restored, but we might not have parallel trend assumptions going into this experiment. So while I trust the first experiment (on Small businesses) result, the second one (on Big businesses) is a suspect.

Also, we do not want to answer question why Big and Small firms react differently. We want to explain fundamentally how any firm should react when payments are promised sooner. So far, we identified financial constraints as one force. But there is also project portfolio force. Large and small firm probably have different portfolios of projects. There might be other forces. We need to develop theory more. Again, we should not focus on Big vs Small reaction to QuickPay (other than treating Big as control for Small in DiD). We have other characteristics of firms. We will have more theoretical predictions than possible empirical tests.

Forth, I understand your comment that there is a problem when firm has contracts that are classified as large and small, how do we use this to classify this firm. But my point is that we do not want to use Large and Small as a financial constraint indicator. If so, the question is moot, right. Or does this affect something else?

[vibhuti6]

Hi Jie and Vlad, thanks for this helpful discussion. I wanted to quickly mention a couple of things about the second analysis - when Quickpay was implemented for large businesses:

• It is true that parallel trends may not hold because small businesses have already been treated. In fact, large businesses were also treated for a brief time window from July 2012 to Feb 2013. To avoid some of the issues in trend created by this, our current analysis only focuses on contracts that were not active before March 2013.

• Because the treatment variable is flipped in this time period, we run a reverse difference-in-differences analysis. This approach is also suggested in a seminal DiD paper by Card and Krueger: https://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.90.5.1397 See Section II(A) on page 1406.

I agree that there are still several issues that we need to address -- particularly with why we see different impact on small and large businesses. But just wanted to clarify this bit about our current analysis. Thanks.

[JNing0]

I think I know where I tripped over. Thinking of the effect of QuickPay on small and large business is a trap. It led me to think that the difference in the treatment effect is caused by the difference in the treatment group. Then it follows that if we control for everything perfectly in the treatment groups, we should see the same treatment effect, which is not what we have.

The right way to understand this, I think, is that it is the treatment, rather than the treatment group, that is different.

To clarify the thinking and discussion, I suggest that we drop the notion of small and large businesses. In our analysis, we examine two implementations of QuickPay, at 2011 and 2014. The effects of 2011-treatment and the 2014-treatment differ because

• Businesses may respond to them differently in 2014 than in 2011. This difference in response cannot be accounted for by time fixed-effects. We could potentially quantify it by considering firms that are affected in both treatments.
• The 2011-treatment affects only a fraction of the suppliers whereas the 2014-treatment affects all suppliers. Given the limited amount of government funding, it is plausible that the actual implementation of QuickPay differs in 2011 and 2014. (Barrat and Nanda only verifies that the 2011-QuickPay was implemented as promised.)

Therefore, the right interpretation of our results is that:

• On average, a business delays the project by x days due to the 2011-version of QuickPay.
• On average, a business delays the project by y days due to the 2014-version of QuickPay.

[vob2]

@JNing0 . Yes, that makes sense.

I was hoping that the time differences would be picked up by time fixed effects. But by definition 2011 QuickPay applied to different firms than 2014 QuickPay.

Scale of the law might have industry equilibrium effects. That is how firms compete with each other. We have not discussed that as a possible explanation. Equilibrium where all firms are paid after 30 days is different than the equilibrium where a fraction of the firms is paid after 15 days, and is different from the equilibrium where all firms are paid after 15 days.

Not sure what this means for us telling the story though.

[JNing0]

@vob2 I agree with what you said... We can only estimate the treatment effect on the treatment group. With the data we have, we cannot carve out the effect of QuickPay on an arbitrary business. I am closing this issue.