Generate hypothesis

[vob2]

Now that we have the main effect, let's crowdsource ideas for what might be driving delays (both increasing and decreasing) in projects.

Once we have an a theory, what are possible ways of testing it (how would we slice the sample, find a new data, etc)?

[JNing0]

In the data, there is a column "Performance-based Service Acquisition". A PBSA:

1. Describes the requirements in terms of results required rather than the methods of performance of the work
2. Uses measurable performance standards (i.e. terms of quality, timelines, quantity etc.) and quality assurance surveillance plans (refer to 46.103(a) and 46.401(a))
3. Specifies procedures for reductions of fee or for reductions to the price of a fixed-price contract when services are not performed or do not meet contract requirements (refer to 46.407)
4. Includes performance incentives where applicable.

The entry for PBSA is "Yes" for contract actions for services where: For FY2004 and prior - 80% of the requirement is performance based, as measured in dollars; For FY 2005 and later - 50% of the requirement is performance based, as measured in dollars.

Here is a hypothesis: If the delays are truly driven by the moral hazard, then QuickPay should have lower impact, if at all, in contracts that are performance-based.

[JNing0]

BTW: Here is a convenient data dictionary that explains all entries the data set.

In the "Content" box on the right, Section 6 "Contract information" is particularly relevant to us. Definition of "Performance-based service aquisition" is item 6.6.

[vibhuti6]

Below is a summary of the number of contracts under performance-based acquisition in our sample. Code here

	performance_based_service_acquisition_code	contracting_officers_determination_of_business_size	Number of contracts
0	N	OTHER THAN SMALL BUSINESS	57012
1	N	SMALL BUSINESS	69939
2	X	OTHER THAN SMALL BUSINESS	229008
3	X	SMALL BUSINESS	687989
4	Y	OTHER THAN SMALL BUSINESS	80614
5	Y	SMALL BUSINESS	100660

Key:		performance_based_service_acquisition_code	performance_based_service_acquisition
0	X	NOT APPLICABLE
1	N	NO - SERVICE WHERE PBA IS NOT USED.
2	Y	YES - SERVICE WHERE PBA IS USED.

[JNing0]

Great! We do have sizable samples with performance-based contracts, small business and non-small business.

Could you please draw the average delay plots of the groups with and without performance-based contracts as you did here? Let's have a figure that compares the small businesses with and without performance-based contracts.

We can also run simple DiD models around April 2011 using the large businesses as control. Let's do DiD in two sub-samples separately, one sub-sample with performance-based contracts (rows labeled 4 and 5 in your table) and the other without performance-based contracts (rows labeled 2 and 3 in your table).

Thanks!

[vibhuti6]

Hi Jie, I have posted the plots here. And the corresponding DiD results for the 2009-2012 sample are here. Thanks.

[vob2]

I added a few notes on why reduction in government's payment delay could result in suppliers expediting other projects https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Theories%20of%20expediting%20projects%202020-06-29.pdf

[vob2]

https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Theories%20of%20expediting%20projects%202020-06-29.pdf

[JNing0]

[vob2]

See related comments #29

Good work, Jie! Let me see if I understood your ideas correctly: Supplier has a liquidity shock at time T.

Without QuickPay, the supplier must choose which project to prioritize because it cannot complete both before the liquidity shock.

With QuickPay, the supplier can complete both before the liquidity shock. So, after QuickPay, the supplier gives priority to a more profitable project. Right?

What is not clear is why the supplier would give priority to the Government project without QuickPay. Sure, it is less risk, but project P might succeed as well. In fact, it is more likely to succeed if the supplier prioritizes it, according to the assumption about the survival rate.

A few comments: (1) There is no need for the survival rate as a function of time feature. You will get the same result even if project P has a fixed failure rate. (2) There needs to be more nuanced analysis on why the government project would be chosen without QuickPay (e.g., bankruptcy cost, loss of government project as well, etc). (3) For this to work, the 15-20 days that QuickPay saves the supplier must be critical for being able to complete both projects. This is OK as a first stab at the model, but given that most projects are multi-year, this seems unlikely. We need a more refined story.

But overall direction is good. Let's continue accumulating theories.

Response from Jie: Hi Vlad @vob2, I did not explain it clearly. Project P always succeeds, but the supplier may not get the payment because the customer may default with probability 1-exp(-st). That is, the further into the future, the more likely that the customer may default, which reflects the increasing uncertainty with the forecast horizon. So without QuickPay, to guarantee that it stays in business, the supplier works on the project with guaranteed payment, i.e., Project G.

This is an example, which I completely agree that we need to modify. The basic idea is that the payment from private projects is high but may not realize due to the default risk and that the supplier has a fixed financing deadline. So without QuickPay, the government project that generates safe payment is prioritized. With QuickPay, it is pushed back because the supplier can finish it after finishing the more profitable private project and still make it to the financing deadline.

I will move it to the right issue.

[vob2]

Yes, the troublesome part is that two/three weeks of expedited payment make a difference for the choice of a multi-year project.

Can we go to another extreme and say that the supplier constantly faces liquidity demands. Can the story be made to work in this case?

[JNing0]

I think so. I think a multi-year project is likely to have a number of payment during its course, as we know from the "obligations" in the data. So the liquidity shock can be periodic, likely monthly facility rent, utility, salary, loan interests, etc. So in each billing cycle the supplier faces this problem.

[JNing0]

Hi all @QuickPay-Operational-Performance/quickpay-team Below are some hypotheses about small-business suppliers based on the default risk and project portfolio idea that we can test using the data.

Hypothesis 1: Effect of contract financing.

• Hypothesis 1a: Small businesses that receive contract financing have shorter delay under QuickPay than those that don't.
• Reasoning: With contract financing, the suppliers are less dependent on the payment for invoices to deal to financing needs. So it may not prioritize government project before QuickPay in the first place, which means no delay. Furthermore, forms of contract financing such as "Progress Payments" (Code A), "Percentage of Completion Progress Payments" (Code C), "Unusual Progress Payments or Advance Payments" (Code D) may be a counteractive force that works against delays, because "progress" is directly related to the time of the project.
• Hypothesis 1b: Small businesses that receive contract financing have longer delay under QuickPay than those that don't.
• Reasoning: We don't know how the government grants contract financing. If a criterion for receiving contract financing is that the supplier is truly financially constrained, then the ones receiving contract financing are essentially in a different group than the ones without it. In this case, receiving contract financing does not imply less dependence on government invoice payment. It may be the opposite.
• COMMENT: Without knowing the guidelines of granting contract financing, it is unclear what to expect. But we may be able to use "Progress Payments" (Code A) and "Percentage of Completion Progress Payments" (Code C) in contract financing. This leads to the following idea.

Hypothesis 1.1: Effect of "Progress Payments" (Code A) and "Percentage of Completion Progress Payments" (Code C).

• Hypothesis 1.1a: Of all small businesses that receive contract financing, those who receive payments based on progress have shorter delay after QuickPay than those who receive other forms of contract financing.
• COMMENT: I am not sure about "Unusual progress payments or advance payments." That's why I left it out in the hypothesis.

Hypothesis 2: Effect of the weight of government business in a small-business supplier portfolio.

• Hypothesis 2a: The delay caused by QuickPay increases with the weight of government business.
• Reasoning: When the government business is a tiny portion of the business portfolio, its payoff is unlikely to be enough to cover the operating expenses. So it is not prioritized before QuickPay and we should see small effect of QuickPay in delay. When the weight of government business is high, the opposite happens. It is likely to be prioritized before QuickPay so we should see delay after QuickPay.
• Hypothesis 2b: The delay caused by QuickPay decreases with the weight of government business.
• Reasoning: The less weight of the private-sector customer in the portfolio, the less important the default risk is. To maximize the total expected profit, the small-business supplier trades off the time value of money from the government projects with the payment from the private customer and do not delay the government project as much.
• COMMENT: I suspect that the effect of the weight of government business may not monotone. When the weight is from low to moderate, I expect the forces under Hypothesis 2a to be at work. When the weight is from moderate to high, I expect the forces under Hypothesis 2b to be at work.

Hypothesis 3: Effect of government project weight on projects with performance-based contract.

• COMMENT: We already know that the first-order effect of QuickPay on performance-based contract is the same as non-performance-based contract, i.e., QuickPay results in project delays. The earlier idea explains the longer delay of performance-based projects from the positive and negative incentives about the quality of the project. Under QuickPay, the small-business supplier gains more time to improve the quality of the project to improve payoff or avoid loss. Thus, the negative and positive incentives of performance-based contract shift the delay to the longer end. This shift is orthogonal to the financing needs of the supplier. Therefore, the effect of government project weight on performance-based projects should be the same as in Hypothesis 2, only that the effect is stronger.
• Hypothesis 3a: If Hypothesis 2a is true, then the same relationship should apply for performance-based contracts.
• Hypothesis 3b: If Hypothesis 2b is true, then the same relationship should apply for performance-based contracts.

[vob2]

Thanks Jie. For contract financing, do we know how it works? Is the government paying sooner? Is the government giving a loan? Is the government underwriting a loan for the supplier from a financial institution? What is the theory for how contract financing should affect delays? What are the details?

[vob2]

For hypothesis 3, why does small business supplier gain more time to improve product quality under QuickPay?

[JNing0]

Thank you for the questions, Vlad! I created a wiki page on contract financing. There are several methods of contract financing, some of which such as progress payments, advance payments, and performance-based payments, involves the government paying sooner. These early payments are liquidated through the course of the project so that the supplier does not owe the government when the project is complete. Contract financing also includes government providing loan guarantees.

Effects of contract financing on delays From the FAR, contract financing payments are made on the 30th day after receiving the financing request. From the Memorandum M-11-32, QuickPay applies to Prompt Payment, which is not a form of contract financing payment. Thus, QuickPay does not affect when contract financing payments are made. This means that if the business receives contract financing based on its progress, e.g., Progress Payments, then delaying the government project would delay its receipt of the financing payments. That is a counter force against delaying. This leads me to Hypothesis 1.1 above.

Effect of QuickPay on quality improvement efforts A cash-constrained supplier under a performance-based contract has two objectives: to finish the project to get paid early, and to improve the quality to get paid more. Intuitively, improving quality needs more time and effort, especially for a small business. With QuickPay, these businesses essentially earn an extra two weeks (15 days) to work on the project without delaying the receipt of the payment. So they take their time to improve the quality of the project.

[vob2]

Thanks, Jie! This is helpful.

I think I understand the logic behind quality improvement efforts theory of the QuickPay. Here is a distilled version of it. Please correct me if I got it wrong.

Supplier to the gov has a liability due on date L. If the supplier finishes the project and submits an invoice on date T and payment delay is \tau, then the supplier will receive a payment on T+\tau. To meet the liability date, the supplier must ensure that

$$ T+\tau\leq L $$

The quality of the delivered project is a function of T. The longer the supplier works on it the higher the quality. Let's say the quality affect the probability that finished project is approved is p(T), increasing in T (concave?). We might also need c(T) the cost of exerting effort, but maybe this is not necessary.

What is important is that there is optimal, unconstrained T^0. If T^0<L-\tau then supplier will deliver project too soon, relative to the optimal time. So, if \tau is reduced, the deliveries will happen closer to T^0 and later than the original date L-\tau.

Theoretically, we need to specify a simple model for determining T^0. Econometrically, we need proxies for relationship between L and T^0 (e.g., firms that are financially constrained probably have L sooner and the effect of missing L are greater).

The advanced payments is a good way of testing if this theory has explanatory power. Similar for percentage of gov business in the supplier's portfolio.

Even though projects are delayed, the quality of the projects (project acceptance rate) should be higher. Is there a way for us to measure that? Are there projects where delaying completion would not affect the quality very much? By using those as a control we could test this theory, because the theory would not apply to them. I would guess that certain projects are pretty standard, while others require creative solutions. Creative projects should depend on T more.

Note, the theory does not require the supplier to have other projects. So this is different from previous theories we discussed and similar to the ideas Harish presented (@harishk05 did you have a chance to write those up?).

[JNing0]

Thank you for the nice summary, Vlad. I have one minor modification. Instead of modeling the approval probability of the finished project, we can use the project payment to better reflect the positive and negative incentives of performance-based contract. Let R(q) denote the revenue of the project when the quality is q and R is an increasing function. Given that the small-business supplier needs cash, I think it is reasonable to assume that the supplier makes sure that the project is accepted so that it receives some revenue. Improving the quality increases the revenue further.

By "T^0<L-tau", I guess you meant "T0>L-tau"?

A critical assumption in this theory is that quality q is an increasing function of T. It may be true for small businesses who have limited workforce and when the quality is NOT measured in terms of the timeliness of project. Unfortunately, we don't have information on how "quality" is specified for a contract.

We do need the portfolio idea of the supplier here. Because we need to explain why the small-business suppliers under performance-based contracts have MORE delay under QuickPay than those who are not. The quality conjecture explains the additional delay.

[vob2]

Yes, Jie, good catch. I meant that if the optimal T^0>L-\tau, and the optimization problem in T is concave, the firm will choose T = L-\tau because this gives the highest profit, under constraints.

I do not see why we need a portfolio with performance-based contracts. If a firm works on a single project, and this project's payment is a function of quality, there is optimal T^0 and the constraints that keep the firm away from reaching it. What is different for performance-based contracts? Or perhaps the right question is if revenue R(q) depends on quality, isn't every contract a performance based contract?

In this way of thinking what is the difference between performance-based and other contracts?

[vibhuti6]

Hi Jie and Vlad, thanks very much for posting this helpful discussion -- I am still going through it and will follow up if I have any questions.

@vob2 , just wanted to mention that Harish had posted his ideas here last week. I think he mentioned it in the wiki but it probably didn't show up in the notification emails.

Also, I will post the results for the sales-to-obligation regressions later today. Thanks.

[JNing0]

@vob2 I did not explain it clearly. We observed that small businesses who are under performance-based contract delays more under QuickPay than those who are not under performance-based contract. We want to explain why. There are two separate mechanisms for delay. (1) A small business who has a portfolio of projects may delay the government project under QuickPay. (2) A small business who has a performance-based project with the government may delay the project under QuickPay. Let's say the expected delay due to the portfolio effect is D_p and the expected delay due to the performance-based project is D_f. Then a small business who has a project portfolio and a performance-based project would have delay D_p + D_f under QuickPay, which is longer than one who only has a project portfolio and thus has delay D_p. This is an explanation for the empirical observation.

The assumption that R(q) is increasing in q is about the revenue from a performance-based contract. It arises from the negative and positive incentives from the contract. I did not mean that R(q) increases in q for all projects. In fact, for non-performance-based projects, I would conjecture that the project revenue is fixed once the quality passes the acceptance bar.

[vob2]

Thanks, Jie. Here is a simpler explanation:

Non-performance based contract: Revenue is R(q) = R if q\geq T_n and 0 otherwise. The optimal time = quality is T = T_n Performance based contract: Revenue is R(q) is increasing concave in q. Assuming suitable cost of quality, there is T^0=T_p that is optimal.

Assumption: T_n<T_p Probably true if non-performance based projects are just about meeting min quality requirement, whereas performance based ones have quality and revenue variation. But still an assumption, not a fact.

When \tau decreases, for non-performance based contracts, suppliers will increase T less to reach T_n, compared with performance based contracts where they stop after reaching T_p>T_n.

No need for a portfolio. I am not saying that portfolio explanation is not correct, but trying to generate as many simple explanations as possible to understand what can be happening and be able to tease them apart.

[vob2]

Collated theories we have so far (from Harish and Jie, as interpreted by me; please correct me if I got them wrong).

https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Theories%20of%20delays%202020-07-21.pdf

[harishk05]

I have updated my notes here: https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Model%20to%20explain%20early%20payment%20trade-off.pdf. Sorry about the delay. The idea is a combination of theory 1 and theory 3 in your note. Also, assume that small firms are more likely to be financially constrained (and face a binding "liability" constraint) and large firms do not. This will then lead to the hypotheses that Quickpay leads to small firms taking longer to finish projects and large firms taking less time. This is consistent with what we see in the data.

[harishk05]

As discussed at our last meeting, I have combined theory 1 and theory 3 to have one theory that combines (1) the effect of strategic complementarity between payment time and project time and (2) the liability constraint. I have updated my notes here: https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Model%20to%20explain%20early%20payment%20trade-off.pdf. Please review and comment. Thanks.

[vob2]

Thanks! Makes sense

[JNing0]

I have uploaded another portfolio theory here. Please review and comment. Thanks!

[vob2]

Thank you, Jie!

Yes, this makes sense. You are correct: the critical assumption is that QuickPay 15 days reduction allowed firms to meet deadlines on both government and private projects, whereas without QuickPay they could only meet one of the deadlines. I still find this assumption hard to accept, but my opinion is not based on empirical facts, so it is just that, an opinion. You are probably right.

In my handwritten notes from 2020-07-21, I have Theory 4 --- the critical path to loan repayment. I think it is essentially the same as the theory you just posted, except instead of assuming a hard deadline, in that theory later payments come at costs, and instead of selecting PG or GP sequence of projects, we allow the supplier to work a little on both projects alternatively, depending which project is on the critical path of paying off loans (in fact when documenting Theory 4, I just summarizing your earlier ideas that became the theory you just posted). Do you agree? If they are different, do you have any thoughts on what might we test empirically to tell them apart? I like Theory 4 better, because instead of all-or-nothing costs of delays, it has continuous costs. Therefore, no need to make an assumption about 15 days being critical. In fact Theory 4 would work if the reduction were 10 days or 5 days (it is just the effect would be harder to detect empirically).

Re: your comment about trucking firms. Whose bankruptcy went down? Trucking firms that had to cut their payment times or their suppliers who received payments sooner? I assume the latter.

[vob2]

I jotted down notes on the strategic explanation of QuickPay delays in the Wiki: https://github.com/QuickPay-Operational-Performance/Data-and-code/wiki/Theory-and-Hypotheses

@harishk05 please take a look as you are coming up with the grand unified theory. Everyone, please comment.

At a high level, I think this is OK. But to actually write down a mathematical model requires a bit of work.

[JNing0]

@vob2 About the question on the trucking firm: The trade credit given by the trucking firm to its customers is restricted to within 60 days under the law. As a result, the bankruptcy rate of the trucking firm reduced.

I don't quite understand theory 4. What does "Pre-payment is not allowed" mean? My understanding is that the loan is a fixed-term loan, i.e., it has a fixed calendar date as the deadline. This is reasonable as it allows the bank to lock the interest amount. This happens in e.g., some mortgage loans.

But it does not seem to be the case in the theory 4, where the payment term floats with the maximum time it takes to finish both projects. This implies that the supplier can pay off the loan any time before the deadline, which seems to be infinity.

I feel theory 4 truly assumes that only one payment is allowed to pay off the loan. It feels a bit at odds with the fact that the bank allows the firm to pay down the loan early (if it finishes the projects early), which suggests that the bank does not care about locking in the loan interest. Then why doesn't the bank allow multiple payments to pay down the principal gradually? Do we have empirical evidence of such practice?

The model I proposed today is a very simple one. The supplier is not allowed to change the time to finish the projects. It can only change the sequence. We should be able to embellish the model, I think, so that the completion times becomes a continuous decision of the supplier. Maybe along the similar lines as theory 4?

[vob2]

Thanks, got it!

Pre-payment is not allowed means that the firm cannot reduce the principal amount in several steps. So if a supplier firm receives a payment from a customer, it cannot reduce the loan principal and with it the costs of the loan. Perhaps this is not the right term. What we need is that the firm cannot pay the principal amount of the loan in several steps. This requires a renegotiation of the loan and there are costs of a renegotiation.

The key element we want to capture is that there is critical path towards reducing financing costs (these could be penalties for being late or just interest on the loan outstanding). When government expedites payments it takes government project off the critical path of tasks towards loan repayment and the supplier gives priority to the project on the critical path instead delaying working on the government project.

Both versions of the theory should produce the same predictions (hence my question if we can tell them apart empirically). But I find assumption that 15 days makes a difference in whether a supplier can finish just one or both projects hard to believe. Somehow assumption that there is critical path of paying off loan is more palatable to me.

[vob2]

I posted another explanation for project delays after QuickPay on Wiki: https://github.com/QuickPay-Operational-Performance/Data-and-code/wiki/Theory-and-Hypotheses

I am quoting here.

[Vlad:Congestion based explanation for project completion delays due to QuickPay] The following is another explanation for why delays might happen (although it is not obvious that on average we will observe more delays---have to think about that).
It is related to the explanation from the portfolio of projects, but does not really require us to argue which project receives priority and how constraints limit the optimal choices.

The logic chain is as follows: QuickPay increases resources of the firms that receive it (financial resources that might be converted into machines, labor, management hours, etc). With increased resources, the capacity (rate of processing tasks over time) of the firm increases. Increased capacity may make it attractive to take on additional projects. Additional projects increase congestion and slow down completion of all tasks, even the ones from the original project.

A Queueing model is probably appropriate here. Imagine that tasks on project 1 arrive with rate \lambda_1. The original capacity of the server is \mu_a. The objective is the expected revenues per unit of time R\lambda - H W, where W is the expected waiting time of a task.
There is another project with rate \lambda_2. But the firm finds it suboptimal to add it to the portfolio.

After QuickPay, the firm increases capacity to \mu_b. With additional capacity, it now finds that having two projects with combined rate (\lambda_1+\lambda_2) more profitable and takes on project 2. But the expected wait time for all projects increases. In particular the wait time for the original project increases and we observe project delays.

I have used M/M/1 model to find an instance that fits this story.
Mathematica file is here: https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/codes/Congestion-based-delays.nb PDF file of the code is here: https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Congestion-based-delays.pdf

As I mentioned, it is not clear if on average we should see this behavior dominating the reduction in wait times due to increased \mu.

To see if this explanation might be plausible empirically, we can check if Treated firms after QuickPay had more projects with the government or had larger Sales (Assuming Sales increased because there are more projects and not because the payment per project is higher).

[harishk05]

Added notes on Theories and Hypotheses, please see here: https://github.com/QuickPay-Operational-Performance/Data-and-code/blob/master/notes/Theories%20and%20Hypotheses.pdf

It may be a bit dense; will explain tomorrow.