Print and summary for DKQS (and other procedures)

a-torgovitsky commented 4 years ago

Same idea as in #26 but for the other testing procedures as well.

conroylau commented 4 years ago

I got a quick question when I am updating the print and summary messages for subsample -- should I also allow multiple phi input for the procedure? (phi is the parameter that controls the size of each subsample).

Thanks!

a-torgovitsky commented 4 years ago

This I don't think is necessary. The reason is because there are natural and easy-to-gain efficiency savings for both fsst and DKQS (less for DKQS) from looking at all of the tuning parameters together instead of in separate calls. For subsample that is not the case.

conroylau commented 4 years ago

Done for subsample - I have updated the print and summary messages. The sample output are as follows:

Print:

p-value: 0.43

Summary:

p-value: 0.43
Test statistic: 0.00213
Solver used: gurobi
Norm used: 2
Phi used: 0.66667
Number of cores used: 8

conroylau commented 4 years ago

I just updated the print and summary messages for estbounds and mincriterion.

Sample output for estbounds:

Print:
```
Estimated bounds: [0.253, 0.25642] 
```

Summary:

Estimated bounds: [0.253, 0.25642] 
Norm used: 2 
Solver: gurobi

Sample output for mincriterion:

Print:
```
Minimum value: 0
```

Summary:

Minimum value: 0 
Norm used: 2 
Solver: gurobi

Thanks!

Edit: I made a typo in the last comment, where I should have typed estbounds instead of subsample here.

a-torgovitsky commented 4 years ago

Looks good. For subsample summary I would add number of observations in each subsample.

conroylau commented 4 years ago

Sure! The updated summary message of subsample test is:

p-value: 0.47
Test statistic: 0.00055
Solver used: gurobi
Norm used: 2
Phi used: 0.66667
Size of each subsample: 99
Number of cores used: 1

Thanks!

conroylau commented 4 years ago

I have further fine-tuned the print and summary messages for invertci in order to be consistent with the other testing procedures. The original output messages for invertci can be found in issue #10.

In the updated version, print will only show the confidence interval(s) in a "table" format like the other print messages. The summary command will shows more details. In addition, I have added an argument to the summary command for invertci that users can choose which confidence interval(s) that they wish to show because the details of the iterations can potentially be a very long list of outputs.

Below are the sample output for invertci (assuming that x is the object that stores the results):

print(x):

Significance level Confidence interval
0.05                [0.29375, 0.70625]
0.1                 [0.30625, 0.70625]
0.2                 [0.33125, 0.68125]

summary(x):

Maximum number of iterations: 5
Tolerance level: 0.001

Significance level Confidence interval
0.05                [0.29375, 0.70625]
0.1                 [0.30625, 0.70625]
0.2                 [0.33125, 0.68125]

Details:

<Confidence interval for significance level = 0.05>
=== Iterations in constructing upper bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.60000            NA      0.60000   0.09000     FALSE
Right end pt.            NA       1.00000      1.00000   0.00000      TRUE
1                   0.60000       1.00000      0.80000   0.00000      TRUE
2                   0.60000       0.80000      0.70000   0.04000     FALSE
3                   0.70000       0.80000      0.75000   0.00000      TRUE
4                   0.70000       0.75000      0.72500   0.01000      TRUE
5                   0.70000       0.72500      0.71250   0.01000      TRUE
Reason for termination: Reached maximum number of iterations

=== Iterations in constructing lower bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.00000            NA      0.00000   0.00000      TRUE
Right end pt.            NA       0.40000      0.40000   1.00000     FALSE
1                   0.00000       0.40000      0.20000   0.00000      TRUE
2                   0.20000       0.40000      0.30000   0.05000     FALSE
3                   0.20000       0.30000      0.25000   0.00000      TRUE
4                   0.25000       0.30000      0.27500   0.00000      TRUE
5                   0.27500       0.30000      0.28750   0.00000      TRUE
Reason for termination: Reached maximum number of iterations

<Confidence interval for significance level = 0.1>
=== Iterations in constructing upper bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.60000            NA      0.60000   0.15000     FALSE
Right end pt.            NA       1.00000      1.00000   0.00000      TRUE
1                   0.60000       1.00000      0.80000   0.00000      TRUE
2                   0.60000       0.80000      0.70000   0.06000     FALSE
3                   0.70000       0.80000      0.75000   0.01000      TRUE
4                   0.70000       0.75000      0.72500   0.03000      TRUE
5                   0.70000       0.72500      0.71250   0.03000      TRUE
Reason for termination: Reached maximum number of iterations

=== Iterations in constructing lower bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.00000            NA      0.00000   0.00000      TRUE
Right end pt.            NA       0.40000      0.40000   1.00000     FALSE
1                   0.00000       0.40000      0.20000   0.00000      TRUE
2                   0.20000       0.40000      0.30000   0.03000      TRUE
3                   0.30000       0.40000      0.35000   0.21000     FALSE
4                   0.30000       0.35000      0.32500   0.07000     FALSE
5                   0.30000       0.32500      0.31250   0.06000     FALSE
Reason for termination: Reached maximum number of iterations

<Confidence interval for significance level = 0.2>
=== Iterations in constructing upper bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.60000            NA      0.60000   0.11000     FALSE
Right end pt.            NA       1.00000      1.00000   0.00000      TRUE
1                   0.60000       1.00000      0.80000   0.00000      TRUE
2                   0.60000       0.80000      0.70000   0.05000      TRUE
3                   0.60000       0.70000      0.65000   0.39000     FALSE
4                   0.65000       0.70000      0.67500   0.17000     FALSE
5                   0.67500       0.70000      0.68750   0.07000      TRUE
Reason for termination: Reached maximum number of iterations

=== Iterations in constructing lower bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.00000            NA      0.00000   0.00000      TRUE
Right end pt.            NA       0.40000      0.40000   1.00000     FALSE
1                   0.00000       0.40000      0.20000   0.00000      TRUE
2                   0.20000       0.40000      0.30000   0.00000      TRUE
3                   0.30000       0.40000      0.35000   0.30000     FALSE
4                   0.30000       0.35000      0.32500   0.09000      TRUE
5                   0.32500       0.35000      0.33750   0.18000     FALSE
Reason for termination: Reached maximum number of iterations

The following command can be used to only print the confidence intervals of some confidence intervals: summary(x, alphas = .05)

Maximum number of iterations: 5
Tolerance level: 0.001

Significance level Confidence interval
0.05                [0.29375, 0.70625]

Details:

<Confidence interval for significance level = 0.05>
=== Iterations in constructing upper bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.60000            NA      0.60000   0.09000     FALSE
Right end pt.            NA       1.00000      1.00000   0.00000      TRUE
1                   0.60000       1.00000      0.80000   0.00000      TRUE
2                   0.60000       0.80000      0.70000   0.04000     FALSE
3                   0.70000       0.80000      0.75000   0.00000      TRUE
4                   0.70000       0.75000      0.72500   0.01000      TRUE
5                   0.70000       0.72500      0.71250   0.01000      TRUE
Reason for termination: Reached maximum number of iterations

=== Iterations in constructing lower bound:                                                                          
Iteration       Lower bound   Upper bound   Test point   p-value   Reject?
Left end pt.        0.00000            NA      0.00000   0.00000      TRUE
Right end pt.            NA       0.40000      0.40000   1.00000     FALSE
1                   0.00000       0.40000      0.20000   0.00000      TRUE
2                   0.20000       0.40000      0.30000   0.05000     FALSE
3                   0.20000       0.30000      0.25000   0.00000      TRUE
4                   0.25000       0.30000      0.27500   0.00000      TRUE
5                   0.27500       0.30000      0.28750   0.00000      TRUE
Reason for termination: Reached maximum number of iterations

Thanks!

conroylau / lpinfer

Print and summary for DKQS (and other procedures) #34