search

vtjeng / MIPVerify.jl

Evaluating Robustness of Neural Networks with Mixed Integer Programming
MIT License
113 stars 31 forks source link

Reduce test output #129

Closed maleadt closed 1 year ago

maleadt commented 1 year ago

The tagged version of this package produces an enormous amount of output during testing:

Optimal - objective value 0.25565824
After Postsolve, objective 0.25565824, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.2556582393 - 35 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
Perturbing problem by 0.001% of 13.250513 - largest nonzero change 0.00011311144 ( 55.162455%) - largest zero change 0.00011304592
0  Obj 0.35523064 Primal inf 276.25861 (11) Dual inf 366.63573 (30)
29  Obj 0.45994785
Optimal - objective value 0.46231129
After Postsolve, objective 0.46231129, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.4623112903 - 29 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
0  Obj 0.29475826 Primal inf 276.25861 (11) Dual inf 123.19236 (56)
29  Obj 0.36961064
Optimal - objective value 0.36961064
After Postsolve, objective 0.36961064, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.3696106427 - 29 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
0  Obj 0.11558455 Primal inf 276.25861 (11) Dual inf 97.656124 (50)
32  Obj 0.18756853
Optimal - objective value 0.18756853
After Postsolve, objective 0.18756853, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.1875685272 - 32 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
0  Obj 0.21135814 Primal inf 276.25861 (11) Dual inf 138.84452 (44)
33  Obj 0.32504101
Optimal - objective value 0.32504101
After Postsolve, objective 0.32504101, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.3250410068 - 33 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
0  Obj -0.071581534 Primal inf 276.25861 (11) Dual inf 99.600807 (46)
42  Obj 0.024102245
Optimal - objective value 0.024102245
After Postsolve, objective 0.024102245, infeasibilities - dual 2.627105e-09 (1), primal 0 (0)
Presolved model was optimal, full model needs cleaning up
0  Obj 0.024102245 Dual inf 1.0041974e-05 (5)
2  Obj 0.024102337
Optimal - objective value 0.024102337
Optimal objective 0.02410233733 - 44 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
Perturbing problem by 0.001% of 7.4128592 - largest nonzero change 0.00010728096 ( 93.520456%) - largest zero change 0.00010721881
0  Obj 0.086722099 Primal inf 276.25861 (11) Dual inf 20.285288 (21)
29  Obj 0.11029348
Optimal - objective value 0.1125012
After Postsolve, objective 0.1125012, infeasibilities - dual 0 (0), primal 0 (0)
Optimal objective 0.1125012013 - 29 iterations time 0.002, Presolve 0.00
Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements
0  Obj 0.11830246 Primal inf 276.25861 (11) Dual inf 264.99704 (52)

This results in PkgEval marking this package as broken. Would it be possible to fix this such that the package can be used for evaluating Julia changes?

vtjeng commented 1 year ago

Thanks for the report! I tried doing so in the past but couldn't find the right flag to silence the solver printing the logs. Will take another stab at it this week.

Would it be possible to fix this such that the package can be used for evaluating Julia changes?

I didn't unite understand what you meant by "for evaluating Julia changes". Do you mean that we would run the test suites to provide a signal of whether a change in core Julia code was breaking?

On Tue, Feb 14, 2023, 11:14 PM Tim Besard @.***> wrote:

The tagged version of this package produces an enormous amount of output during testing:

Optimal - objective value 0.25565824 After Postsolve, objective 0.25565824, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.2556582393 - 35 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements Perturbing problem by 0.001% of 13.250513 - largest nonzero change 0.00011311144 ( 55.162455%) - largest zero change 0.00011304592 0 Obj 0.35523064 Primal inf 276.25861 (11) Dual inf 366.63573 (30) 29 Obj 0.45994785 Optimal - objective value 0.46231129 After Postsolve, objective 0.46231129, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.4623112903 - 29 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements 0 Obj 0.29475826 Primal inf 276.25861 (11) Dual inf 123.19236 (56) 29 Obj 0.36961064 Optimal - objective value 0.36961064 After Postsolve, objective 0.36961064, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.3696106427 - 29 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements 0 Obj 0.11558455 Primal inf 276.25861 (11) Dual inf 97.656124 (50) 32 Obj 0.18756853 Optimal - objective value 0.18756853 After Postsolve, objective 0.18756853, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.1875685272 - 32 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements 0 Obj 0.21135814 Primal inf 276.25861 (11) Dual inf 138.84452 (44) 33 Obj 0.32504101 Optimal - objective value 0.32504101 After Postsolve, objective 0.32504101, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.3250410068 - 33 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements 0 Obj -0.071581534 Primal inf 276.25861 (11) Dual inf 99.600807 (46) 42 Obj 0.024102245 Optimal - objective value 0.024102245 After Postsolve, objective 0.024102245, infeasibilities - dual 2.627105e-09 (1), primal 0 (0) Presolved model was optimal, full model needs cleaning up 0 Obj 0.024102245 Dual inf 1.0041974e-05 (5) 2 Obj 0.024102337 Optimal - objective value 0.024102337 Optimal objective 0.02410233733 - 44 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements Perturbing problem by 0.001% of 7.4128592 - largest nonzero change 0.00010728096 ( 93.520456%) - largest zero change 0.00010721881 0 Obj 0.086722099 Primal inf 276.25861 (11) Dual inf 20.285288 (21) 29 Obj 0.11029348 Optimal - objective value 0.1125012 After Postsolve, objective 0.1125012, infeasibilities - dual 0 (0), primal 0 (0) Optimal objective 0.1125012013 - 29 iterations time 0.002, Presolve 0.00 Presolve 111 (-821) rows, 298 (-1344) columns and 1353 (-1605) elements 0 Obj 0.11830246 Primal inf 276.25861 (11) Dual inf 264.99704 (52)

This results in PkgEval marking this package as broken. Would it be possible to fix this such that the package can be used for evaluating Julia changes?

— Reply to this email directly, view it on GitHub https://github.com/vtjeng/MIPVerify.jl/issues/129, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNIOELHA5FCJUQMTAN26CDWXR63TANCNFSM6AAAAAAU4PD7FI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

maleadt commented 1 year ago

Do you mean that we would run the test suites to provide a signal of whether a change in core Julia code was breaking?

That is the purpose of PkgEval, so you don't need to do anything except make sure that the package can be tested reasonably (i.e. without generating the amount of output it currently does) 🙂

vtjeng commented 1 year ago

131 silences the output, leaving ~200 lines from the progress bar + summary report of memory usage and runtime. Is that tolerable for PkgEval? Example CI run: https://github.com/vtjeng/MIPVerify.jl/actions/runs/4201056793/jobs/7287686573#step:6:275; total lines of output from julia-runtest is <600.

vtjeng commented 1 year ago

Seperately, Tim, I'm curious why you reached out (and definitely pleased that you did ...). Were you looking at packages with recent activity?

maleadt commented 1 year ago

No, just looking at the latest daily report, https://github.com/JuliaCI/NanosoldierReports/, and happened to bump on this package. The goal is to test as much of the ecosystem as possible, so when I have some time to kill I look at random logs and file issues in the hope we can get to test that package as well 🙂

maleadt commented 1 year ago

~200 lines from the progress bar + summary report of memory usage and runtime. Is that tolerable for PkgEval?

That should be fine, but ideally the progress is disabled when you aren't in an interactive mode (which can be done simply by setting enabled=isinteractive() to the Progress constructors, IIRC)

vtjeng commented 1 year ago

~200 lines from the progress bar + summary report of memory usage and runtime. Is that tolerable for PkgEval?

That should be fine, but ideally the progress is disabled when you aren't in an interactive mode (which can be done simply by setting enabled=isinteractive() to the Progress constructors, IIRC)

Thanks for the tip! Will cut a new release when #134 is submitted; hopefully this means my package is passing.

vtjeng commented 1 year ago

Will close this issue when I see a passing status in https://github.com/vtjeng/MIPVerify.jl/pull/135. For my convenience:

PkgEval

vtjeng commented 1 year ago

Feel free to reopen this if there is an issue.