more support for out-of-place trust-region solvers

[longemen3000]

this is focused in the NWI trust region, but TCG also supports out of place now (there was some work on NTR, but some parts are still missing)

summary of the changes:

• update_H!(H,h,lamda) -> update_H!(mstyle, H,h,lamda)
• new function: trs_supports_outofplace(trs), that turns the support for out-of-place solvers for an specific trust region method.
• dot(x, H*x) -> dot(x, H, x) (available since julia 1.4, it should reduce an allocation in the inplace methods)

[codecov[bot]]

Codecov Report

Attention: 20 lines in your changes are missing coverage. Please review.

Comparison is base (6fd621a) 76.91% compared to head (97a2dc7) 77.11%.

Files	Patch %	Lines
src/globalization/trs_solvers/solvers/NTR.jl	61.29%	12 Missing :warning:
src/globalization/trs_solvers/solvers/NWI.jl	79.16%	5 Missing :warning:
src/globalization/trs_solvers/TRS.jl	0.00%	1 Missing :warning:
src/globalization/trs_solvers/root.jl	96.29%	1 Missing :warning:
src/globalization/trs_solvers/solvers/TCG.jl	96.42%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #66 +/- ## ========================================== + Coverage 76.91% 77.11% +0.20% ========================================== Files 54 54 Lines 2807 2867 +60 ========================================== + Hits 2159 2211 +52 - Misses 648 656 +8 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[longemen3000]

The tests here pass with rosenbrock and NWI, but fails with my test case:

import Clapeyron, ForwardDiff
const C = Clapeyron

#obtain critical point of water with PC-SAFT eos
function test_critical_point()
  model = C.PCSAFT("water")
  function f_crit_static(Fx, x)
    Ts = T_scale(model,SVector(1.0))
    T_c = x[1]*Ts
    V_c = exp10(x[2])
    ∂²A∂V², ∂³A∂V³ = ∂²³f(model, V_c, T_c, SA[1.0])
    F1 = -∂²A∂V²
    F2 = -∂³A∂V³
    return SVector(F1,F2)
  end
  f_crit_static(x) = f_crit_static(nothing, x)
  j_crit_static(J,x) = ForwardDiff.jacobian(f_crit_static,x)
  fj_crit_static(F,J,x) = f_crit_static(x),j_crit_static(J,x)
  obj = NLSolvers.VectorObjective(
          f_crit_static,
          j_crit_static,
          fj_crit_static,
          nothing,
  )
  prob_static =  NLSolvers.NEqProblem(obj; inplace=false)
  x01,x02 = C.x0_crit_pure(model)
  x0_static= SVector(x01,x02)
  NLSolvers.solve(prob_static, x0_static, TrustRegion(Newton(), NWI()), NEqOptions(maxiter = 20))
end

on the allocating version: this is the output of the trust region solver:

spr = (p = [-0.119191334506046, 0.04355412894426012], mz = -4.018965005920141e36, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 20.0)
spr = (p = [-0.0027029758072131525, 0.04499612541909675], mz = -4.7551669760870206e35, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 35.0)
spr = (p = [0.018511262372664327, 0.045777571320208765], mz = -6.205904430656341e34, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 61.25)
spr = (p = [0.013106086890848268, 0.04429461125778653], mz = -8.344042103677409e33, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 107.1875)
spr = (p = [0.002156155676237826, 0.03809921614778], mz = -1.063615592722335e33, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 187.578125)
spr = (p = [-0.004636514979904277, 0.025680354545900588], mz = -1.1154169725112833e32, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 328.26171875)
spr = (p = [-0.0037736725583891557, 0.010414501514063087], mz = -6.834517914423543e30, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 574.4580078125)
spr = (p = [-0.0007013620414804606, 0.0014772805678870081], mz = -9.525776840814146e28, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1005.301513671875)
spr = (p = [-1.4438428337513423e-5, 2.7098407728549498e-5], mz = -3.2082145157529907e25, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1759.2776489257812)
spr = (p = [-5.073980186604831e-9, 9.04901950919703e-9], mz = -3.700814555101485e18, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 3078.735885620117)
spr = (p = [-7.883218328763044e-15, -7.778461704842644e-15], mz = -577.1355732863631, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 5387.787799835205)
spr = (p = [3.845471800641081e-16, 4.870267452997498e-16], mz = -256.50452171776385, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 9428.628649711609)
spr = (p = [3.607523191190752e-17, 1.5088257208636439e-16], mz = -256.5045209330882, interior = false, λ = 2.4796342977323244e25, hard_case = false, solved = false, Δ = 1.5513534097936166e-16)

whereas the out-of-place version returns:

spr = (p = [-0.119191334506046, 0.043554128944260126], mz = -4.018965005920142e36, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 20.0)
spr = (p = [-0.0027003500478230986, 0.04499632231262638], mz = -4.755166976087023e35, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 35.0)
spr = (p = [0.018510197652017554, 0.04577760432131138], mz = -6.205942098689902e34, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 61.25)
spr = (p = [0.013104649946379613, 0.04429447425986432], mz = -8.344077069894073e33, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 107.1875)
spr = (p = [0.0021561276055823263, 0.03809921508070954], mz = -1.0636125973524221e33, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 187.578125)
spr = (p = [-0.004636552983922482, 0.025680336293903983], mz = -1.1154135772229733e32, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 328.26171875)
spr = (p = [-0.0037737304612361034, 0.010414445181253269], mz = -6.834487533690134e30, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 574.4580078125)
spr = (p = [-0.0007013635448527738, 0.0014772628407236513], mz = -9.525623669547625e28, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1005.301513671875)
spr = (p = [-1.443704729940474e-5, 2.7098884597450056e-5], mz = -3.2080808621100005e25, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1759.2776489257812)
spr = (p = [-5.068404359382654e-9, 9.05447388244233e-9], mz = -3.700854793463538e18, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 3078.735885620117)
#difference on mz
spr = (p = [-1.8265991947435736e-15, 7.918581472135602e-16], mz = -124148.18807046987, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 5387.787799835205)
spr = (p = [-6.729569458391707e-16, -3.0869047683300955e-15], mz = -113118.4936590613, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 2693.8938999176025)
spr = (p = [-1.3459155687153562e-15, 3.2686829703639614e-16], mz = -50274.88608109882, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 4714.314324855804)
spr = (p = [2.88410656444924e-16, 3.9429121117859366e-16], mz = -256.50452129970915, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 8250.050068497658)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133)
spr = (p = [2.8841587101657233e-16, 3.942962412761004e-16], mz = -256.50452129973655, interior = true, λ = 0.0, hard_case = false, solved = true, Δ = 1.2354744340255133) 

[pkofod]

I'll look into it, thanks