Robust Singular Jacobian Handling

[avik-pal]

Current approach is:

1. Proceed as usual
2. On first linear solve failure:
   1. Construct a QRPivoted Cache and cache it
   2. Solve using Pivoted QR but warn the user that there is a potential singular Jacobian detected
   3. If successful, proceed as usual
   4. Otherwise:
      1. For "current" jacobian: stop nonlinear solve
      2. For approximate methods: follow a policy similar to line search routines
      3. For non-current exact jacobian: force a reset
3. On later linear solve failure:
   1. Goto 2.2

TODOs

• [x] Switching to Pivoted QR
• [x] Add tests
• [x] Use verbose to turn off printing
• [x] Integrate it correctly into the algorithms -- currently assumes that switching is always successful.
• [x] How do we deal with non-standard Arrays? Basically how do we determine that PivotedQR is defined for the matrix A
• [x] Add missing docstrings to the documentation

@ChrisRackauckas any idea how to handle this last part?

[avik-pal]

With this PR all of the following work:

using NonlinearSolve, LinearSolve

function nlls!(du, u, p)
    du[1] = 2u[1] - 2
    du[2] = (u[1] - 4u[2])^2 + 0.1
    du[3] = 0
end

u0 = [0.0, 0.0]
prob = NonlinearLeastSquaresProblem(
    NonlinearFunction(nlls!, resid_prototype = zeros(3)), u0)

solve(prob)

solve(prob, GaussNewton(; linsolve = QRFactorization()))

solve(prob, GaussNewton(; linsolve = LUFactorization()))

I always display a warning rn, but I will configure the verbose to not show the warning if needed.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 73.73737% with 26 lines in your changes are missing coverage. Please review.

Project coverage is 86.34%. Comparing base (b389d0e) to head (914f556).

Files	Patch %	Lines
src/core/approximate_jacobian.jl	28.57%	10 Missing :warning:
src/core/generalized_first_order.jl	64.28%	5 Missing :warning:
src/descent/steepest.jl	16.66%	5 Missing :warning:
src/internal/linear_solve.jl	86.66%	4 Missing :warning:
src/descent/damped_newton.jl	85.71%	1 Missing :warning:
src/descent/newton.jl	92.85%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #414 +/- ## ========================================== - Coverage 86.77% 86.34% -0.44% ========================================== Files 46 47 +1 Lines 2851 2914 +63 ========================================== + Hits 2474 2516 +42 - Misses 377 398 +21 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[avik-pal]

Needs https://github.com/SciML/LinearSolve.jl/pull/494 before the tests pass

[avik-pal]

@ChrisRackauckas do you have a suggestion on how to deal with this

How do we deal with non-standard Arrays? Basically how do we determine that PivotedQR is defined for the matrix A

Apart for this, we are good to go here.

Okay maybe we can use ArrayInterface.isstructured and AnyGPUArray to safeguard against this