Q extraction from QR-decomposition fails

[whilo]

Hi Dragan,

I try to extract Q for my core.matrix wrapper. It works on the REPL and returns the proper result, but I get the following error on the compliance test suite:

"QR" #uncomplicate.neanderthal.internal.common.OrthogonalFactorization{:eng #object[uncomplicate.neanderthal.internal.host.mkl.DoubleGEEngine 0x607e6542 "uncomplicate.neanderthal.internal.host.mkl.DoubleGEEngine@607e6542"], :or #RealGEMatrix[double, mxn:3x4, layout:column, offset:0]
   ▥       ↓       ↓       ↓       ↓       ┓    
   →      -3.16   -0.63   -5.69   -6.96         
   →       0.00   -1.90   -1.26   -1.90         
   →       0.72   -1.00   10.00    0.00         
   ┗                                       ┛    
, :jpiv nil, :tau #RealBlockVector[double, n:3, offset: 0, stride:1]
[   1.32    1.00    0.00 ]
, :master true, :fresh #object[clojure.lang.Atom 0x3654474 {:status :ready, :val true}], :m 3, :n 3, :or-type :qr, :api-orm #object[uncomplicate.neanderthal.internal.api$eval12129$fn__12213$G__12070__12226 0x298e3c19 "uncomplicate.neanderthal.internal.api$eval12129$fn__12213$G__12070__12226@298e3c19"], :api-org #object[uncomplicate.neanderthal.internal.api$eval12129$fn__12308$G__12102__12317 0x2945ae72 "uncomplicate.neanderthal.internal.api$eval12129$fn__12308$G__12102__12317@2945ae72"]}

Intel MKL ERROR: Parameter 2 was incorrect on entry to DORGQR.

lein test :only denisovan.core-test/compliance-test

ERROR in (compliance-test) (core.clj:4617)
Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ExceptionInfo: There has been an illegal argument in the native function call.
 at clojure.core$ex_info.invokeStatic (core.clj:4617)
    clojure.core$ex_info.invoke (core.clj:4617)
    uncomplicate.neanderthal.internal.host.mkl.DoubleGEEngine.gqr (mkl.clj:725)
    uncomplicate.neanderthal.internal.api$eval12129$fn__12308$G__12103__12312.invoke (api.clj:59)
    uncomplicate.neanderthal.internal.api$eval12129$fn__12308$G__12102__12317.invoke (api.clj:59)
    uncomplicate.neanderthal.internal.common.OrthogonalFactorization.org (common.clj:262)
    uncomplicate.neanderthal.linalg$org.invokeStatic (linalg.clj:419)
    uncomplicate.neanderthal.linalg$org.invoke (linalg.clj:409)
    denisovan.core$eval40810$fn__40811.invoke (core.clj:570)

Any ideas?

[blueberry]

If I understood correctly, this is an error that appears only in core.matrix compliance tests, and does not appear when you try it directly with neanderthal?

You posted a result, but I would also need to know what function was called, and the context in which it was called to see what's happening.

Can you please construct an example test that causes this error when you use Neanderhtal directly? It would be best if you can put it in a github project with a test that I can run and debug.

[whilo]

Sure. I will try then to pin it down anyway. I just thought you had some idea what I could have done wrong. This will not give more insight, I think, but for reference, the code is here: https://github.com/whilo/denisovan/blob/master/src/denisovan/core.clj#L559

[whilo]

So I can reduce the problem to:

  (def bar (nat/dge 2 3 [1 0 1 2 0 0]))
  (def qr (lin/qrf bar))
  (lin/org qr)

;; =>

1. Unhandled clojure.lang.ExceptionInfo
   There has been an illegal argument in the native function call.
   {:arg-index 3}

                  core.clj: 4739  clojure.core/ex-info
                  core.clj: 4739  clojure.core/ex-info
                   mkl.clj:  776  uncomplicate.neanderthal.internal.host.mkl.DoubleGEEngine/gqr
                   api.clj:   59  uncomplicate.neanderthal.internal.api/eval12944/fn/G
                   api.clj:   59  uncomplicate.neanderthal.internal.api/eval12944/fn/G
                common.clj:  286  uncomplicate.neanderthal.internal.common.OrthogonalFactorization/org
                linalg.clj:  419  uncomplicate.neanderthal.linalg/org
                linalg.clj:  409  uncomplicate.neanderthal.linalg/org
                      REPL:  602  denisovan.core/eval66400
                      REPL:  602  denisovan.core/eval66400
             Compiler.java: 7062  clojure.lang.Compiler/eval
             Compiler.java: 7025  clojure.lang.Compiler/eval
                  core.clj: 3206  clojure.core/eval
                  core.clj: 3202  clojure.core/eval
                  main.clj:  243  clojure.main/repl/read-eval-print/fn
                  main.clj:  243  clojure.main/repl/read-eval-print
                  main.clj:  261  clojure.main/repl/fn
                  main.clj:  261  clojure.main/repl
                  main.clj:  177  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj:  652  clojure.core/apply
                regrow.clj:   18  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj: 1965  clojure.core/with-bindings*
                  core.clj: 1965  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   85  clojure.tools.nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   55  clojure.tools.nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  222  clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
    interruptible_eval.clj:  190  clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
                  AFn.java:   22  clojure.lang.AFn/run
   ThreadPoolExecutor.java: 1149  java.util.concurrent.ThreadPoolExecutor/runWorker
   ThreadPoolExecutor.java:  624  java.util.concurrent.ThreadPoolExecutor$Worker/run
               Thread.java:  748  java.lang.Thread/run

For dimensions 3 2 it works.

[blueberry]

You get this error because the number of columns to be computed must be <= the number of rows. Therefore, the dimensions you supplied are invalid.

There are two ways to handle this:

• Accept the widest span of dimensions (including those that are potentially invalid), and throw errors like this if they turn out to not be good.
• Create a more complex checking, and in cases like this one compute only the part of the structure that is possible to compute.

This procedure works in the first way, since it is difficult to handle all possible cases in advance. Maybe it would be better to make this checking more thorough, but it would require much more involvement from the community to create many test cases...

[blueberry]

To make this clearer: For this particular case, the solution would be to either refuse to factorize this matrix at all (although it can be factorized without problem), or to refuse to compute Q (since it can't be fully computed inside the available memory space).

[whilo]

You will hate me for this: https://github.com/whilo/denisovan/blob/master/src/denisovan/core.clj#L560

All core.matrix tests pass now (I cannot just throw exceptions for that). I probably should notify the user somehow that a fallback is applied. I could not make it work by using transposes and I am not familiar enough with QR decomposition to see a proper linear algebra fix atm.

[blueberry]

Why would I hate you? :)

[whilo]

Because I fallback to a low-perf core.matrix implementation in vectorz that does everything that you try to avoid in neanderthal, in particular implicit copying. But it was maybe a bit drastic :joy: . If you know a direct better linear algebra hack that would work with neanderthal I would do that.

[blueberry]

If you're sure that the hack is correct, then why would I worry about that? Hell, even if hack is not correct, why would I worry? I may have a reason to start worrying only if I had to use it :)

[whilo]

Hehe, ok, sure. I don't want to write a wrapper that gives people the wrong impressions about neanderthal, so I try not to screw it if possible.