Integrate chemometric model_W

[BrianClement]

[Issue originally created 2016-Oct-04 21:48:01 UTC by rgommers]

Mark wrote this in C++. Grew a bit ad-hoc, could use some rework/rewrite. Uses OpenCV including its BLAS operations. Options discussed with Marc:

• Use a chemometric library instead of the handwritten C++ code. R has one apparently (look for it).
• Use a proper BLAS library.
• Look at command-line interface. May fit for Luigi, but are some bindings needed to use it from other languages? Or rewrite the whole thing in Python or R?

[BrianClement]

[Comment originally created 2016-Dec-01 02:51:43 UTC by mrkriddell]

FYI @hammockman we're working on this now. Going to add C++ code to this repo

[BrianClement]

[Comment originally created 2016-Dec-01 19:50:19 UTC by rgommers]

A bit more detailed, the plan is:

• figure out whether to use Python or R for reimplementing a first model. There are some Python modules (example: http://openresearchsoftware.metajnl.com/articles/10.5334/jors.bs/), but need to figure out if they're as extensive / enough for this modeling. If so, Python is preferred - otherwise use R. (action Ralf to look into this)
• depending on that, determine where to put Mark's C++ code in the repo (action Ralf)
• commit all C++ code (action Mark, with help of Ralf)
• create a test case for the algorithm to be translated. Outputs: the command to run with the command-line interface of the C++ code, and the expected output (verified for correctness). action Mark
• set up versioning so that with the output of the chemometric model we get a commit hash, so we know exactly what result was generated with what version of the model (action Ralf)
• translate the first model to R/Python (action Mark and Ralf together, TBD exactly who does what)

[BrianClement]

[Comment originally created 2016-Dec-03 07:49:34 UTC by rgommers]

Okay, there's much more chemometrics functionality in R, so we'll look at using that via RPy2. I have no desire to deal with binding C++ code or HDF5 access in R, so we'll do all that in Python and only if a model is missing we do that in R.

Let's put the C++ code in a submodule, proposed name discbot.nir.

[BrianClement]

[Comment originally created 2016-Dec-03 07:52:47 UTC by rgommers]

And no point in worrying about GPL code. In the unlikely case that we will actually sell/license a DiscBot, we'll probably provide source code with it anyway and otherwise we'll remove the code then.

[BrianClement]

[Comment originally created 2016-Dec-05 00:09:51 UTC by rgommers]

second submodule: discbot.coordinates?

[BrianClement]

[Comment originally created 2016-Dec-05 00:18:32 UTC by rgommers]

Start with the NIR one, because the coordinate transform project also uses @BrianClement 's HDF5 library. So that really spans both the SVN repo and this one ....

[BrianClement]

[Comment originally created 2016-Dec-05 00:19:33 UTC by rgommers]

OK final names: chemometrics and coregistration.

[BrianClement]

[Comment originally created 2016-Dec-05 00:45:14 UTC by rgommers]

To generate test data, run chemometrics program in console mode:

• choose: 5 (PLS model development)
• cal. set identifier: NIR/CHEMOMODEL/Calibration/cal11to14SubsetX
• Y integer ID: cal11to14SubsetY
• Y variable dataset: cal11to14KlasonligninSubset
• number of PLS factors in model: 4

<starts running>, 81.77% of variance explained

press y

do you want to save model coefficients? --> y name: Klasontest

Inputs and outputs at Q:\Bitbucket\Mark Riddell\PLSmodel4factors