Suggestions for storing distance matrices

[talegari]

Hi,

I am trying to implement a out-of-memory storage for distances matrices where I need to store the lower diagonal entries only. lvec seems like a natural fit over bigmemory family of packages where its require to store the whole matrix (practically repeating everything twice and the diagonal).

I have a few questions which might help my design decisions:

• Can I use Rcpp to talk to lvec vectors?
• Is it possible to modify c++ functions from proxy or parallelDist packages to write to lvec vectors directly without bringing them into R?
• Would it be efficient to implement matrix like apply functions on lvec using C++ functions without bringing each row or column into R?

Regards, Srikanth KS

[djvanderlaan]

Sorry it took a while (combination of being out of the country and being busy)

Yes, you can write Rcpp code that talks to lvec vectors. This is what I also do with the ldat package. Unfortunately, this isn't documented very well. I am planning to start documenting this, but at the moment the best you can do is look at some of the code in the ldat package. You could also look at the https://github.com/djvanderlaan/lvec_example repo, where I create an example package linking against lvec. However, this package still uses the old (pre-cran-release of lvec) interface.