search

kaneplusplus / bigmemory

126 stars 24 forks source link

Is it possible to access a memory-backed bigmemory deterministically? #40

Closed adamryczkowski closed 8 years ago

adamryczkowski commented 8 years ago

I want to use the bigmemory::matrix as a means of communication between two unrelated R processes. The bigmemory::matrix can be shared using its descriptor file, but one needs to send the descriptor file to the other process, so there is a chicken-and-the-egg problem.

One way of solving the problem is to save the descriptor file in some place on the mutually agreed location within the file structure.

Is there any way to get the access to the bigmemory::matrix without using the filesystem? Something like "named bigmemory::matrix" that work similar to the named mutexes?

phaverty commented 8 years ago

Hi Adam,

The two R processes both need to get the metadata from the description file one way or another. They don't both need to read the description file, however. You can send the description information from one process to another however you like. I will often share one bigmemory object among multiple threads by loading it in one session/script and then passing it to a mclapply call. In the Bioconductor world, we also have the BigMemoryExtras package. This package defines an object type that holds the metadata info and also "remembers" where the data matrix file is. You can save these BigMatrix objects like any other R object (e.g. in an RData file) and then use them where ever without having to think about file paths.

Does that help?

Pete

Peter M. Haverty, Ph.D. Genentech, Inc. phaverty@gene.com

On Sun, Sep 11, 2016 at 6:09 AM, Adam Ryczkowski notifications@github.com wrote:

I want to use the bigmemory::matrix as a means of communication between two unrelated R processes. The bigmemory::matrix can be shared using its descriptor file, but one first to send the descriptor file to the other process first, so there is a chicken-and-the-egg problem.

One way of solving the problem is to save the descriptor file in some place of the mutually agreed location within the file structure.

Is there any way to get the access to the bigmemory::matrix without using the filesystem? Something like "named bigmemory::matrix" that work similar to the named mutexes?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kaneplusplus/bigmemory/issues/40, or mute the thread https://github.com/notifications/unsubscribe-auth/AH02KwCjFlgFkVTcSIw-6W3XPiSHHd_wks5qo_1sgaJpZM4J6ANH .

adamryczkowski commented 8 years ago

Does that help?

Not really, but thank you for your time.

I wanted my package a) not needing the server part (the one that initializes the big.matrix) being launched prior the client and preferably b) not to touch the filesystem.

I understand, that I need to let the "b)" part go.

In theory it is possible to use the message queue with a message containing the description of the big.matrix (or just the memory pointer, as it is the only non-deterministic information contained therein) just for this purpose of breaking this "chicken and egg" problem.

Now I understand that the message queues are not in the scope of the problems that "big.matrix" solves, so I would need to address the problem myself.

I am closing the issue now, and sorry for the unnecessary traffic.