quil-lang / qvm

The high-performance and featureful Quil simulator.
Other
415 stars 57 forks source link

Implement and expose allocation API #150

Open notmgsk opened 5 years ago

notmgsk commented 5 years ago

There are at least three modes of allocation:

  1. Native, with memory being handled by the lisp image. This is generally best-suited for the batch-mode execution of the QVM. The size of memory is restricted by the image -- e.g. SBCL configures its heap size with --dynamic-space-size.
  2. Foreign, with memory being handled by the system kernel (we request a pointer to some malloc'd memory). Useful where workloads require more memory than is available in the lisp image.
  3. Shared, with a region of memory being described by some name that can be accessed by multiple distinct processes. The memory requirements of a full wavefunction grow exponentially in the number of qubits. Imagine that you were doing a 26-qubit simulation from pyQuil, and you wanted to inspect the wavefunction. That is roughly 1GB of data that needs to be sent from the QVM to pyQuil (over a HTTP connection). This can be alleviated with shared memory by providing pyQuil (python) the name of the shared memory region: the user can then directly inspect the memory on the host machine.

All of these modes exist in QVM currently (with varying levels of completeness). We should take some time to clarify the interface, and complete unfinished work (namely shared memory). Shared memory will require pyQuil changes.

jmbr commented 5 years ago

Something we might want to keep in mind is how changes in the memory management of our code interact with ECL.

Also, @stylewarning pointed out to me recently that there is opportunistic pinning of memory happening in magicl.cffi-types:with-array-pointers. Those pieces of memory are temporarily prevented from being garbage collected but they are Lisp-allocated.

stylewarning commented 5 years ago

Is the API implemented sufficiently well with the allocator classes? Or do you mean something higher level?