Open MarquessV opened 1 year ago
One wrinkle that has come up so far in #3:
When pulling the library into qcs-sdk-rust
, there are issues with memory limits:
ensure_space: failed to allocate 1048576 bytes at 0x50000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)
This occurs even after maxing out all limits.
ulimit unlimited
ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 65532
-c: core file size (blocks) unlimited
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 5568
-n: file descriptors 256
~Some cursory google search hints towards this having something to do with sbcl
, but I haven't found a fix yet. It's also curious that this doesn't happen inside of the crate. For example, if we run the bell_state
example in this crate, everything works fine, but an equivalent example in qcs-sdk-rust
blows up.~
EDIT: This fixes the issue within the libquil-sys
repo itself, but doesn't prevent the error from happening in packages that depend on libquil-sys
.
I've tried using git submodules to make it possible to build the crate with just sbcl
and quicklisp
installed but with no real luck. There doesn't seem to be a good API for triggering the builds of any of the dependent packages from within Rust. Additionally, quicklisp
also reaches out to the internet to download other dependencies while building quilc
, which doesn't feel good to have as part of a build step.
libquilc-sys
as a dynamically linked library in a Rust crateThree things need to happen for libquilc-sys
to work in a dependent crate properly:
The libquilc dynamic library (ie. libquilc.dylib
) needs to be in some known library search path so the Rust compiler can pick up on it. There are a handful of default search paths, so we can document some common ones as options. (As an aside: Rust and Python don't use all the same search paths for dynamic libraries).
We need to figure out a better way to load the core
file from a known location. Currently, dependents need to copy the core
file to the root of their project so libquilc-sys
can find it.
For the crate to work on Intel macs, we need to add the following linker argument to build.rs
(see quil-lang/sbcl-librarian for why this is a thing):
if cfg!(target_os = "macos") {
println!("cargo:rustc-link-arg=-pagezero_size 0x100000");
}
We set this in the build script for libquilc-sys
as well, but it doesn't propagate down to dependents of the crate.
libquilc-sys
as a dynamically linked library in a Rust crate with Python bindingsAll the same steps taken above apply, but (3) poses a bigger issue in this context. Because -pagezero_size
is a linker arg, it needs to be applied to an executable during a link step. The bindings generated by pyo3
is a Python library, so upon building them, you get this error:
= note: ld: -pagezero_size option can only be used when linking a main executable
clang: error: linker command failed with exit code 1 (use -v to see invocation)
If you omit the linker arg, the library crashes when trying to load the core file, so there is no working around this. The only fix would be to build Python from source with -pagezero_size 0x100000
which obviously is a ton to ask of users (on top of an already exceedingly complex build process).
The sbcl-librarian
README strongly states that this linker flag "MUST" be used for it to work so I don't think this is something we'll easily be able to workaround ourselves. We'll have to followup with the quil-lang
group and see if they have any thoughts or advice.
Rust and Python don't use all the same search paths for dynamic libraries
Specifically, Rust will search $HOME/lib
(at least on mac), while Python restricts itself to /usr/local/lib
For the crate to work on Intel macs, we need to add the following linker argument to build.rs... but it doesn't propagate down to dependents of the crate.
I want to call extra attention to this: every executable binary crate consuming libquil-sys
, no matter how transitively, has to have a build.rs
with this. If we could get everything else working, we'd probably offer a function to make this a one-liner, similar to what PyO3 does.
Because -pagezero_size is a linker arg, it needs to be applied to an executable during a link step...
Quick correction on this: pagezero_size
specifically can only be applied to executables. AFAIK, most linker args can be applied to libraries. The reason for this is a little beyond my understanding, but I will try to explain.
PAGEZERO is a section of an executable that, AFAICT, defines the range of pointers to consider invalid/null. On Intel Macs, this is defined as 4GB, effectively disallowing 32-bit pointers on the 64-bit system. The rationale is that it helps prevent bad things from happening if a program accidentally truncates a 64-bit pointer to 32-bit.
sbcl-librarian
, from what I understand, depends on this address space for the embedding lisp interpreter, which is why trying to use the quilc
library without this flag will fail. As @MarquessV mentions, though, the only way to make this work with Python is to compile the Python executable with this linker flag.
From my understanding, this is very much a problem with Intel MacOS. Maybe M1s wouldn't have an issue. Sounds like Linux shouldn't have an issue either. So it may be possible that we can get this working on Linux, which would at least help avoid the need for sidecar docker images. We would have to figure out how to support MacOS in that case, though.
We were pointed towards this branch of sbcl that has some patches that workaround the need for pagezero_size
. Unfortunately it's based on 2.1.10
, which is just short of the 2.2.0
version that we know works. On my machine at least, this version of SBCL doesn't build.
The build process for this library is relatively complex and likely sensitive to different platforms. How can we distribute this crate in a way that it works on each platform? How can we ease compilation as much as possible for downstream users? For our use case specifically,
qcs-sdk-rust
compiles Python wheels for every major platform, how can we make sure that still works as needed?