search

janestreet / pythonlib

A library to help writing wrappers around ocaml code for python
MIT License
99 stars 12 forks source link

Error in trying to build examples/ocaml.bc.so #1

Open jonathan-laurent opened 4 years ago

jonathan-laurent commented 4 years ago

I have been very interested by the blog post Using Python and OCaml in the same Jupyter notebook and I tried to build pythonlib/examples/ocaml.bc.so in order to write my own Python extensions.

I got it to compile after making some minor fixes (like adding a ~loc argument to Env.lookup_value in toploop_bindings.ml or changing the libpython path in examples/init/dune), but then I got stuck with the following linking error:

(base) jonathan@jonathan-aurora:~/pythonlib$ dune build examples/ocaml.bc.so
      ocamlc examples/ocaml.bc.so (exit 2)

(cd _build/default && /home/jonathan/.opam/default/bin/ocamlc.opt -w @1..3@5..28@30..39@43@46..47@49..57@61..62-40 -strict-sequence -strict-formats -short-paths -keep-locs -linkall 
-g -o examples/ocaml.bc.so -output-complete-obj -runtime-variant _pic /home/jonathan/.opam/default/lib/base/base_internalhash_types/base_internalhash_types.cma /home/jonathan/.opam/
default/lib/base/caml/caml.cma /home/jonathan/.opam/default/lib/sexplib0/sexplib0.cma /home/jonathan/.opam/default/lib/base/shadow_stdlib/shadow_stdlib.cma /home/jonathan/.opam/defa
ult/lib/base/base.cma /home/jonathan/.opam/default/lib/ocaml/compiler-libs/ocamlcommon.cma /home/jonathan/.opam/default/lib/ocaml/compiler-libs/ocamlbytecomp.cma /home/jonathan/.opa
m/default/lib/ocaml/compiler-libs/ocamltoplevel.cma /home/jonathan/.opam/default/lib/ocaml/unix.cma /home/jonathan/.opam/default/lib/result/result.cma /home/jonathan/.opam/default/l
ib/stdcompat/stdcompat.cma /home/jonathan/.opam/default/lib/pyml/pyml.cma /home/jonathan/.opam/default/lib/pyml/numpy.cma /home/jonathan/.opam/default/lib/stdio/stdio.cma /home/jona
than/.opam/default/lib/typerep/typerep_lib.cma /home/jonathan/.opam/default/lib/ppx_sexp_conv/runtime-lib/ppx_sexp_conv_lib.cma /home/jonathan/.opam/default/lib/ppx_compare/runtime-
lib/ppx_compare_lib.cma /home/jonathan/.opam/default/lib/ppx_enumerate/runtime-lib/ppx_enumerate_lib.cma /home/jonathan/.opam/default/lib/ppx_hash/runtime-lib/ppx_hash_lib.cma /home
/jonathan/.opam/default/lib/ppx_python/runtime/ppx_python_runtime.cma /home/jonathan/.opam/default/lib/ppx_inline_test/config/inline_test_config.cma /home/jonathan/.opam/default/lib
/ppx_inline_test/runtime-lib/ppx_inline_test_lib.cma /home/jonathan/.opam/default/lib/ppx_here/runtime-lib/ppx_here_lib.cma /home/jonathan/.opam/default/lib/ppx_expect/common/expect
_test_common.cma /home/jonathan/.opam/default/lib/ppx_expect/config/expect_test_config.cma /home/jonathan/.opam/default/lib/fieldslib/fieldslib.cma /home/jonathan/.opam/default/lib/
ppx_expect/collector/expect_test_collector.cma src/python_lib.cma /home/jonathan/.opam/default/lib/ppx_assert/runtime-lib/ppx_assert_lib.cma /home/jonathan/.opam/default/lib/ppx_ben
ch/runtime-lib/ppx_bench_lib.cma /home/jonathan/.opam/default/lib/base/md5/md5_lib.cma /home/jonathan/.opam/default/lib/ocaml/bigarray.cma /home/jonathan/.opam/default/lib/variantsl
ib/variantslib.cma /home/jonathan/.opam/default/lib/bin_prot/shape/bin_shape_lib.cma /home/jonathan/.opam/default/lib/bin_prot/bin_prot.cma /home/jonathan/.opam/default/lib/jane-str
eet-headers/jane_street_headers.cma /home/jonathan/.opam/default/lib/time_now/time_now.cma /home/jonathan/.opam/default/lib/ppx_module_timer/runtime/ppx_module_timer_runtime.cma /ho
me/jonathan/.opam/default/lib/splittable_random/splittable_random.cma /home/jonathan/.opam/default/lib/base_quickcheck/base_quickcheck.cma examples/init/python_ocaml_init.cma -I exa
mples/init examples/.ocaml.eobjs/byte/toploop_bindings.cmo examples/.ocaml.eobjs/byte/ocaml.cmo)
/usr/bin/ld: cannot find -ltime_now_stubs
/usr/bin/ld: cannot find -lbin_prot_stubs
/usr/bin/ld: cannot find -lexpect_test_collector_stubs
/usr/bin/ld: cannot find -lnumpy_stubs
/usr/bin/ld: cannot find -lpyml_stubs
/usr/bin/ld: cannot find -lstdcompat__stubs
/usr/bin/ld: cannot find -lbase_stubs
/usr/bin/ld: cannot find -lbase_internalhash_types_stubs
collect2: error: ld returned 1 exit status
File "_none_", line 1:
Error: Error while building custom runtime system

Does anyone have an idea what's wrong here? :-)

My configuration

LaurentMazare commented 4 years ago

I've never encountered this issue before but looking at the error message there seems to be an issue with your pyml install. Just to check, I was able to compile ocaml.bc.so with the pyml.20200222 opam package on ocaml 4.08 (I also had to edit examples/init/dune as we do not detect the python installation properly in this example).

jonathan-laurent commented 4 years ago

I have the exact same problem using Ocaml 4.08 and pyml.20200222 (installed through opam), both on MacOS ans Ubuntu 18.04.

By curosity, what makes you think the problem is with pyml while the linking errors involve a list of libraries such as time_now or bin_prot?

LaurentMazare commented 4 years ago

Ah, indeed I didn't notice these so it's probably not pyml related, I'm a bit out of idea then. Fwiw I use dune 2.0.0 and opam 2.0.3 (on an ubuntu 20.04).

jonathan-laurent commented 4 years ago

@LaurentMazare Thanks for having looked at this! I am gonna leave the issue open and I'll tell you if I find a fix.

gbdrt commented 4 years ago

I got a similar issue on MacOS 10.14.6 with OCaml 4.10.0 and Dune 2.5.1. Is there a work around?

ld: library not found for -lbin_prot_stubs
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "_none_", line 1:
Error: Error while building custom runtime system
jonathan-laurent commented 3 years ago

I am still encountering the same issue on a new config: Ubuntu 20.04, OCaml 4.11.1 and Dune 2.7.1.

/usr/bin/ld: cannot find -lbin_prot_stubs
/usr/bin/ld: cannot find -lexpect_test_collector_stubs
/usr/bin/ld: cannot find -ltime_now_stubs
/usr/bin/ld: cannot find -lnumpy_stubs
/usr/bin/ld: cannot find -lpyml_stubs
/usr/bin/ld: cannot find -lstdcompat__stubs
/usr/bin/ld: cannot find -lbase_stubs
/usr/bin/ld: cannot find -lbase_internalhash_types_stubs
collect2: error: ld returned 1 exit status
File "_none_", line 1:
Error: Error while building custom runtime system

I am also interested in a workaround, or any hint that may help me debug this link error.

LaurentMazare commented 3 years ago

I'm still not sure what is going on here, however I was getting this issue and was able to fix it by setting LIBRARY_FLAGS as follows (note that your opam path may vary).

LIBRARY_PATH=$HOME/.opam/4.11.1/lib/base/base_internalhash_types:$HOME/.opam/4.11.1/lib/ppx_expect/collector:$HOME/.opam/4.11.1/lib/time_now:$HOME/.opam/4.11.1/lib/bin_prot:$HOME/.opam/4.11.1/lib/base:$HOME/.opam/4.11.1/lib/pyml:$HOME/.opam/4.11.1/lib/stdcompat dune build examples/ocaml.bc.so

Hope this will work for you too.

jonathan-laurent commented 3 years ago

This fix works for me. Thanks!

Here is a cleaned-up build script in case anyone else encounters this issue:

LIBS="
  base base/base_internalhash_types
  ppx_expect/collector time_now bin_prot pyml stdcompat"

OPAM_LIBS="$HOME/.opam/default/lib"

for l in $LIBS
do
  export LIBRARY_PATH="$OPAM_LIBS/$l:$LIBRARY_PATH"
done

dune build examples/ocaml.bc.so
ejgallego commented 3 years ago

Note that the regression in dune causing this has been identified, c.f. https://github.com/ocaml/dune/issues/3910

Still unclear what this fix but hopefully it will land in next dune version.

nilsbecker commented 3 years ago

I got a similar issue on MacOS 10.14.6 with OCaml 4.10.0 and Dune 2.5.1. Is there a work around?

ld: library not found for -lbin_prot_stubs
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "_none_", line 1:
Error: Error while building custom runtime system

I have the same problem on macos, dune 2.7.1.

the only missing library listed is -lbin_prot_stubs.

(I also had to edit toploop_bindings.ml, not only dune as @jonathan-laurent mentioned. apparently the examples use some versions of libraries that are older than those listed in the dependencies of the package)

nilsbecker commented 3 years ago

The fix worked for me as well. I did have to add all the extra directories as listed, rather than just bin_prot.

in previous attempts i had added some of the libraries whose stubs are not found as ppx preprocessors, e.g. ppx_expect . this appeared to remove some of those linking errors but never all of them.

jonathan-laurent commented 3 years ago

To people reading this issue, I just wanted to point out that despite the problems encountered when building the examples in this repo (which still persist on master), the latest release of pythonlib that is available on opam is fully functional and makes it very easy to write python bindings for ocaml libraries. For example, you can look at an example repo here.