brotli

[jayvdb]

Adding brotlipy fails looking for -llibbrotli, which is its own internal copy of the library.

  Running command git clone -q https://github.com/jayvdb/brotlipy /tmp/pip-req-build-0y42i1hu
  Running command git submodule update --init --recursive -q
   Compiling pyembed v0.4.0 (/path/to/project/brotli/pyembed)
   Compiling brotli v0.1.0 (/path/to/project/brotli)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.15kbg8p4dc6ijdu1.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1jc1u2iuh94lh70o.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.22q3drr76jbpis3w.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.2syjn2icu1vwbxik.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.3bu23526ltwdhtc7.rcgu.o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.4y1e3ncj44u9ud8.rcgu.o" "-o" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/brotli-f90a18d4864a2a16.1fz3gb6hjy58le1w.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "/path/to/project/brotli/build/target/debug/deps" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/build/jemalloc-sys-6a2dcd7182e77aca/out/build/lib" "-L" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/pyoxidizer" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpyembed-f6d8ce5fcfbe2b59.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libuuid-40f161259e10575f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand-4587c77d50bf082f.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_xorshift-e19a683b8360bb74.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_pcg-161749b1ee75ec4d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_hc-90ed91a123f3b276.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_chacha-bb755f08c6cd079a.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_isaac-bcff91db4f001673.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-2c32e48f67c62942.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_os-47519e8a6d5b29ed.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_jitter-4cb1ba9a013dd964.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/librand_core-3c552559ee9910d6.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libjemalloc_sys-ceb86c614fe13a93.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-03ee1ee8a2342a8e.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libcpython-d6d9ce6fb32c53cc.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libnum_traits-76cec740f4fe2b5d.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libbyteorder-9bfb225a74a4f789.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/libpython3_sys-539fc76e435ddd51.rlib" "/path/to/project/brotli/build/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-dae06a64a5f75d6d.rlib" "-Wl,--start-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-d655238cc7536e49.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-a1048e3b4b8487f7.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-e303706b808d25cf.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-7c4c4a242f593b83.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-3a3b1d4df4be24e8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-9782daa5de8380d8.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-58ece968a1125984.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-97498a9a22285b62.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ecc3ad89a49704cf.rlib" "-Wl,--end-group" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-aeea5fc255482634.rlib" "-Wl,-Bdynamic" "-ldl" "-lm" "-llibbrotli" "-lstdc++" "-lpthread" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/lib64/gcc/x86_64-suse-linux/9/../../../../x86_64-suse-linux/bin/ld: cannot find -llibbrotli
          collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile `brotli`.

To learn more, run the command again with --verbose.

Using git+https://github.com/jayvdb/brotlipy@libbrotli-v1.0-support with envvar USE_SHARED_BROTLI=1 works better, but then fails with

>>> import brotli
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "brotli", line 3, in <module>
  File "brotli.brotli", line 5, in <module>
ModuleNotFoundError: No module named 'brotli._brotli'

The brotlipy package shadows the real package Brotli from Google, sharing the same installed package name. Luckily they share a similar Python API. If I can install brotlipy, and then install Brotli afterwards, the result should be the real Brotli is embedded.

It looks like they are processed in the correct order

adding embedded module source: brotli
adding embedded module bytecode: brotli
adding embedded module source: brotli.brotli
adding embedded module bytecode: brotli.brotli
adding embedded module source: brotli.build
adding embedded module bytecode: brotli.build
adding embedded module source: brotli
adding embedded module bytecode: brotli

I know brotli.build only exists in brotlipy, so that was included first, and then hopefully overwritten by google Brotli.

It looks like the build is going well

resolving inputs for 3 built extension modules...
library brotlienc required by _brotli
library brotlidec required by _brotli
library stdc++ required by _brotli
library ffi required by _cffi_backend

I have a static libs build of brotli locally, but I do not see the two brotli static libs being included.

Despite that, it builds ok, but then the error is

  File "/usr/lib/python3.7/site-packages/requests_mock/adapter.py", line 15, in <module>
    from requests.adapters import BaseAdapter
  File "requests", line 43, in <module>
  File "urllib3", line 7, in <module>
  File "urllib3.connectionpool", line 29, in <module>
  File "urllib3.connection", line 40, in <module>
  File "urllib3.util", line 5, in <module>
  File "urllib3.util.request", line 9, in <module>
  File "brotli", line 12, in <module>
AttributeError: module '_brotli' has no attribute '__version__'

Which looks like the real Google brotli was used, as it does the following during init,

import _brotli

# The library version.
__version__ = _brotli.__version__

But that suggests something is also wrong with the build of the real Google brotli library.

[jayvdb]

Hmm, the build log contains

package brotli not initially detected as such; is package detection buggy?

[jayvdb]

For brotlipy, using USE_SHARED_BROTLI=1, it appears that ModuleNotFoundError: No module named 'brotli._brotli' is happening because pyoxidizer is creating _brotli as a root module instead of being a module within package brotli.

[jayvdb]

I've split off the shared lib subdirectory problem to https://github.com/indygreg/PyOxidizer/issues/169 , as even with that solved there is still the problem of getting both brotli building out of the box, and how to reliably choose which one to include.

[jayvdb]

Google's Brotli overrides build_ext and invokes compiler.link_shared_object, thereby bypassing the distutils hack.