Building with pip fails with missing headers

[GodTamIt]

Trying to build this repository with pip install -r requirements.txt, where the repository is listed as a Git hash is broken:

rocksdb/_rocksdb.cpp:1232:10: fatal error: 'rocksdb/slice.h' file not found
#include "rocksdb/slice.h"
         ^~~~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

Running python -m build also results in this error.

[wbarnha]

You need to install and link the RocksDB header files, it's a major pain :frowning_face:, which is why I created a workflow to build wheel files to get around this issue.

Alternatively, you may find this SO answer of interest: https://stackoverflow.com/a/70501371/9852671

[GodTamIt]

@wbarnha I tried building RocksDB as a shared library, but it didn't seem to like that either.

I'm doing the following in a Dockerfile:

RUN apt-get install -y --no-install-recommends libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev
RUN git clone https://github.com/facebook/rocksdb --depth 1 --branch "v6.29.5" rocksdb
RUN cd rocksdb && make install-shared -j 32
RUN ls /usr/local/lib && test -f /usr/local/lib/librocksdb.so.6.29

# Remove temporary dir
RUN rm -rf rocksdb

RUN pip install git+https://github.com/faust-streaming/python-rocksdb.git@9b81bed3ffa55a1662811e9ad9fea652f42047f6

RUN python -c 'import rocksdb'

I'm getting:

ImportError: librocksdb.so.6.29: cannot open shared object file: No such file or directory

[wbarnha]

Out of curiosity, what's the motivation for building this library from source? Are you trying to support another OS or architecture? I'm considering marking this project as deprecated since rocksdict has taken over.

[GodTamIt]

I'm trying to build it for ARM64. I want to move to rocksdict, but there are a few legacy instances that rocksdict isn't binary compatible with, I don't think.

[wbarnha]

Got it. The easiest solution to be honest would to be extending the CI/CD pipeline for more architectures. The complexity of this library's build process is horrific and I apologize.

[wbarnha]

Got your ARM wheels built in https://github.com/faust-streaming/python-rocksdb/pull/18. Holy crap, I can't believe QEMU slowed it down to take 5h8m.

To be honest I'm contemplating this being my final release and then archiving the project. This has been an unwieldy library for some time and maintaining it is incredibly difficult. I've tried reviewing the internals of RocksDB to see if I can improve build times or reduce wheel sizes, but there's not much I can currently achieve with my skillset at the moment. Plus not many other people have actively stepped up to help me maintain this project, and I think I've been able to move more attention to rocksdict, which in my opinion, is substantially easier to maintain. Unless other people volunteer to support this project, I will be likely archiving it due to lack of additional support and outdated functionality.