Build Fails on MacOS

[mhworth]

Building python-rocksdb fails on recent version of MacOS. The final error you get is when linking with libstdc++:

clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1

Summary of how to work around right now:

brew install rocksdb
export MACOSX_DEPLOYMENT_TARGET=10.9
pip install python-rocksdb

The pandas folks ran into this too; here is a discussion on how to resolve the issue by passing additional args to Clang: https://github.com/pandas-dev/pandas/issues/23424

[zzzhc]

export CFLAGS=-stdlib=libc++
pip install python-rocksdb

this works for me on macOS 10.14.3

[cvarma21]

Thanks. This worked perfectly

[deed02392]

I'm getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

[vincentwyshan]

I'm getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

I get same problem, guess we need install rocksdb in mac and make complier link to proper headers

[mrugacz95]

Installing rocksdb with brew helped: brew install rocksdb

[alexreg]

None of the above solutions work on macOS 12.2, I'm afraid...

[faberchri]

The following procedure worked for me on:

• MacBook Pro M1
• macOS 12.2.1 Monterey
• rocksdb 6.29.3

Install rocksdb with brew:

brew install rocksdb

Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy

I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb

The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

[alexreg]

@faberchri Thank you for sharing, although that didn't work for me either. The only thing that worked in the end was to use the more recent fork of this library, patch that for the latest rocksdb (just removing a few definitions), and then build under brew sh. I'll try to get a PR submitted to that project.

[Littlecowherd]

The following procedure worked for me on:

• MacBook Pro M1
• macOS 12.2.1 Monterey
• rocksdb 6.29.3

Install rocksdb with brew:

brew install rocksdb

Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy

I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb

The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

Thank you anyway, the installation succeeded but runs failed ……

[rollue]

@faberchri Thank you for sharing, although that didn't work for me either. The only thing that worked in the end was to use the more recent fork of this library, patch that for the latest rocksdb (just removing a few definitions), and then build under brew sh. I'll try to get a PR submitted to that project.

Could you share how this could be done in more detail?

I tried to install from your PR branch by adding the below line to poetry,

rocksdb = { git = "https://github.com/alexreg/python-rocksdb.git", branch = "update-api" }

but still getting this error

...
rocksdb/_rocksdb.cpp:796:10: fatal error: 'rocksdb/utilities/backupable_db.h' file not found
#include "rocksdb/utilities/backupable_db.h"
...

P.S: what do you mean by build under brew sh?

[alexreg]

@rollue I mean to start the Homebrew shell using brew sh and run the build from within that. But anyway, the way I recommend using my branch (and when it's merged, NightTsarina's fork), is to configure the rocksdb dependency exactly how you did above, but also, before running pip install (or the poetry equivalent), to export some environment variables:

export INCLUDE_PATH="$(brew --prefix)/include" LIBRARY_PATH="$(brew --prefix)/lib"

This presumes you installed rocksdb with Homebrew. Note that $(brew --prefix) is Bash syntax, but if you're using another shell, replace it with the equivalent (or just hard code it to /opt/homebrew or /usr/local, as appropriate).

[rollue]

@alexreg thanks for the explanation, but pardon my ignorance. When I try to poetry install your branch after installing rocksdb with Homebrew (with brew install rocksdb). I keep getting the following error. (I'm on M1 Mac pro)

.... 
    rocksdb/_rocksdb.cpp:819:10: fatal error: 'rocksdb/utilities/backupable_db.h' file not found
    #include "rocksdb/utilities/backupable_db.h"
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    3 warnings and 1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    [end of output]
...

I've seen in your conversation with NightTsarina, especially in https://github.com/NightTsarina/python-rocksdb/pull/13#issuecomment-1062002636 that this may be deprecated. But I can't find a workaround for this. Any ideas how to go about this?

[alexreg]

@rollue I don't use poetry so I can't say much about it, but if you set the environment variables like above and then pip install my branch, it seems to work fine on macOS. And yes, backupable_db is deprecated, though it still exists with rocksdb 6.

[jrj-d]

Hello @alexreg, I'm experiencing the same issue as @rollue. We're blocked because there's no easy way to install rocksdb v6 on mac (with brew). I'll try to find a workaround

[alexreg]

Hello @alexreg, I'm experiencing the same issue as @rollue.

We're blocked because there's no easy way to install rocksdb v6 on mac (with brew). I'll try to find a workaround

brew install alexreg/dev/rocksdb@6

I tried to get the formula accepted in homebrew-core but sadly no luck. At least this should make it easy though.

[KochankovID]

@Littlecowherd I faced with the same problem and managed to solve it by moving from anaconda to simple venv or poetry. I'm not sure why it works so, but I suggest anyone to try it. It worked for me.

[alexreg]

@KochankovID Better to use NightTsarina's fork these days. Actively maintained.

[faberchri]

Regarding my previous comment in this thread:

The following procedure worked for me on:

MacBook Pro M1 macOS 12.2.1 Monterey rocksdb 6.29.3 Install rocksdb with brew:

brew install rocksdb Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib" pip install python-rocksdb The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

I Followed this same procedure just now again in order to successfully install python-rocksdb on my new computer (Apple M1 Max, 2021).

Crucial missing information from my original post: the procedure works only for rocksdb v6. If you run now brew install rocksdb, you install version 7+ and for version 7+ the procedure does not work.

For me rocksdb v6 is fine, so I used this procedure to install rocksdb v6 with brew. I also had to change the path to the lz4 lib, now /opt/homebrew/Cellar/lz4/1.9.4/lib.

[ccz5]

Here is how I made it to work.

Setup:

1. brew info rocksdb lz4 snappy find out if there is any missing package and install it.
2. brew install rocksdb (this is to install latest rocksdb, v7)
3. brew install alexreg/dev/rocksdb@6
4. Find out where you installed the rocksdb in step 2 and 3, for me it's /usr/local/Cellar/rocksdb@6 and /usr/local/Cellar/rocksdb a. export CPPFLAGS="-I/usr/local/Cellar/rocksdb@6/6.29.5/include -I/usr/local/Cellar/rocksdb/7.7.2/include" (I had to include both 6 and 7 header files because the install seems to need both). b. export LDFLAGS="-L/usr/local/Cellar/rocksdb@6/6.29.5/lib -L/usr/local/Cellar/rocksdb/7.7.2/lib -L/usr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

Install

1. downloaded tar.gz file and unzip it https://pypi.org/project/python-rocksdb/#files
2. go to the directory and vim setup.py, change extra_compile_args "-std=c++" to "-std=c++17" because c++17 functions are used.
3. python3 setup.py install

Good luck!

[alexreg]

Glad it helped. That looks good for this repo. For anyone using (NightTsarina's more up-to-date fork (but still based on rocksdb v6), you don't need to make the change to extra_compile_args, and you can just modify PKG_CONFIG_PATH as indicated by my Homebrew formula.

export PKG_CONFIG_PATH="/opt/homebrew/opt/rocksdb@6/lib/pkgconfig"

[jagalactic]

Regarding my previous comment in this thread:

The following procedure worked for me on: MacBook Pro M1 macOS 12.2.1 Monterey rocksdb 6.29.3 Install rocksdb with brew: brew install rocksdb Use brew to look up the path on your system of the libs required for the install of python-rocksdb: brew info rocksdb lz4 snappy I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them. Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup): export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib" pip install python-rocksdb The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

I Followed this same procedure just now again in order to successfully install python-rocksdb on my new computer (Apple M1 Max, 2021).

Crucial missing information from my original post: the procedure works only for rocksdb v6. If you run now brew install rocksdb, you install version 7+ and for version 7+ the procedure does not work.

For me rocksdb v6 is fine, so I used this procedure to install rocksdb v6 with brew. I also had to change the path to the lz4 lib, now /opt/homebrew/Cellar/lz4/1.9.4/lib.

Thank you! The procedure you linked worked for me! One caveat: I ended up installing a slightly older version of rocksdb (6.27.3_1 IIRC) because 2.29.3 had some sort of CRC error or something. So first try fail, second try success.

[itadventurer]

I had the same problem and for me it seems like the problem is, that python-rocksdb only supports rocksdb 6 and not the latest one (present in Homebrew).

The following approach worked for me (use at your own risk):

1. Install old Rocksdb

brew tap-new $USER/local-rocksdb
brew extract --version=6.29.3 rocksdb $USER/local-rocksdb
brew install rocksdb@6.29.3

2. Export Buildflags

export CPPFLAGS=-I/usr/local/Cellar/rocksdb@6.29.3/6.29.3/include
export LDFLAGS="-L/usr/local/Cellar/rocksdb@6.29.3/6.29.3/lib -Lusr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

3. Install packages

pip install python-rocksdb

[alexreg]

@itadventurer This is all covered fully in the above comments.

[ly2020888]

very useful , I appreciate it

[Tsunaou]

@itadventurer It's very useful to me.

[marghidanu]

This does not work in macOS at all. First, the slice error and then the backupable error.

[utahkay]

@itadventurer Your comment was the only thing that worked for me, thank you!

[Eliacy]

I tried the itadventurer way but have no success. So I put here some other walk-around choices.

Choice 1: Run x86 Conda environment in M1 Mac

1. Have a "Rosetta Terminal": In Finder, Right Click the "Terminal.app", Enable "Open using Rosetta" option in "Get Info". Then all command in Terminal will run under Rosetta mode. (If you run "arch", it will return "i386".)
2. Install the Intel version Anaconda.
3. Create Python environment, install python-rocksdb under Rosetta Termianl. (I tried Python version 3.9)

   conda create -n "myenv" python=3.9
   conda activate myenv
   conda install -c conda-forge python-rocksdb

Choice 2: Use Arm64 Linux in Docker

I create a Docker image with tag "python:3.9-bullseye", and use the bullseye "python3-rocksdb" deb package. Such as:

docker run -it --name mypy --volume $(pwd):/usr/src/app --net=host py-dev:latest bash
apt install python3-rocksdb

Now /usr/lib/python3/dist-packages/rocksdb is the python-rocksdb lib.

[brunneis]

I managed to make it work on a MacBook Air M1 with RocksDB 6.29.3

git clone https://github.com/Homebrew/homebrew-core
cd homebrew-core
git checkout b3597b729a2
brew install Formula/rocksdb.rb
brew pin rocksdb

export CPPFLAGS="-I/opt/homebrew/Cellar/rocksdb/6.29.3/include"
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.10/lib -L/opt/homebrew/Cellar/lz4/1.9.4/lib"

pip install python-rocksdb

For any other package and version you can check this gist.

[e-compagno]

After having rocksdb installed with brew

brew install rocksdb

I tried to install the library with

 export CPPFLAGS="-I/opt/homebrew/Cellar/rocksdb/9.6.1/include"
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/9.6.1/lib -L/opt/homebrew/Cellar/snappy/1.2.1/lib -L/opt/homebrew/Cellar/lz4/1.10.0/lib"
pip install python-rocksdb

in MacOs Sequoia (15.0.1) and Python 3.11.9, but there are compiler errors, e.g.

 rocksdb/_rocksdb.pyx:104:11: 'Slice' is not a constant, variable or function identifier
 rocksdb/_rocksdb.pyx:645:40: cimported module has no attribute 'NewBlockBasedTableFactory'
 rocksdb/_rocksdb.pyx:671:55: cimported module has no attribute 'kPlain'
 rocksdb/_rocksdb.pyx:673:55: cimported module has no attribute 'kPrefix'
 rocksdb/_rocksdb.pyx:679:41: cimported module has no attribute 'NewPlainTableFactory'
...