Unsupported leveldb 1.21 on MacOS

[nimmortal]

Hello guys.

I had an issue with plyvel and leveldb on macos. It was about installing plyvel. The issue was similar to the #80. However, I discovered that leveldb was updated from 1.20 (build from 2017) to the newest 1.21 (https://github.com/google/leveldb/releases/tag/1.21 - 10 days ago). So I tried to downgrade leveldb to 1.20 and it works now, I assume that the newest version is not compatible with plyvel on macos.

Installation log:

  Downloading https://files.pythonhosted.org/packages/b3/1d/05e3555abddb49b9a2f3aa1268bec4933bf377ffcad3c1b2630fcfa597eb/plyvel-1.0.5.tar.gz (143kB)
    100% |████████████████████████████████| 153kB 4.2MB/s
Installing collected packages: plyvel
  Running setup.py install for plyvel ... error
    Complete output from command /Users/IdeaProjects/neo-blockchain/neo-local/smart-contracts/venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-_upjtvg9/plyvel/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-w5wkvstj/install-record.txt --single-version-externally-managed --compile --install-headers /Users/IdeaProjects/neo-blockchain/neo-local/smart-contracts/venv/include/site/python3.7/plyvel:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.7
    creating build/lib.macosx-10.14-x86_64-3.7/plyvel
    copying plyvel/_version.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
    copying plyvel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
    running build_ext
    building 'plyvel._plyvel' extension
    creating build/temp.macosx-10.14-x86_64-3.7
    creating build/temp.macosx-10.14-x86_64-3.7/plyvel
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/Users/IdeaProjects/neo-blockchain/neo-local/smart-contracts/venv/include -I/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c plyvel/_plyvel.cpp -o build/temp.macosx-10.14-x86_64-3.7/plyvel/_plyvel.o -Wall -g -mmacosx-version-min=10.7 -stdlib=libc++
    In file included from plyvel/_plyvel.cpp:589:
    In file included from /usr/local/include/leveldb/db.h:11:
    In file included from /usr/local/include/leveldb/iterator.h:19:
    /usr/local/include/leveldb/slice.h:41:25: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      Slice(const Slice&) = default;
                            ^
    /usr/local/include/leveldb/slice.h:42:36: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      Slice& operator=(const Slice&) = default;
                                       ^
    In file included from plyvel/_plyvel.cpp:589:
    In file included from /usr/local/include/leveldb/db.h:11:
    In file included from /usr/local/include/leveldb/iterator.h:20:
    /usr/local/include/leveldb/status.h:26:11: error: expected ';' at end of declaration list
      Status() noexcept : state_(nullptr) { }
              ^
              ;
    /usr/local/include/leveldb/status.h:32:16: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
      Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
                   ^
    /usr/local/include/leveldb/status.h:32:23: error: expected ';' at end of declaration list
      Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
                          ^
                          ;
    /usr/local/include/leveldb/status.h:102:16: error: 'Status' is missing exception specification 'throw()'
    inline Status::Status(const Status& rhs) {
                   ^
                                             throw()
    /usr/local/include/leveldb/status.h:23:22: note: previous declaration is here
    class LEVELDB_EXPORT Status {
                         ^
    /usr/local/include/leveldb/status.h:114:40: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
    inline Status& Status::operator=(Status&& rhs) noexcept {
                                           ^
    /usr/local/include/leveldb/status.h:114:48: error: expected function body after function declarator
    inline Status& Status::operator=(Status&& rhs) noexcept {
                                                   ^
    In file included from plyvel/_plyvel.cpp:589:
    In file included from /usr/local/include/leveldb/db.h:11:
    /usr/local/include/leveldb/iterator.h:28:31: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      Iterator(const Iterator&) = delete;
                                  ^
    /usr/local/include/leveldb/iterator.h:29:42: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      Iterator& operator=(const Iterator&) = delete;
                                             ^
    /usr/local/include/leveldb/iterator.h:80:27: warning: alias declarations are a C++11 extension [-Wc++11-extensions]
      using CleanupFunction = void (*)(void* arg1, void* arg2);
                              ^
    In file included from plyvel/_plyvel.cpp:589:
    In file included from /usr/local/include/leveldb/db.h:12:
    /usr/local/include/leveldb/options.h:45:26: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool create_if_missing = false;
                             ^
    /usr/local/include/leveldb/options.h:48:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool error_if_exists = false;
                           ^
    /usr/local/include/leveldb/options.h:55:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool paranoid_checks = false;
                           ^
    /usr/local/include/leveldb/options.h:65:20: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      Logger* info_log = nullptr;
                       ^
    /usr/local/include/leveldb/options.h:78:28: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      size_t write_buffer_size = 4 * 1024 * 1024;
                               ^
    /usr/local/include/leveldb/options.h:83:22: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      int max_open_files = 1000;
                         ^
    /usr/local/include/leveldb/options.h:90:22: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      Cache* block_cache = nullptr;
                         ^
    /usr/local/include/leveldb/options.h:96:21: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      size_t block_size = 4 * 1024;
                        ^
    /usr/local/include/leveldb/options.h:101:30: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      int block_restart_interval = 16;
                                 ^
    /usr/local/include/leveldb/options.h:111:24: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      size_t max_file_size = 2 * 1024 * 1024;
                           ^
    /usr/local/include/leveldb/options.h:127:31: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      CompressionType compression = kSnappyCompression;
                                  ^
    /usr/local/include/leveldb/options.h:133:19: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool reuse_logs = false;
                      ^
    /usr/local/include/leveldb/options.h:138:37: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      const FilterPolicy* filter_policy = nullptr;
                                        ^
    /usr/local/include/leveldb/options.h:148:25: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool verify_checksums = false;
                            ^
    /usr/local/include/leveldb/options.h:152:19: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool fill_cache = true;
                      ^
    /usr/local/include/leveldb/options.h:158:28: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      const Snapshot* snapshot = nullptr;
                               ^
    /usr/local/include/leveldb/options.h:160:19: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      ReadOptions() = default;
                      ^
    /usr/local/include/leveldb/options.h:179:13: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
      bool sync = false;
                ^
    /usr/local/include/leveldb/options.h:181:20: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      WriteOptions() = default;
                       ^
    In file included from plyvel/_plyvel.cpp:589:
    /usr/local/include/leveldb/db.h:56:10: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      DB() = default;
             ^
    /usr/local/include/leveldb/db.h:58:19: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      DB(const DB&) = delete;
                      ^
    /usr/local/include/leveldb/db.h:59:30: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      DB& operator=(const DB&) = delete;
                                 ^
    In file included from plyvel/_plyvel.cpp:593:
    /usr/local/include/leveldb/write_batch.h:37:35: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      WriteBatch(const WriteBatch&) = default;
                                      ^
    /usr/local/include/leveldb/write_batch.h:38:47: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      WriteBatch& operator =(const WriteBatch&) = default;
                                                  ^
    In file included from plyvel/_plyvel.cpp:597:
    /usr/local/include/leveldb/cache.h:35:13: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
      Cache() = default;
                ^
    /usr/local/include/leveldb/cache.h:37:25: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      Cache(const Cache&) = delete;
                            ^
    /usr/local/include/leveldb/cache.h:38:36: warning: deleted function definitions are a C++11 extension [-Wc++11-extensions]
      Cache& operator=(const Cache&) = delete;
                                       ^
    plyvel/_plyvel.cpp:21159:125: warning: offset of on non-POD type 'struct __pyx_obj_6plyvel_7_plyvel_Iterator' [-Winvalid-offsetof]
      if (__pyx_type_6plyvel_7_plyvel_Iterator.tp_weaklistoffset == 0) __pyx_type_6plyvel_7_plyvel_Iterator.tp_weaklistoffset = offsetof(struct __pyx_obj_6plyvel_7_plyvel_Iterator, __pyx_base.__weakref__);
                                                                                                                                ^                                                    ~~~~~~~~~~
    /usr/local/Cellar/llvm@4/4.0.1/bin/../lib/clang/4.0.1/include/stddef.h:120:24: note: expanded from macro 'offsetof'
    #define offsetof(t, d) __builtin_offsetof(t, d)
                           ^                     ~
    35 warnings and 4 errors generated.
    error: command 'clang' failed with exit status 1```

[wbolster]

could you try adding -std=c++11 to setup.py like the other flags here: https://github.com/wbolster/plyvel/blob/1a04cfb09af19568bdbfbd531b115262eefc29a0/setup.py#L18-L19

and then try building plyvel from source? (typing make usually suffices)

[wbolster]

or better yet, try out the branch from #91

[fenidik]

pip3 install git+https://github.com/ryandesign/plyvel.git@fix-build-on-10.6

` clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c plyvel/_plyvel.cpp -o build/temp.macosx-10.13-x86_64-3.7/plyvel/_plyvel.o -Wall -g -x c++ clang: error: no such file or directory: 'plyvel/_plyvel.cpp' clang: error: no input files error: command 'clang' failed with exit status 1

----------------------------------------

Command "/usr/local/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;file='/private/var/folders/44/qt_sltss2ld3mr9dtfhhd21r0000gn/T/pip-req-build-kd6mc_t4/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/44/qt_sltss2ld3mr9dtfhhd21r0000gn/T/pip-record-lj52dzzg/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/44/qt_sltss2ld3mr9dtfhhd21r0000gn/T/pip-req-build-kd6mc_t4/`

[fenidik]

to downgrade leveldb use: brew uninstall leveldb brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86c8b39428a4e485e8afcc02f2e31da9d53007a0/Formula/leveldb.rb

[wbolster]

@fenidik please make a normal checkout and follow the build steps as described in the developer/contributor docs. there is an extra cython build step for git checkouts.

[kyuupichan]

@wbolster I also had this issue installing from PyPi. I added the switch you indicated, built locally and installed, and it works fine.

[kyuupichan]

I should clarify - by works fine, I mean "import plyvel" now returns without error. It used to throw errors about a missing dylib.

[wbolster]

@kyuupichan thanks. re the checkout, did you actually use the branch from #91 or current master?

[kyuupichan]

I just used master - simply git cloned it.

[wbolster]

ok so that doesn't add much then. it's #91 that supposedly fixes this (i don't develop on osx myself)

[kyuupichan]

@wbsolster I can confirm that #91, but only with an extra -std=c++11 works. This includes removal of the platform code.

extra_compile_args = ['-Wall', '-g', '-x', 'c++', '-std=c++11']

I can also confirm that keeping the platform code and adding -std=c++11 to the end of the Darwin list also works. I hope this is helpful.

[IcyW]

to downgrade leveldb use: brew uninstall leveldb brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86c8b39428a4e485e8afcc02f2e31da9d53007a0/Formula/leveldb.rb

Save my life! I am trying to install caffe on my MacOS (Mojave 10.14.3). It failed when make all.