robinhood / faust

Python Stream Processing
Other
6.7k stars 538 forks source link

Make Faust compatible with Python 3.9+ for production environment (rocksdb: faust-streaming[rocksdb] ) on MacOS Monterey 12.3.1 #756

Closed dcs3spp closed 2 years ago

dcs3spp commented 2 years ago

Checklist

Is Faust in production environments (rocksdb package dependencies) only compatible with python 3.6-python3.8???

Steps to reproduce

Python 3.9.12+ on macOS Monterey Compile and install latest version of rocksdb from source Install faust-streaming[rocksdb] via poetry add faust-streaming[rocksdb]"

Expected behavior

faust-streaming[rocksdb] should install successfully without further devleopment intervention

Actual behavior

  1. faust-streaming[rocksdb], python-rockdb dependency is unable to locate utilities/backupable_db.h. I have the latest version of rocksdb installed, for which this header is deprecated!
  2. Uninstall rocksdb latest version and install rocksdb 6.29.3I
  3. Attempt to reinstall faust-streaming[rocksdb]
  4. Install of faust-streaming[rocksdb] fails again due to deprecation of PyEval_InitThreads!
        rocksdb/_rocksdb.cpp:54353:3: warning: 'PyEval_InitThreads' is deprecated [-Wdeprecated-declarations]
         PyEval_InitThreads();
          ^
        /Users/me/.pyenv/versions/3.9.12/include/python3.9/ceval.h:130:1: note: 'PyEval_InitThreads' has been explicitly marked deprecated here
        Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
        ^
        /Users/me/.pyenv/versions/3.9.12/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED'
        #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                             ^

Full traceback

        rocksdb/_rocksdb.cpp:54353:3: warning: 'PyEval_InitThreads' is deprecated [-Wdeprecated-declarations]
         PyEval_InitThreads();
          ^
        /Users/me/.pyenv/versions/3.9.12/include/python3.9/ceval.h:130:1: note: 'PyEval_InitThreads' has been explicitly marked deprecated here
        Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
        ^
        /Users/me/.pyenv/versions/3.9.12/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED'
        #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))

Versions

dcs3spp commented 2 years ago

Apologies I have managed to get the faust-streaming[rocksdb] package installed by checking the dependencies were installed, e.g. snappy, lz4. This now works. If anyone else experiences issues, at the time of writing python-rocksdb is only compatible with rocksdb version < v7. After that time backupable_db.h file was removed.