search

faust-streaming / python-rocksdb

Python bindings for RocksDB used by faust-streaming
https://faust-streaming.github.io/python-rocksdb/
BSD 3-Clause "New" or "Revised" License
24 stars 5 forks source link

Installation instructions for Windows and macOS environments #11

Open eblis opened 1 year ago

eblis commented 1 year ago

I'm trying to install faust-streaming-rocksdb package on Windows and I'm getting various errors.

  1. First error was related to pkg-config missing, for which I installed this lite version

  2. Now I'm getting errors with my Visual Studio compiler, namely this:

    The following error occurred when trying to handle this error:

EnvCommandError

Command <local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9\Scripts\python.exe -m pip install --use-pep517 --disable-pip-version-check --prefix <local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9 --no-deps <local user>\AppData\Local\pypoetry\Cache\artifacts\a3\db\2a\56fd9cba93b126c74d8376724d69cb1aa975a488d396619bbdee3836fa\faust-streaming-rocksdb-0.9.1.tar.gz errored with the following return code 1, and output:
Processing <local user>\AppData\Local\pypoetry\cache\artifacts\a3\db\2a\56fd9cba93b126c74d8376724d69cb1aa975a488d396619bbdee3836fa\faust-streaming-rocksdb-0.9.1.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: faust-streaming-rocksdb
Building wheel for faust-streaming-rocksdb (pyproject.toml): started
Building wheel for faust-streaming-rocksdb (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error

Building wheel for faust-streaming-rocksdb (pyproject.toml) did not run successfully.
exit code: 1

[99 lines of output]
<local user>\AppData\Local\Temp\pip-build-env-7s172fm3\overlay\Lib\site-packages\setuptools\config\setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
  warnings.warn(msg, warning_class)
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-39
creating build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\errors.py -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\interfaces.py -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\merge_operators.py -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\__init__.py -> build\lib.win-amd64-cpython-39\rocksdb
creating build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\test_db.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\test_memtable.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\test_options.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\test_transactiondb_options.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\test_transaction_db.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
copying rocksdb\tests\__init__.py -> build\lib.win-amd64-cpython-39\rocksdb\tests
running egg_info
writing faust_streaming_rocksdb.egg-info\PKG-INFO
writing dependency_links to faust_streaming_rocksdb.egg-info\dependency_links.txt
writing requirements to faust_streaming_rocksdb.egg-info\requires.txt
writing top-level names to faust_streaming_rocksdb.egg-info\top_level.txt
reading manifest file 'faust_streaming_rocksdb.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'faust_streaming_rocksdb.egg-info\SOURCES.txt'
<local user>\AppData\Local\Temp\pip-build-env-7s172fm3\overlay\Lib\site-packages\setuptools\command\build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'rocksdb.cpp' as data is deprecated, please list it in `packages`.
    !!

    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'rocksdb.cpp' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'rocksdb.cpp' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'rocksdb.cpp' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.

!!

  check.warn(importable)
copying rocksdb\_rocksdb.cpp -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\_rocksdb.pyx -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\advanced_options.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\backup.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\cache.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\checkpoint.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\compaction_filter.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\comparator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\concurrent_task_limiter.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\env.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\filter_policy.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\iterator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\logger.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\memtablerep.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\merge_operator.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\options.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\slice_.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\slice_transform.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\snapshot.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\stackable_db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\statistics.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\status.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\std_memory.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\table_factory.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\table_properties.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\transaction_db.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\types.pxd -> build\lib.win-amd64-cpython-39\rocksdb
copying rocksdb\universal_compaction.pxd -> build\lib.win-amd64-cpython-39\rocksdb
creating build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\comparator_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\filter_policy_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\memtable_factories.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\merge_operator_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\slice_transform_wrapper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\utils.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
copying rocksdb\cpp\write_batch_iter_helper.hpp -> build\lib.win-amd64-cpython-39\rocksdb\cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build\temp.win-amd64-cpython-39
creating build\temp.win-amd64-cpython-39\Release
creating build\temp.win-amd64-cpython-39\Release\rocksdb
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Irocksdb -I.\rocksdb -I<local user>\AppData\Local\pypoetry\Cache\virtualenvs\main-sfnLs_Sh-py3.9\include "-IC:\Program Files\Python39\include" "-IC:\Program Files\Python39\Include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\cppwinrt" /EHsc /Tprocksdb/_rocksdb.cpp /Fobuild\temp.win-amd64-cpython-39\Release\rocksdb/_rocksdb.obj -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing -fno-rtti
cl : Command line error D8021 : invalid numeric argument '/Wextra'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for faust-streaming-rocksdb
Failed to build faust-streaming-rocksdb
ERROR: Could not build wheels for faust-streaming-rocksdb, which is required to install pyproject.toml-based projects
eblis commented 1 year ago

Maybe it's possible to use vcpkg when installing on Windows ?

wbarnha commented 1 year ago

Sorry to see you're having this issue, I didn't expect this library to be used on Windows hence why it's incompatible. I'll see if I can extend the CI/CD configuration to support Windows somehow.

wbarnha commented 1 year ago

Looks like https://github.com/AustEcon/python-rocksdb/tree/develop was attempting to build python-rocksdb for Windows, could probably try where they left off and cross-reference with https://github.com/facebook/rocksdb/wiki/Building-on-Windows.

mohammad-aqajani commented 1 year ago

Same on macos apple M1, any solution?

wbarnha commented 1 year ago

I'm currently working on creating wheels for macOS and Windows. The plan currently is to get macOS wheels rolled out first, then Windows, since macOS is typically easier to get initially working.

@mohammad-aqajani you can test this project's wheel builds for macOS via pipx run cibuildwheel --platform macos --archs aarch64. You may experience some issues with delocate as I did in https://github.com/faust-streaming/python-rocksdb/actions/runs/3743356084/jobs/6355429481. I don't have immediate access to a macOS development environment, so adding support is a bit difficult on my end. If you'd like to assist, feel free to join the Faust Slack.

wbarnha commented 1 year ago

There are notes to support macOS on https://github.com/twmht/python-rocksdb/issues/47

martinffx commented 1 year ago

Does this support Rocksdb 7 installed from homebrew?

wbarnha commented 1 year ago

This project doesn't have support for RocksDB 7 unfortunately, but I have been contemplating an upgrade.

~I should note this project uses RocksDB 6, you may see errors similar to https://github.com/twmht/python-rocksdb/issues/105 for example if you use versions of rocksdict later than 0.3.2 (which uses RocksDB 7) in tandem with this library.~ Edit: rocksdb==0.3.7 is safe to use alongside this project now.