FBThrift 'py3' support needed: No module named 'openr.thrift'

[guodong]

Issue Description

Hi, I've build openr on my machine according the README. After openr/py installed, the breeze command throws several module not found error like:

File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/clients/openr_client.py", line 17, in <module>
ModuleNotFoundError: No module named 'openr.thrift'

It seems some python modules for py3 part is missing and I have no clue where is openr.thrift module and like openr.thrift.OpenrCtrlCpp.clients?

Can someone give me some hints to fix so that maybe I can contribute.

Environment

• tag or commit hash on which this occured master
• OS version: ubuntu 1804

Minimal test code / Steps to reproduce the issue

After built, run any breeze cmd

What's the actual result?

throws module not found error

$ breeze kvstore keys
Traceback (most recent call last):
  File "/usr/local/bin/breeze", line 33, in <module>
    sys.exit(load_entry_point('py-openr==1.0', 'console_scripts', 'breeze')())
  File "/home/gd/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 473, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/gd/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/home/gd/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2447, in load
    return self.resolve()
  File "/home/gd/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2453, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/cli/breeze.py", line 22, in <module>
  File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/cli/clis/config.py", line 12, in <module>
  File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/cli/commands/config.py", line 16, in <module>
  File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/cli/utils/utils.py", line 28, in <module>
  File "/usr/local/lib/python3.6/dist-packages/py_openr-1.0-py3.6.egg/openr/clients/openr_client.py", line 17, in <module>
ModuleNotFoundError: No module named 'openr.thrift'

What's the expected result?

no module issues.

[cooperlees]

breeze (and OpenR components) all use fbthrift (fork of Apache thrift) to communicate. I believe these modules are missing as we’re not building the ‘py3’ thrift language client libraries and structures.

These are a little more involved as they are Cython libraries linked to the C++ libraries. I haven’t yet, but need to workout how to get these building in OSS world.

We could see if we can just add thrift compiler to use ‘py3’ as well, but I feel we’ll need cpp2 first for py3 to build.

[sgiannoulis]

I also see the same problem with making breeze operational. Thrift python libraries are not built, and trying to manually build them for the thrift repo is not working as the setup script requires specific built files to be in a specific folder within the trift repo (old way was all deps were in /build directory of openr and actually being installed, so the setup.py was working). When trying to run the thrift setup.py file, python cannot find module X ( where X was trift, then its dependency fb303, then neteng etc). Please provide an integrated solution for breeze to be operational through your install scripts.

[cooperlees]

I have attempted to do exactly this and the problem is much deeper. py3 thrift is Cython wrapped C++ and is off by default in the Open Source fbthrift builds. I've approached the team for support in making this work, but I have not been successful thus far unfortunately.

Some of my work can be seen in this branch: https://github.com/facebook/openr/tree/breeze_docker

I will try find cycles again some day soon, but without FB Thrift teams help, it's going to need to be very hacky. I want this for my home OpenR install, but may end up writing a lightweight non thrift py3 breeze for OSS to work around it.

[sgiannoulis]

I have managed in the past to get fbthrift to compile the python modules as well. (it was with the Release Candidate 20191208-10906 codebase). It needed to find the python-six package during configuration of cmake and then you were able to run the related setup.py to build and install also the related python wrappers. If that helps, that was the cmake call that enabled python support build (running cmake inside the build dir of fbthrift root dir. You needed to have python-six project built and:

cmake -Dpython-six_DIR=/opt/python-six/python-six-Q8Be2n47Opv-n5B-m-qxwvUlULZRCVNgM-AharcWT7Q/lib/cmake/python-six/ -DBUILD_SHARED_LIBS=ON ../

[cooperlees]

Ok, we have thrift py3 modules compiling in the Docker container - Just a few more install placements for the thrift files ans we should have breeze workable in Docker.

We can then use these learning to try make this more friendly and widely usable.