search

fedden / RenderMan

Command line C++ and Python VSTi Host library with MFCC, FFT, RMS and audio extraction and .wav writing.
The Unlicense
355 stars 44 forks source link

ImportError: dynamic module does not define module export function (PyInit_librenderman) #7

Closed drscotthawley closed 6 years ago

drscotthawley commented 6 years ago

Thanks for making this!

I installed boost-python, successfully built the XCode project, renamed the built file to remove the ".dylib", but when I run the "Does It Work?" check, I get...

$ python
Python 3.5.3 | packaged by conda-forge | (default, Feb 10 2017, 07:09:50) 
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import librenderman as rm
JUCE v5.2.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_librenderman)
>>>

Any suggestions on how one might fix this?

In the current directory I have...

$ ls
librenderman.so

Running Mac OS X 10.13.3 (High Sierra), XCode 9.2, Python 3.5.2 (Anaconda)

drscotthawley commented 6 years ago

Seems this is a Python 2 vs. Python 3 thing. https://stackoverflow.com/questions/36723206/cython-compilation-error-dynamic-module-does-not-define-module-export-function

Apparently I need to tell it to use python3 when running setup.py. Challenge: Can't find where you're running setup.py! Where (in the XCode project) does that happen?

fedden commented 6 years ago

Hi Scott, thanks for getting in touch!

Admittedly the repo needs a bit of work and I don't currently have an OSX build system, but I do need to go through this on Linux and have a look at things, as it has been a while since I really evaluated the state of the code. I'll do so later today and get back to you.

fedden commented 6 years ago

Okay, so a few things. Firstly as is, the repository is only going to support Python 2.7.

This is because of both the boost python bindings being set up for Python2.7, and, the makefile that is generated by JUCE supporting Python2.7.

Background

JUCE is a C++ cross platform framework that allows projects to be managed through something called the Projucer. It provided the functionality to make a VST host. If you download the latest version of JUCE you should be able to manage the RenderMan.jucer file at the root of the repository using the Projucer that comes with the source code. This repository also makes use of the Boost library, to make python bindings to the C++ functions defined here. The .jucer file links to the Boost headers that are installed onto your system in the installation process.

Some Targets For Me

I think really it is time to move everything to Python 3, as well as to massively simplify the installation process. I'm admittedly unsure of the correct way at this point. My todo list is essentially:

  1. Figure out a way to author the .jucer file so arguments could be sent to the respective makefile to allow the user to select Python2 / Python3
  2. Create an easy install method via pip or some other method.

Solutions

I have got RenderMan working in Python3 on Linux. It can definitely be done on OSX.

However, with respect to your problem, it's very hard for me to advise as I do not have a working OSX system. But I am here to help!

Here is what I did on Ubuntu 16.04. The steps won't be the same, some paths and environment variables may be different, but the process should be similar.

harritaylor commented 6 years ago

I tried your solution @fedden but ran into a few issues. I am using MacOS 10.13 and Python 3.6 installed via Homebrew. I did eventually get it working by following the following steps (All commands are given for my exact installation, so some modification of commands may be required):

On terminal:

  1. Remove boost-python and install the equivalent library for python3:
brew remove boost-python && brew install boost-python3
  1. Provided that python3 is installed via Homebrew, add a symlink to /usr/local/include pointing to the python3.x headers that Brew has installed. In my case:
sudo ln -s /usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/include/python3.6m /usr/local/include/python3.6
  1. Add a symlink for the boost library:
sudo ln -s /usr/local/Cellar/boost/1.66.0/include/boost /usr/local/include/boost
  1. Add a symlink for the python3.x library that Brew has installed inside the /usr/local/lib folder. Cannot add directly as "python3.x" is being used for site-packages. There is probably a more elegant solution but this is what I came up with: 
sudo ln -s /usr/local/Cellar/python3/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib /usr/local/lib/python3.6/lib

Now, inside Projucer:

  1. Set the "Extra Linker Flags" in Exporters → Xcode(MacOS) to:

    -shared -lpython3.6m -lboost_python3

  2. In both Exporters → Xcode(MacOS) → debug & Exporters → Xcode(MacOS) → release set: 6.1 "header search paths":

    /usr/local/include
/usr/local/include/python3.6

6.2 "Extra library search paths":

/usr/local/include
/usr/local/include/python3.6

Hit save in Projucer, and then Build inside Xcode (⌘B), and hopefully the build will succeed. If not, clean (⌘⇧K) and build again. This is the most system-independent solution I could come up with so far.

drscotthawley commented 6 years ago

@harritaylor @fedden Sorry for the delay, I've been out of it for a bit. I will gladly try out @harritaylor's patch! :-)

fedden commented 6 years ago

No worries at all :)

As a note - I am currently knee-deep amongst papers and dodgy code for my thesis, but I am nearly finished!

After I am done, I will be renovating this library (about 40 days) for a creative deep learning course I'll be instructing, so that all of this can be abstracted away to a simple pip install!

drscotthawley commented 6 years ago

Following @harritaylor's instructions, I was able to get it to build & load on Mac!

$ python3.6
Python 3.6.5 (default, Apr 25 2018, 14:23:58) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import librenderman as rm
JUCE v5.2.0
>>>

Yay! Next up to make it work with my Anaconda python. ;-)

Linux: On Linux, I'm still getting the same error that started this Issue. To be specific.... I pulled the latest Renderman, installed all the dependencies with no errors, built the librenderman.so library, and after fighting errors involving Anaconda's libstdc++.so.6 vs. the system's version of same (hint: solution here), I now, with the updated-and-merged-for-Python3 version of RenderMan, get...

>>> import librenderman as rm
JUCE v5.2.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_librenderman)

i.e., exactly the same error as in the title of this Issue.

Any thoughts? I find it odd that what seems to be a 'python error' is not a problem with python 3.6 on the Mac, but is on Linux.

harritaylor commented 6 years ago

@drscotthawley For the Mac example, you are running Python 2

$ python $ Python 2.7.13 |Anaconda custom (x86_64)| (default, Dec 20 2016, 23:05:08)

Please try running the command with python3 to start your REPL with python3.6, and see if there still is an issue.

Also, side note: I got it running on linux by modifying the makefile and replacing "python2.6" and "boost_python" with "python3.6m" and "boost_python3" respectively, then making the file. A change in the version numbers may be needed for your platform though. I remember it took a bit of fiddling around.

drscotthawley commented 6 years ago

@harritaylor Right, I noticed that error and already went back to delete most of my post, to say that I got it working on Mac once I used the correct python!
I'll try the makefile changes you suggest for Linux, thanks!

harritaylor commented 6 years ago

@drscotthawley Great! Easy mistake. Looking forward to not having to deal with 2 different versions of python soon :-). Let me know if you still have issues with the Linux version.

drscotthawley commented 6 years ago

Got! And what's more, it's with my Anaconda python on Linux! Had to add some -L/opt/anaconda/envs/py36/lib and -I/opt/anaconda/envs/py36/include in various places, but it worked.

Also ...this seems crazy but so far no errors: I linked the libboost_python-py35 libraries even though I'm running python 3.6. (Because I had a hard time finding py36 version of libboost for Ubuntu).

...In summary, it seems the 'dynamic module does not define...' message is essentially coming from boost, because using the same RenderMan code will give you the error or not, depending on whether you've linked against a py27 or a py37 version of boost. Not actually an issue with Renderman itself.

So. I'm going to close this issue.

drscotthawley commented 6 years ago

Just as a follow-up: Whether on Linux or Mac, to get it to work with Anaconda python, you just need to give the build system the locations of the Anaconda libraries.

So, on Mac, in the Projucer, in Exporters > Xcode(Mac), I have the following lines:

Header search paths: /usr/local/include /Users/shawley/anaconda/envs/py36/include /Users/shawley/anaconda/pkgs/python-3.6.0-2/include/python3.6m/

Extra library search paths: /usr/local/lib /Users/shawley/anaconda/envs/py36/lib

fedden commented 6 years ago

Great thanks for this - I'll make sure this is all automated in the coming months :)

jwignall commented 5 years ago

Hi there,

I realise the previous comments were added nearly a year ago, but I am trying to use this library and I had the same Module Import error on Ubuntu. I tried to edit the makefile, but cannot get it to compile. Can someone explain which lines I need to edit? I tried:

I still get errors about not finding -lpython and lboost_python3 etc. I am using standard Python rather than Anaconda.

Thanks in advance! I am relatively new to Linux and haven't needed to do this before! Jake

harritaylor commented 5 years ago

Hi Jake,

I’m away from my Ubuntu machine at the moment, however I’d recommend trying the following steps as you’re installing for python 3.

  1. Install the JUCE app from roli
  2. Open the RenderMan-py36.jucerhttps://github.com/fedden/RenderMan/blob/master/RenderMan-py36.jucer file
  3. Export the required makefile from within the Projucer app

Then this should produce the correct binary for your system. If not, let me know and I’ll have a proper look tomorrow for you.

Cheers, Harri

On 18 Feb 2019, at 18:51, jwignall notifications@github.com<mailto:notifications@github.com> wrote:

Hi there,

I realise the previous comments were added nearly a year ago, but I am trying to use this library and I had the same Module Import error on Ubuntu. I tried to edit the makefile, but cannot get it to compile. Can someone explain which lines I need to edit? I tried:

I still get errors about not finding -lpython and lboost_python3 etc.

Thanks in advance! I am relatively new to Linux and haven't needed to do this before! Jake

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffedden%2FRenderMan%2Fissues%2F7%23issuecomment-464841992&data=01%7C01%7Ctaylorh23%40cardiff.ac.uk%7C3c238e3df8394ec78b0a08d695d21819%7Cbdb74b3095684856bdbf06759778fcbc%7C1&sdata=jVvoTYpad0QUPV7s2G0F2RiSWzY1vvtXWsruW1dbM4g%3D&reserved=0, or mute the threadhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3n0a5KNzOljuRyzGZWu4uV7t_yTLWrks5vOvYwgaJpZM4SAr7y&data=01%7C01%7Ctaylorh23%40cardiff.ac.uk%7C3c238e3df8394ec78b0a08d695d21819%7Cbdb74b3095684856bdbf06759778fcbc%7C1&sdata=gydA%2FNr8sHykwXM6tVzIY28CoQ0kDwjVJ%2B%2BOTvj%2FTes%3D&reserved=0.

jwignall commented 5 years ago

Hi Harri,

Unfortunately this still doesn't work. I followed all three steps, but still get the ImportError when attempting to import the module.

I noticed that the header search path in the Projucer app for this is still Python2.7, However, changing to Python3.5 I get the following error: [path]/librenderman.so: undefined symbol: _ZN5boost6python6detail11init_moduleER11PyModuleDefPFvvE

Any idea how to solve that?

Thanks in advance, I appreciate your time and efforts!

Thanks, Jake

On Mon, 18 Feb 2019 at 21:58, Harri Taylor notifications@github.com wrote:

Hi Jake,

I’m away from my Ubuntu machine at the moment, however I’d recommend trying the following steps as you’re installing for python 3.

  1. Install the JUCE app from roli
  2. Open the RenderMan-py36.jucer< https://github.com/fedden/RenderMan/blob/master/RenderMan-py36.jucer> file
  3. Export the required makefile from within the Projucer app

Then this should produce the correct binary for your system. If not, let me know and I’ll have a proper look tomorrow for you.

Cheers, Harri

On 18 Feb 2019, at 18:51, jwignall <notifications@github.com<mailto: notifications@github.com>> wrote:

Hi there,

I realise the previous comments were added nearly a year ago, but I am trying to use this library and I had the same Module Import error on Ubuntu. I tried to edit the makefile, but cannot get it to compile. Can someone explain which lines I need to edit? I tried:

  • Changing -I/usr/include/python2.7 to -I/usr/include/python3.5 (I'm running Python 3.5.2)
  • Changing -lpython2.7 -lboost_python to -lpython3 -lboost_python3.5 or -lpython3 -lboost_python3.5, lpython3 -lboost_python-py35 etc.

I still get errors about not finding -lpython and lboost_python3 etc.

Thanks in advance! I am relatively new to Linux and haven't needed to do this before! Jake

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffedden%2FRenderMan%2Fissues%2F7%23issuecomment-464841992&data=01%7C01%7Ctaylorh23%40cardiff.ac.uk%7C3c238e3df8394ec78b0a08d695d21819%7Cbdb74b3095684856bdbf06759778fcbc%7C1&sdata=jVvoTYpad0QUPV7s2G0F2RiSWzY1vvtXWsruW1dbM4g%3D&reserved=0>, or mute the thread< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3n0a5KNzOljuRyzGZWu4uV7t_yTLWrks5vOvYwgaJpZM4SAr7y&data=01%7C01%7Ctaylorh23%40cardiff.ac.uk%7C3c238e3df8394ec78b0a08d695d21819%7Cbdb74b3095684856bdbf06759778fcbc%7C1&sdata=gydA%2FNr8sHykwXM6tVzIY28CoQ0kDwjVJ%2B%2BOTvj%2FTes%3D&reserved=0>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/fedden/RenderMan/issues/7#issuecomment-464895538, or mute the thread https://github.com/notifications/unsubscribe-auth/AsegDAYc5oe6_eKOpYXkXWiqIsoHOg8nks5vOyILgaJpZM4SAr7y .