Compile LevelDB if not available

[k4nar]

Hi, During the installation, Plyvel could try to compile LevelDB if it's not already installed. That's what PyZMQ is doing :

When compiling pyzmq (e.g. installing with pip on Linux), it is generally recommended that zeromq be installed separately, via homebrew, apt, yum, etc. If this is not available, pyzmq will try to build libzmq as a Python Extension, though this is not guaranteed to work.

I think it would require quite a bit of work in order to work on a maximum of platform, but it would be a really great advantage for Plyvel. I'll try do to it, but I won't have much time in the next few days, so it might be in a while.

[wbolster]

Yes, it would be nice but it's quite a bit of work. See #7 for an earlier discussion about this topic. My main concern is maintainability. I don't want to have very esoteric and complex build systems that I cannot easily maintain.

[k4nar]

Oh I didn't see this issue, sorry for that.

Well, I understand your concerns. I'll take a better look at how pyzmq manages do to it, and try to find if it would be a burden to maintain.

Of course, compiling LevelDB should be a fallback, and using a version provided by the distribution should always be the main option.

[wbolster]

Yes, to be clear: I'm open to the idea of including a fallback copy, but maintaining a very complex build system (possibly with os dependent hacks, and so on) is not very attractive to me.

[k4nar]

I'll try to ping @minrk to get an insight of how complex it is inside PyZMQ.

[minrk]

I don't know how complex leveldb is to install, but what I do in PyZMQ isn't very complex (it is pretty gross implementation-wise, because I didn't know what I was doing as I put it together). There are two basic approaches to do this:

1. build the library using its own Makefile, etc., and update link/load paths
2. build the library as a Python Extension

I do the second in pyzmq, and I would recommend it if is available to you and configuring leveldb for compilation is not complex. On first glance, it looks like it is pretty simple. Making it an Extension means all the usual compilation customization works without any special instructions.

The gist of it:

• add a detect-library build step
  • if the library isn't found:
  • construct the Extension
  • insert it first in the extension list
  • modify the include and link paths of the other Extensions to point to the bundle
• at runtime, load the Extension with ctypes, as seen here (this is not necessary if you set rpaths correctly, but I haven't done that reliably).

I should write up a simple example package that links a trivial C library, because navigating pyzmq's code that does this is a tangled nightmare, complicated by handling Windows, PyPy, libsodium, etc.

[k4nar]

Wow, thank you very much ! I'll take a deeper look when I have some free time. And yes, a simpler example would be very nice, I think it could help a lot of people :) .

[minrk]

Hopefully this example package has the relevant bits in an intelligible form.

[k4nar]

That's neat, thanks !

[numion]

Maybe you can also use some pieces from here: https://github.com/numion/plyvel It includes leveldb and snappy and works on Linux and Windows.

[k4nar]

@numion : Could you give me some hints to build Plyvel and LevelDB on Windows using your fork ? I tried but without success.

[numion]

@k4nar : There might have been an issue if setuptools was installed in site-packages. Should be fixed now. To build Plyvel, run these commands from a source checkout:

python bootstrap.py
bin\buildout.exe
bin\python.exe setup.py bdist_egg

[k4nar]

Ok, thanks very much, I'll try that. I managed to install it a few days ago, but with a lot of ugly hacks.

[wbolster]

i consider this ‘fixed’ for linux since i do provide ‘manylinux1’ wheels on pypi nowadays.

for other platforms i have no clue/experience, sorry.