search

opticspy / lightpipes

LightPipes for Python, "Pure Python version"
https://opticspy.github.io/lightpipes/
BSD 3-Clause "New" or "Revised" License
228 stars 53 forks source link

build linux wheels #3

Closed guyskk closed 7 years ago

guyskk commented 7 years ago

I had build LightPipes-1.1.0-cp36-cp36m-linux_x86_64.whl successfully. Take notes:

  1. install cython and numpy: pip install cython numpy
  2. install libfftw3, the package name in different os can be found in http://rosindex.github.io/d/libfftw3/
  3. run python setup.py bdist_wheel

Next step, I will try to build wheels using docker, it can create manylinux1 wheels.

FredvanGoor commented 7 years ago

I am busy as well. I made setup.py multi-platform (linux + win 32 for now) Don't we interfere?

guyskk commented 7 years ago

No, it won't. I only follow README to build source code but meet difficult in installing libfftw3, so I take some notes. Sorry for misleading you.

FredvanGoor commented 7 years ago

Good to hear! Yes you need different fftw3 libraries for each platform. Maybe we better download the fftw3 source code from fftw.org and compile/link it with lightpipes? I have never tried it because the binaries worked well.

I could build the linux wheel LightPipes-1.1.0-cp34-cp34m-linux_x86_64.whl as well with the current setup.py. The windows wheel failed. I will check that... (In the past it worked!)

In both windows and linux I could successfully do the following build commands with the multi-platform setup.py: windows: py -3 setup.py build_ext --inplace linux: python3 setup.py build_ext --inplace

and test with Young.py

guyskk commented 7 years ago

But fftw3 seems hard to compile, especially in Windows, binaries is OK.

guyskk commented 7 years ago

I build manylinux1 wheels successful, the scripts are pushed.

Usage:

  1. install docker
  2. run bash build.sh waiting...for a long time, all wheels will be in wheelhouse dir
guyskk commented 7 years ago

LightPipes-1.1.0-cp27-cp27m-manylinux1_x86_64.whl LightPipes-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl LightPipes-1.1.0-cp35-cp35m-manylinux1_x86_64.whl LightPipes-1.1.0-cp36-cp36m-manylinux1_x86_64.whl LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl wheelhouse.zip

FredvanGoor commented 7 years ago

Good work! I will try to install on my linux mint machine

FredvanGoor commented 7 years ago

Hi guyskk,

I tried to install the 3 of the wheels on my linux mint machine ("Monster-pc") using pip. The cp27mu wheel installed successfully. The cp27m and the cp34m did not work on my machine. I have python 3.4.3 and python 2.7.6 installed. Below are the commands I gave.

Fred.

fred@Monster-pc ~/Downloads/wheelhouse $ sudo -H pip2 install LightPipes-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl Processing ./LightPipes-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl Installing collected packages: LightPipes Successfully installed LightPipes-1.1.0

fred@Monster-pc ~/Downloads/wheelhouse $ sudo -H pip2 uninstall LightPipes Uninstalling LightPipes-1.1.0: /usr/local/lib/python2.7/dist-packages/.libsLightPipes/libfftw3-b7228d9c.so.3.5.6 /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/DESCRIPTION.rst /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/INSTALLER /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/METADATA /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/RECORD /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/WHEEL /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/metadata.json /usr/local/lib/python2.7/dist-packages/LightPipes-1.1.0.dist-info/top_level.txt /usr/local/lib/python2.7/dist-packages/LightPipes.so Proceed (y/n)? y Successfully uninstalled LightPipes-1.1.0

fred@Monster-pc ~/Downloads/wheelhouse $ sudo -H pip2 install LightPipes-1.1.0-cp27-cp27m-manylinux1_x86_64.whl LightPipes-1.1.0-cp27-cp27m-manylinux1_x86_64.whl is not a supported wheel on this platform.

fred@Monster-pc ~/Downloads/wheelhouse $ sudo -H pip3 install LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl is not a supported wheel on this platform.

guyskk commented 7 years ago

do those works?:

FredvanGoor commented 7 years ago

Hi guyskk, I renamed wheel, installed successfully, but failed to import LightPipes in python 3.4:

fred@Monster-pc ~/Downloads $ mv LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl LightPipes-1.1.0-cp34-cp34m-linux_x86_64.whl

fred@Monster-pc ~/Downloads $ sudo -H pip3 install LightPipes-1.1.0-cp34-cp34m-linux_x86_64.whl Unpacking ./LightPipes-1.1.0-cp34-cp34m-linux_x86_64.whl Installing collected packages: LightPipes Successfully installed LightPipes Cleaning up...

fred@Monster-pc ~/Downloads $ python3 Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.

import LightPipes Traceback (most recent call last): File "", line 1, in File "init.pxd", line 155, in init LightPipes (LightPipes.cpp:9973) ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96 quit()

fred@Monster-pc ~/Downloads $ sudo pip3 install --upgrade numpy . . . Successfully installed numpy Cleaning up...

fred@Monster-pc ~/Downloads $ python3 Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.

import LightPipes Traceback (most recent call last): File "", line 1, in File "init.pxd", line 155, in init LightPipes (LightPipes.cpp:9973) ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96 quit()

guyskk commented 7 years ago

please run this script and paste it's result: https://gist.github.com/guyskk/7d87b6c8ca26beec9bb70d1598d2f805

FredvanGoor commented 7 years ago

Here is the info of my platform (linux mint "Monster-PC") --------------------platform information-------------------- architecture = ('64bit', 'ELF') dist = ('LinuxMint', '17', 'qiana') java_ver = ('', '', ('', '', ''), ('', '', '')) libc_ver = ('glibc', '2.9') linux_distribution = ('LinuxMint', '17', 'qiana') mac_ver = ('', ('', '', ''), '') machine = x86_64 node = Monster-pc platform = Linux-3.13.0-24-generic-x86_64-with-LinuxMint-17-qiana processor = x86_64 python_branch = python_build = ('default', 'Nov 17 2016 01:08:31') python_compiler = GCC 4.8.4 python_implementation = CPython python_revision = python_version = 3.4.3 python_version_tuple = ('3', '4', '3') release = 3.13.0-24-generic system = Linux system_alias = version = #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 win32_ver = ('', '', '', '') --------------------package information--------------------- setuptools = 3.3 pip = 1.5.4 wheel = 0.29.0 virtualenv Not Found distutils = 3.4.3 cython = 0.25.2

(program exited with code: 0) Press return to continue

FredvanGoor commented 7 years ago

And here the info of my windows10 laptop:

--------------------platform information-------------------- architecture = ('64bit', 'WindowsPE') dist = ('', '', '') java_ver = ('', '', ('', '', ''), ('', '', '')) libc_ver = ('', '') linux_distribution = ('', '', '') mac_ver = ('', ('', '', ''), '') machine = AMD64 node = UT-laptop platform = Windows-10-10.0.14393-SP0 processor = Intel64 Family 6 Model 42 Stepping 7, GenuineIntel python_branch = v3.5.1 python_build = ('v3.5.1:37a07cee5969', 'Dec 6 2015 01:54:25') python_compiler = MSC v.1900 64 bit (AMD64) python_implementation = CPython python_revision = 37a07cee5969 python_version = 3.5.1 python_version_tuple = ('3', '5', '1') release = 10 system = Windows system_alias = version = 10.0.14393 win32_ver = ('10', '10.0.14393', 'SP0', 'Multiprocessor Free') --------------------package information--------------------- setuptools = 20.2.2 pip = 9.0.1 wheel = 0.29.0 virtualenv Not Found distutils = 3.5.1 cython = 0.23.4 Press any key to continue . . .

FredvanGoor commented 7 years ago

And here the info of my Mac Mini: --------------------platform information-------------------- architecture = ('64bit', '') dist = ('', '', '') java_ver = ('', '', ('', '', ''), ('', '', '')) libc_ver = ('', '') linux_distribution = ('', '', '') mac_ver = ('10.12.2', ('', '', ''), 'x86_64') machine = x86_64 node = 192.168.2.18 platform = Darwin-16.3.0-x86_64-i386-64bit processor = i386 python_branch = python_build = ('default', 'Jul 30 2016 19:40:32') python_compiler = GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34) python_implementation = CPython python_revision = python_version = 2.7.10 python_version_tuple = ('2', '7', '10') release = 16.3.0 system = Darwin system_alias = version = Darwin Kernel Version 16.3.0: Thu Nov 17 20:23:58 PST 2016; root:xnu-3789.31.2~1/RELEASE_X86_64 win32_ver = ('', '', '', '') --------------------package information--------------------- setuptools = 18.7.1 pip = 7.1.2 wheel = 0.26.0 virtualenv Not Found distutils = 2.7.10 cython = 0.23.4

guyskk commented 7 years ago

Your mint's pip version seems too lower, try upgrade to latest version.

Here is my laptop(arch linux): $ python diagnose.py --------------------platform information-------------------- architecture = ('64bit', 'ELF') dist = ('arch', '', '') java_ver = ('', '', ('', '', ''), ('', '', '')) libc_ver = ('glibc', '2.3.4') linux_distribution = ('arch', '', '') mac_ver = ('', ('', '', ''), '') machine = x86_64 node = localhost.localdomain platform = Linux-4.9.6-1-ARCH-x86_64-with-arch processor = python_branch = python_build = ('default', 'Jan 16 2017 12:12:55') python_compiler = GCC 6.3.1 20170109 python_implementation = CPython python_revision = python_version = 3.6.0 python_version_tuple = ('3', '6', '0') release = 4.9.6-1-ARCH system = Linux system_alias = version = #1 SMP PREEMPT Thu Jan 26 09:22:26 CET 2017 win32_ver = ('', '', '', '') --------------------package information--------------------- setuptools = 34.0.2 pip = 9.0.1 wheel = 0.29.0 virtualenv = 15.1.0 distutils = 3.6.0 cython = 0.25.2

guyskk commented 7 years ago

And I installed LightPipes in mint in VirtualBox succeed.

mint@mint ~/Downloads $ pip3 install LightPipes-1.1.0-cp35-cp35m-manylinux1_x86_64.whl 
Processing ./LightPipes-1.1.0-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: LightPipes
Successfully installed LightPipes
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
mint@mint ~/Downloads $ python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import LightPipes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "__init__.pxd", line 155, in init LightPipes (LightPipes.cpp:9973)
ImportError: No module named 'numpy'
>>> 
mint@mint ~/Downloads $ python3 diagnose.py 
--------------------platform information--------------------
architecture = ('64bit', 'ELF')
dist = ('LinuxMint', '18.1', 'serena')
java_ver = ('', '', ('', '', ''), ('', '', ''))
libc_ver = ('glibc', '2.9')
linux_distribution = ('LinuxMint', '18.1', 'serena')
mac_ver = ('', ('', '', ''), '')
machine = x86_64
node = mint
platform = Linux-4.4.0-53-generic-x86_64-with-LinuxMint-18.1-serena
processor = x86_64
python_branch = 
python_build = ('default', 'Nov 17 2016 17:05:23')
python_compiler = GCC 5.4.0 20160609
python_implementation = CPython
python_revision = 
python_version = 3.5.2
python_version_tuple = ('3', '5', '2')
release = 4.4.0-53-generic
system = Linux
system_alias = 
version = #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016
win32_ver = ('', '', '', '')
--------------------package information---------------------
setuptools Not Found
pip = 8.1.1
wheel = 0.29.0
virtualenv Not Found
distutils = 3.5.2
cython Not Found
guyskk commented 7 years ago

pip 8.1 and later is required: https://github.com/pypa/manylinux#manylinux

Wheel packages compliant with those tags can be uploaded to PyPI (for instance with twine) and can be installed with pip 8.1 and later.

FredvanGoor commented 7 years ago

I tried to upgrade pip, but Mint says:

Monster-pc ~ # sudo pip3 install --upgrade pip Downloading/unpacking pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144 Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded Installing collected packages: pip Found existing installation: pip 1.5.4 Not uninstalling pip at /usr/lib/python3/dist-packages, owned by OS Successfully installed pip Cleaning up...

And the pip-version remains 1.5.4! What to do? Can I take ownership? How? (I tried to do it in root as well (sudo -i), but same message)

guyskk commented 7 years ago

run this script: https://bootstrap.pypa.io/get-pip.py see also: https://pip.pypa.io/en/stable/installing/

guyskk commented 7 years ago

if not work, try sudo pip uninstall pip , then run sudo python3 get-pip.py to install a new pip.

FredvanGoor commented 7 years ago

I installed pip version 9.0.1 using sudo python3 get-pip.py.

Then I did: fred@Monster-pc ~/Downloads/wheelhouse $ sudo -H pip3 install LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl Processing ./LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl Installing collected packages: LightPipes Successfully installed LightPipes-1.1.0

Testing: fred@Monster-pc ~/Downloads/wheelhouse $ python3 Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.

import LightPipes Traceback (most recent call last): File "", line 1, in File "init.pxd", line 155, in init LightPipes (LightPipes.cpp:9973) ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96

Still a problem with numpy? installed newest numpy version: 1.12.0

FredvanGoor commented 7 years ago

Hi guyskk,

I tried to install Docker on my linux mint. However no success! Should I change my linux mint into linux ubuntu or so?

guyskk commented 7 years ago

Linux Mint is a community-driven Linux distribution based on Debian and Ubuntu

so the installation guide for Ubuntu may be working: https://docs.docker.com/engine/installation/linux/ubuntu/

If still fail, you'd better change Mint into Ubuntu, I had installed docker in many Ubuntu and Debian servers, it won't be difficult as long as you follow the installation guide.

guyskk commented 7 years ago

I can reproduce ValueError: numpy.dtype has the wrong size by install an older version numpy, the reason may be: http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling

Python 3.6.0 (default, Jan 16 2017, 12:12:55) 
[GCC 6.3.1 20170109] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import LightPipes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "__init__.pxd", line 155, in init LightPipes (LightPipes.cpp:9973)
ValueError: numpy.dtype has the wrong size, try recompiling. Expected 88, got 96
>>> import numpy
>>> numpy.__version__
'1.9.3'
>>>

But I'm sure LightPipes is compile against numpy 1.12.0 , it is strange!

guyskk commented 7 years ago

This is compile against numpy 1.7.1, have a try, numpy 1.7.1 or later(1.12.0) should work: LightPipes-1.1.0-cp34-cp34m-manylinux1_x86_64.whl.zip

guyskk commented 7 years ago

@FredvanGoor Now you can build 64bits and 32bits parallel, just open 2 terminals run build-linux-x32.sh and build-linux-x64.sh respectively.