search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.01k stars 3.49k forks source link

python3 is broken #1152

Closed ClaymorePT closed 9 years ago

ClaymorePT commented 9 years ago

Python 3 package successfully builds but fails to copy the file _decimal.cpython-34.so. File is removed after being compiled along with other python binary modules.

Also, python3-package.mk is redefining the PYTHON_VERSION value to PYTHON3_VERSION which creates a problem when having Python2 and Python3 on the same system. This also prevents boost from being interfaced with both Python 2 and 3 at the same time.

sbyx commented 9 years ago

@commodo

commodo commented 9 years ago

PYTHON3_VERSION has been named differently than PYTHON_VERSION so that users would have the option of including both python-package.mk and python3-package.mk in the same package Makefile and package for both Python 2 and 3. I will admit that I do not know of a case where this is done, and I haven't had the chance to try it. If having PYTHON_VERSION and PYTHON3_VERSION is a problem, I will need a few more details about it.

_decimal.cpython-34.so builds [and installs on the target] when builing on my machine, but I'm on a OpenSUSE x64 host, so I may not be seeing issues regarding Python cross-compilation when building on this distro. Python's setup.py script is too "smart" and tries to do some voodoo magic for cross-compilation, and may vary on some other distros. Could you tell me some details about the build env where you're seeing this build failure ? i.e.: build host (distro version + arch) and target build (arch) Or maybe I should try to build boost to see the err ?

Based on that I'll try to spin up a VM and see/fix the build failure(s).

Thanks

ClaymorePT commented 9 years ago

Thanks for the info. I'm actually in need of using those includes for boost. Since boost can be integrated with multiple python builds. It is possible to give boost enough information during the configuration phase, to integrate with both Python 2.x and Python 3.x at the same time.

My target was x86 and I tried to build python 3 separately (no boost selected). I tried to post the compile log here but it was too big. You can get it from my dropbox: https://www.dropbox.com/s/kx5rzqbpbt6d6wg/python.txt?dl=0

Cybjit has a working config for python3 and other python3 based modules. Take a look at it, it may help you. https://forum.openwrt.org/viewtopic.php?id=36958 https://www.gitorious.org/openwrt-cybjit/python3/source/3ae05dde036e3a2684f79765885f370f2078d352:

Regards, Carlos

commodo commented 9 years ago

The error there is :

/home/ArkHDD/OpenWRT/trunk/build_dir/target-i386_i486_uClibc-0.9.33.2/Python-3.4.3/Modules/_decimal/libmpdec/mpdecimal.c:49:22: fatal error: fenv.h: No such file or directory
     #include <fenv.h>

I recall fixing this recently here: https://github.com/openwrt/packages/issues/1077 Could you try a sync with a more recent Python 3.4 version ? Or, could this be a weird regression ?

ClaymorePT commented 9 years ago

You were right. I forgot to update my trunk. I tried it and it now compiles and installs well.

I will test it with boost and try to include the python includes, to automate the python integration.

Thanks for the info

ClaymorePT commented 9 years ago

Sorry for reopening this issue, but I was building python3 in an updated trunk and it failed with the error

File not found '/home/ArkHDD/OpenWRT/trunk/build_dir/target-i386_i486_uClibc-0.9.33.2/Python-3.4.3/ipkg-install/usr/lib/python3.4/lib-dynload/_curses.cpython-34.so'

I searched for the logs and noticed this

In file included from /home/ArkHDD/OpenWRT/trunk/build_dir/target-i386_i486_uClibc-0.9.33.2/Python-3.4.3/Modules/_curses_panel.c:15:0:
./Include/py_curses.h:50:20: fatal error: curses.h: No such file or directory
 #include <curses.h>
                    ^
compilation terminated.

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_lzma                 _tkinter              nis                
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_curses               _curses_panel

I checked the python3-package-ncurses.mk file and noticed that this module depends upon building libncursesw. Shouldn't it be libncurses instead? I selected libncurses in the Libraries section and after that, python3 ncurses module compiled just fine.

Could you please check this out?

Thank you!

commodo commented 9 years ago

[No worries about re-opening this]

Actually the problem here is a bit more complicated. It's also present with Python. And it's a problem with how libncurses & libncursesw are packaged. They're BUILD_VARIANTs of the same source code. I hate BUILD_VARIANTs, because once libncurses (or libncursesw is built) the other one won't want to build under some conditions (which I have yet to understand). OpenWRT base packages reference libncurses, and some packages in this repo reference either libncurses either libncursesw.

The problem with Python is that once it sees libncursesw it will prefer it over libncurses which is why I changed it to libncursesw. The quick fix here, would be to force Python to libncurses. Don't think there are people that need libncursesw yet with Python. Maybe our Asian (or Russian) friends ? [as they have a completely different charset]

Maybe also split libncursesw from libncurses even though that creates a bit more duplicate stuff. Will come back later with some findings/results.

Thanks

commodo commented 9 years ago

Created PR: https://github.com/openwrt/packages/pull/1157 It seems to work fine.

ClaymorePT commented 9 years ago

I see, it's a messy problem ...

I have one sugestion. Instead of depending on libncurses, why not depending on ncurses directly? Wouldn't that cause both libs to be built? I understand that it would probably create some extra garbage, but if one is building python, then I assume flash space is not a problem...

On 21 April 2015 at 09:15, Alexandru Ardelean notifications@github.com wrote:

Created PR: #1157 https://github.com/openwrt/packages/pull/1157 It seems to work fine.

— Reply to this email directly or view it on GitHub https://github.com/openwrt/packages/issues/1152#issuecomment-94698959.

Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira

commodo commented 9 years ago

It doesn't look like there's a ncurses package. Just libncurses and libncursesw. ncurses is the top folder; I'm not sure if I can reference it as a package dependency directly.

Even so, for the moment I'd say it's fine with forcing libncurses on python + python3 until I can think of a more robust solution. I wanted to try to push libncursesw as the default [at one time], but then I got to a point where I dropped it, because of not enough time. I may re-visit that later.

Regarding flash space being a problem with python (or 3): [even] I am surprised at the usage of Python on barely-sufficient-flash-storages; that's one of the reasons why both python and python3 are split [so granularly] as they are.

ClaymorePT commented 9 years ago

Cybjit here https://www.gitorious.org/openwrt-cybjit/python3/source/3ae05dde036e3a2684f79765885f370f2078d352:lang/python3/Makefile links python3 with libncurses. I never noticed that. Since I always used his makefile to build Python 3 for openwrt and it always worked for me, my opinion is that keeping the dependency for libncurses and not the 'w' version is probably the best way to go.

Thanks for the effort!

On 21 April 2015 at 11:20, Alexandru Ardelean notifications@github.com wrote:

It doesn't look like there's a ncurses' package. Justlibncursesandlibncursesw. ncurses` is the top folder; I'm not sure if I can reference it as a package dependency directly.

Even so, for the moment I'd say it's fine with forcing libncurses on python + python3 until I can think of a more robust solution. I wanted to try to push libncursesw as the default [at one time], but then I got to a point where I dropped it, because of not enough time. I may re-visit that later.

Regarding flash space being a problem with python (or 3): [even] I am surprised at the usage of Python on barely-sufficient-flash-storages; that's one of the reasons why both python and python3 are split [so granularly] as they are.

— Reply to this email directly or view it on GitHub https://github.com/openwrt/packages/issues/1152#issuecomment-94734476.

Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira

Wedmer commented 9 years ago

Also I've found strange bug. If liblzma is already compiled, then we will recieve

Package python3-light is missing dependencies for the following libraries: liblzma.so.5

Way to reproduce: select xz-utils (selects liblzma) and python3 packages then run

make package/python3/clean;make package/xz/compile;make package/python3/compile;

Workaround:

make package/xz/clean;make package/python3/clean;make package/python3/compile;

ClaymorePT commented 9 years ago

Probably a bug related to the dependency check. I also noticed a bug with the Boost Makefile. Since Boost.Locale requires the libiconv-full to be built, the added dependency in the Makefile should had been enough, but for some reason, it does not obey... This may actually be a strange regression, I don't know...

On 23 April 2015 at 16:38, Vladimir Ulrich notifications@github.com wrote:

Also I've found strange bug. If liblzma is already compiled, then we will recieve

Package python3-light is missing dependencies for the following libraries: liblzma.so.5

Way to reproduce: select xz-utils (selects liblzma) and python3 packages then run

make package/python3/clean;make package/xz/compile;make package/python3/compile;

Workaround:

make package/xz/clean;make package/python3/clean;make package/python3/compile;

— Reply to this email directly or view it on GitHub https://github.com/openwrt/packages/issues/1152#issuecomment-95627840.

Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira

commodo commented 9 years ago

I'm guessing I'd have to add a python3-package-lzma.mk file to split the lzma Python3 module out and add liblzma as a dep. The setup.py script will include this lib if it finds it.

ClaymorePT commented 9 years ago

Didn't cybjit solved those issues in his repo ?

On 24 April 2015 at 09:38, Alexandru Ardelean notifications@github.com wrote:

I'm guessing I'd have to add a python3-package-lzma.mk file to split the lzma Python3 module out and add liblzma as a dep. The setup.py script will include this lib if it finds it.

— Reply to this email directly or view it on GitHub https://github.com/openwrt/packages/issues/1152#issuecomment-95853557.

Carlos Miguel Ferreira Researcher at Telecommunications Institute Aveiro - Portugal Work E-mail - cmf@av.it.pt Skype & GTalk -> carlosmf.pt@gmail.com LinkedIn -> http://www.linkedin.com/in/carlosmferreira

commodo commented 9 years ago

Seems he removed it. Although +liblzma seems to be present in the new packages, it does not show up in menuconfig. Some bug I guess.

commodo commented 9 years ago

For python3 at least this seems to help : https://github.com/openwrt/packages/pull/1188 Also added https://github.com/openwrt/packages/pull/1189 to add the python3-lzma package.