Python 2.7: No module named urllib

[enigmagroup]

Has urllib vanished? It seems there is no python-urllib package, even before the big Python updates in OpenWRT. So I assume it should come bundled with Python?

Traceback (most recent call last):
  File "/opt/enigmabox/webinterface/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 8, in <module>
    from django.core.management.base import BaseCommand, CommandError, handle_default_options
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 16, in <module>
    from django.utils.encoding import smart_str
  File "/usr/lib/python2.7/site-packages/django/utils/encoding.py", line 2, in <module>
    import urllib
ImportError: No module named urllib

[commodo]

So, after this error report : https://github.com/openwrt/packages/issues/466 I checked the python package and it seems that I need to re-do a lot of it.

urllib was never in python nor python-mini from what I found. It is in the python-full package, but I think I will remove python-full since it can easily allow bloat, and then re-add the python-mini package which includes some common-sense libs like urllib and others.

I hope I can get this done in a couple of days.

Thanks Alex

[enigmagroup]

Okay! Thanks!

[roger-]

How about a python-base with the bare essentials, python-stdlib-mini with some of the more common libraries and python-stdlib with everything?

[commodo]

Yep, I was thinking of something like that. I started with python3 that way, and I'll do the same with the python package.

I was thinking thatpython-base be called python. I like the python-stdlib-mini name more than python-mini; tho I think users will ask for python-mini because a lot of old packages reference it. Right now I am creating the python-base package, and will get to python[-stdlib]-mini in a while.

Thanks Alex

[roger-]

Sounds good, though I think maybe python should be the complete Python (including base and stdlib) so people aren't confused about why there're so many things missing?

I think then python could be a meta package that requires python-base/mini and python-stdlib (the latter which would require python-stdlib-mini).

[enigmagroup]

@roger +1

[commodo]

Sure, why not. That sounds good too. Unless someone has a better [counter-]argument, I can go about that way.

@roger- I wanted to ask you this for a while: would you like to move micropython under the Python submenu ? It makes sense to have those packages there, since there's an entire submenu for Python. Maybe some packages could be shared later between python3 and micropython.

Thanks Alex

[roger-]

would you like to move micropython under the Python submenu ?

Very good idea, will do.

Maybe some packages could be shared later between python3 and micropython.

Possibly some pure Python packages will work, will have to investigate that.

[commodo]

Allow me to be the first to say that a couple of days turned into more than a week. Sorry about that. Some events/bugs on OpenVSwitch kept me busy + work @ work

In case you want to preview/monitor a bit closer my progress, or help, here's my branch where I'm working on updating the python package. https://github.com/commodo/packages/commits/python-update-4

Thanks Akex

[simper]

@commodo Is there a quick modification method in current Makefile so we will get urllib supported in compiled python? I am trying experiment something in openwrt python which requires urllib heavily.

[commodo]

if the bloat is not too expensive for you, you can use python-full until I repackage

[simper]

I use following commands to update and install newest feeds in trunk branch: ./script/feeds update -a ./script/feeds install -a Then in language section of menuconfig, there is following module selected: python python-mini python-openssl But after the compile and flash, in python CLI, importing urllib still fails, am I doing something wrong in configuration?

[commodo]

isn't there a python-full in make menuconfig ? if there is, you can pick that one;

maybe it would be a good idea to add urllib also under python-openssl right now it's not

[simper]

My feeds conf: src-git packages https://github.com/openwrt/packages.git src-git luci https://github.com/openwrt/luci.git src-git routing https://github.com/openwrt-routing/packages.git src-git telephony http://git.openwrt.org/feed/telephony.git src-git management https://github.com/openwrt-management/packages.git And the options in Language section of menuconfig after feeds update and install: So I assume python = python-full here?

[commodo]

are you sure you don't have any local packages in there ? or that your feeds are updating ?

that python package looks like is before some modifications I made; looks like some other package is overlapping the python package that's in the official packages

could you delete the feeds directory and run update -a and install -a again ?

right now, python packages show up like this:

• if you don't select the python package
• if you select the python package

In the second image there's a python-full package.

Sorry about the weird overlay, that's KSnapshot's fault. Will do better for next snapshots.

[simper]

@commodo Thanks, I refreshed feeds folder and get python-full finally. But yes, python + python-full is more than 3M... It is too bloat to fit into a 8M flash... When will the optimised python with urllib available in feeds? Is it coming recently?

[commodo]

Seems there are now 2+ people interested in this. simper & enigmagroup

I'll try to make some time to finish it up by tomorrow.

The problem is that packaging in general ( and packaging python too ) takes some time to run + re-run several builds to try to get things right.

[enigmagroup]

Thank you very much for your efforts! I know by myself that it always takes time, so don't worry, no hurry.

[commodo]

@simper What do you want to install/package ?

I was thinking yesterday a bit. Seems the effort to create a generic python-mini is big, and in the end it's pointless, because a generic python-mini package will never meet anyone's needs on OpenWRT.

Python is bloated, so making fit onto 8 MB of flash, and meet everyone's needs is impossible.

So, I think we should do things differently:

• python package is the interpreter with the minimum to start; this is done already
• python-full package can be left around for those who have 256 MB of flash
• for everyone else we would need to enforce a new methodology of packaging Python targeted at the end-needs

What I mean by the last part is that if you want to install Django on your target, you should need 3 packages at least:

• python package : the interpreter
• python-base-django : minimum amount of Python libraries to run Django
• python-django : the actual Django package

A python-mini package for Django may be too much for someone who just needs pyserial for example. So for those people they would only need python + python-base-pyserial + pyserial and then there would be enough space left on the flash to do whatever else is needed.

Until simper replies I guess I could start creating a python-base-django package. Maybe an idea would be to add some sort of way for users to create their own python-base-***** packages.

[simper]

@commodo I just need the urllib module to available in a size controllable manner. In fact I not sure if we package all standard python libraries into one package, how big it will be? Is it the original python-full? Or we make separate packages according to the standard python library sections, like python-internet, python-file, python-multimeadia and etc? So every sub package will be relatively small. django is not the standard library and should be packaged in separate package for sure.

[commodo]

Created this pull request: https://github.com/openwrt/packages/pull/514

Adding all those Python libraries adds about 11 MB. I don't know how much space that would take up on the flash, since the files on flash are compressed.

Making separate packages like python-internet, python-file, python-multimeadia and etc sounds like a good idea, but I (personally) do not think this will help with bloat reduction. For example: if the user would need 2 Python libs that are 4 MB together, but the actual files that are needed from each lib are less than 1 MB total, then the user has 3 MB of bloat.

The way the packages are organized here is not 100% how they are organized as files/folders in /usr/lib/python2.7. And even so, they are interdependent, so for example if you need python-file you also need python-io, which also means having to manually check all Python libs interdependency. Then when the Python version changes all packaging needs to be re-evaluated and so do the libs inter-dependencies.

So, I think all this packaging should be driven by the application(s) users are trying build on OpenWRT. As a packager, it's easier for me to try to package Python in a way that works for everybody if possible, and not have to re-do things whenever a new Python comes along. For now, the 2 easiest packages are python and python-full and these 2 are easy to validate/verify on each Python version, but I think a way for each users to customize Python libs/files would be awesome.

[commodo]

One final change. This is the last one, and is final.

python is now python-base python-full is now python (it includes urllib, urlib2, etc)

For devices with small Flash ROMs, python-base should be used + some special packaging. I will create a new method for the "special" packaging. In the next week I'll add 2 packages which will also be examples on how to create packages on top of python-base.

[roger-]

python is now python-base python-full is now python (it includes urllib, urlib2, etc)

+1

I will create a new method for the "special" packaging.

modulefinder might be useful for automating the creation of these packages (if you weren't aware of it).

BTW have you considered only packaging the Python bytecode for the various libraries? Might save some space. Another idea: write a small installer that will figure out the dependencies for a package, download the .py source files, byte compile and then install them, all from the device.

Great work BTW!

[commodo]

Awesome: i was actually thinking about writing a modulefinder code myself at some point. Did not know about it, will check it out. Thanks a million for informing me about it.

Also, the idea of downloading specific files for packages did occur to me as well, and now with the modulefinder, it looks like it's one step closer to it, since I may not have to write that part, just setup a location from where to download the .py files.

I haven't considered packaging Python bytecode yet. The idea sounds interesting. Maybe later when I'll have some time I'll do some size benchmarks to compare. I think it's difficult to tell how much size will be saved, since (for me) I first have to start reading (or getting informed more) about how the whole compressed file systems actually work and where and how stuff gets stored. I have some general ideas/assumptions about it.

Thanks :)

[roger-]

Maybe later when I'll have some time I'll do some size benchmarks to compare.

Just did a quick test with some py/pyc files from Python 2.7 (top is uncompressed, bottom is compressed with 7z -- comparable to SquashFS/JFFS2 I believe). Looks like the .py source compresses much better, and the .pyc bytecode is only slightly better uncompressed (~10% smaller).

Might be worth looking into this more, but it looks like the bytecode is gonna be worse on compressed filesystems.

[roger-]

Tried again with optimized bytecode (.pyo files made with -O -OO) and they seem like they might be worth considering. About ~13% better compressed and ~25% better uncompressed (vs .py).

BTW does your package disable the creation of .pyc/.pyo files? I can see distributing the bytecode only as being advantageous to avoiding that (and loading faster).

[commodo]

So that makes me look kinda lazy, since it took not a lot of effort to compare file-sizes. I was probably thinking of doing something a bit more in-depth, but this is good too. Thanks a lot for the compares :) Not sure how the compression gets influenced when all the target image/files are added.

At the moment I am disabling .pyc/.pyo files and distributing just .py source files. This was (more or less) the way it was done before and how I usually see Python installed on machines.

Now that you're mentioning it, distributing .py files creates bloat because they get compiled on the target, and you end up having both. So, distributing the .pyo files does make more sense. And having an "installer" around could help if the source is wanted.

[roger-]

creates bloat because they get compiled on the target, and you end up having both

Yeah, at the very least it might be good to configure Python not to create them by default.

[commodo]

So, I pushed a patch to openwrt-devel mailing list to add a line with: export PYTHONDONTWRITEBYTECODE=1 Let's hope this gets accepted pretty soon. Link is this: http://patchwork.openwrt.org/patch/6638/

This way of doing it is pretty pythonic (since they put it in there).

As for .pyo files in packages, it looks like the best way is to do it via installer/script on the target. I tried to install directly .pyo files and while they actually got there, problem is they were compiled with the host Python, and not the target Python, and that made them incompatible. There may be a way to cross-compile Python files. I haven't found a way, nor did I manage to put in extra effort (this week) for that.

However, looking at it from a package's perspective, putting the .py files on the target via packages/packaging seems like the best/honest way to do it. From there on, it should be up to the user to optimize his Python install to run faster or takes less space, etc. From a packager's perspective we could write/define some scripts to help with that, but that would be the limit where I think a packager should not cross, since there's too many variables (flash sizes, RAM sizes, etc). And then in the end, Python is Python and everybody expects it to behave and look the same on all platforms.

[roger-]

problem is they were compiled with the host Python, and not the target Python, and that made them incompatible.

Hmm... are you sure that was the problem? I've done some searching (e.g. the bottom here) and the bytecode is supposed to be portable across platforms (but not Python versions).

[commodo]

Not sure 100% Could have been something else.

Maybe some config differences between host python and target python.

[roger-]

Could have been something else.

Yeah I'd guess the version would have to match exactly.

[commodo]

Not sure. I may retry later.

You're welcome to try it :) Right now I don't think I will invest any more effort in this.

My current interest is to scale up python (a bit more) and after that sync python3 with python. Any help is fully appreciated to update Python in any way that works best for OpenWRT and keeps Python standard. Searching for middle ground there is a bit tricky.

[commodo]

Quick update: http://patchwork.openwrt.org/patch/6638/

The proposal got refused, so patching python's will be it.

[roger-]

The proposal got refused, so patching python's will be it.

Another option would be to run a post-install script that modifies /etc/profile, might be cleaner than having to maintain a source patch.

Any help is fully appreciated

Sure, I'll do some experimentation and see what I can come up with. Maybe a python-stdlib-pyc (or something) package for those wanting to save a few extra KB (and reduce startup time) would be good. Issue now seems to be getting the pyc/pyo compiled with the right version of Python (which the build-bots may not have).

[commodo]

So, I made this : https://github.com/openwrt/packages/pull/541

After you mentioned the postinstall script, I checked on it. Seems it's not that popular, I was not aware of it until you mentioned it.

Thanks :)