improve support for standard library

[GoogleCodeExporter]

this issue replaces several existing issues related to improving the support 
for standard library modules.

it should be possible to support much of the following modules, but we'd need 
to add some tweaks to shed skin. I personally won't be working on this, unless 
there is some compelling test case:

-operator (issue 50)
-struct (issue 60)
-array (issue 79)

the following cannot be supported in general. but note that as of 0.6, it 
should be possible to do basic pickling of compiled classes.

-pickle (issue 61)

I haven't looked into the following, but if anyone is interested in working on 
them, please let me know. I'd be happy to try and assist of course:

-bz2 (issue 74)
-async (issue 88)

finally, there was a suggestion to try and compile non-local modules. I don't 
think is useful, because we can just add modules to lib/, if it is possible to 
compile them, but for completeness here it is:

-issue 39

Original issue reported on code.google.com by mark.duf...@gmail.com on 27 Oct 2010 at 12:58

[GoogleCodeExporter]

(with the pickling I mean from CPython, after using shedskin -e)

Original comment by mark.duf...@gmail.com on 27 Oct 2010 at 1:07

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

The attached patch adds binascii support. I'm going to try implementing a 
partial hashlib so I needed binascii's hexlify for the hexdigest function.

Original comment by fahhem%r...@gtempaccount.com on 4 Mar 2011 at 2:17

Attachments:

• 0001-Adding-binascii-builtin-module-new-define-flag-for-u.patch

[GoogleCodeExporter]

that's great, thanks a lot! :D 

I made some changes to get it to work here, and without depending on the Python 
development files (which I prefer not to without using shedskin -e), and pushed 
the result. please diff to see the exact differences with your code, and send 
new patches or discuss these changes to improve things from here.

note there is an initial hashlib implementation in shedskin/lib (used by 
examples/rsync.py), which you might build on for hashlib support. it depends on 
openssl development files. when hashlib is used, shedskin/cpp.py adds -lssl to 
the Makefile.c

so anyway, thanks again! :) how is the program you'd like to compile 
progressing..? how large is it by now?

Original comment by mark.duf...@gmail.com on 4 Mar 2011 at 11:03

[GoogleCodeExporter]

I don't see any hashlib stuff in lib. Is it a local change?

I'm writing one that has md5 and the sha family that comes default with python. 
What does the default one have?

Original comment by fahhem%r...@gtempaccount.com on 4 Mar 2011 at 12:41

[GoogleCodeExporter]

sorry about that, I meant shedskin/examples/lib. this contains several 
partially finished modules needed to compile some of the examples.

Original comment by mark.duf...@gmail.com on 4 Mar 2011 at 12:50

[GoogleCodeExporter]

There's a warning for redefining PY_SSIZE_T_MAX so this patch is a quick-fix.

Original comment by fahhem%r...@gtempaccount.com on 4 Mar 2011 at 12:54

Attachments:

• 0001-Fix-redefinition-of-PY_SSIZE_T_MAX.patch

[GoogleCodeExporter]

I have some questions about the library stuff, why don't they need the __to_py 
and all those shedskin<->python functions? Say I create a hashlib_user.py that 
has a function that returns an md5() instance, won't it crash since Python 
doesn't understand an md5 instance?

Original comment by fahhem%r...@gtempaccount.com on 4 Mar 2011 at 1:07

[GoogleCodeExporter]

it will probably complain about not being able to export the function, which 
will then just be invisible on the CPython side..

currently, only builtin scalars and container types are exported (please see 
the tutorial for an overview of other limitations when generating extension 
modules). it would certainly be possible to support exporting other classes in 
lib/, but I'm not sure it would be worth the effort.. 

if you'd like to support this in binascii/hashlib anyway, I'd be happy to 
accept patches. but please do keep everything that needs python development 
files inside #ifdef __SS_BIND, so it's only included when using shedskin -e 
(see lib/builtin.?pp for examples). the export check in shedskin/extmod.py 
should also be changed for this to work, of course.

Original comment by mark.duf...@gmail.com on 7 Mar 2011 at 10:56

[GoogleCodeExporter]

I just got another request for 'struct', and am considering implementing it (in 
a somewhat restricted but fully usable form) for 0.8.

Original comment by mark.duf...@gmail.com on 27 Apr 2011 at 3:12

[GoogleCodeExporter]

btw, fahrzin, do you have some basic tests for binascii I could add to 
tests/196.py? or did you perhaps send them already, but I lost them? :-)

Original comment by mark.duf...@gmail.com on 1 May 2011 at 1:14

[GoogleCodeExporter]

okay, it looks like we'll have pretty good struct support for 0.8.. I just 
added the basic TI stuff to look at format-strings, and this now works in GIT:

header_format = "<32s2BHHH24s"
s1, b1, b2, h1, h2, h3, s2 = struct.unpack(header_format, 64*'0')
print struct.calcsize(header_format)

it would be great if someone else would like to look into finishing things on 
the C++ side.

Original comment by mark.duf...@gmail.com on 1 May 2011 at 1:17

[GoogleCodeExporter]

No, my tests were part of the hashlib I stopped working on. My own tests were 
mainly just:

st = 'lakcbpaicrbodoel/rc'
assert b2a_uu(a2b_uu(st))==st

Other tests were a little harder since I had to rename my code so I could 
import both python's builtin binascii and mine.

Original comment by fah...@recreclabs.com on 1 May 2011 at 8:32

[GoogleCodeExporter]

I tried to add some tests to tests/196.py (which can then automatically be 
compared between cpython and shedskin using tests/run.py), but I'm getting 
different results.. perhaps you can quickly see what's going wrong here..?

import binascii
b2a = binascii.b2a_uu('my guitar wants to kill your mama')
print repr(b2a)
a2b = binascii.a2b_uu(b2a)
print repr(a2b)

Original comment by mark.duf...@gmail.com on 2 May 2011 at 6:11

[GoogleCodeExporter]

"./run.py 196" on git HEAD doesn't work (http://pastebin.com/EY4g69AS) so I'm 
looking into it using 197.py.

Original comment by fah...@coinquarry.com on 14 May 2011 at 9:58

[GoogleCodeExporter]

thanks! hmm, because this was the first problem to be fixed after 0.7.1, I'm 
guessing 'shedskin' still points to 0.7.1 on your system..?

Original comment by mark.duf...@gmail.com on 14 May 2011 at 10:04

[GoogleCodeExporter]

The test runs the global shedskin instead of the local one? I would assume the 
repo's tests would run local stuff only.

Original comment by fah...@coinquarry.com on 14 May 2011 at 10:10

[GoogleCodeExporter]

you are right, I never thought about this, because they are always the same on 
my system:

http://gitorious.org/shedskin/mainline/commit/eb70419454b28e667a3b117c211f27d5e5
b8c285

Original comment by mark.duf...@gmail.com on 14 May 2011 at 10:19

[GoogleCodeExporter]

Fix for _uu functions being broken: http://pastebin.com/XhEpUzu3

Basically leftchar is supposed to be an int, not a char, both functions had the 
same mistake.

Original comment by fah...@coinquarry.com on 14 May 2011 at 11:10

[GoogleCodeExporter]

thanks, pushed! it would be great if you could add a few more tests. otherwise, 
I will probably try to add some later..

Original comment by mark.duf...@gmail.com on 14 May 2011 at 11:58

[GoogleCodeExporter]

hmm, keyword arguments don't seem to work... http://pastebin.com/CtYLjVPT

I made some quick tests here: http://pastebin.com/1VG7GaMy

Original comment by fahhem%r...@gtempaccount.com on 14 May 2011 at 12:40

[GoogleCodeExporter]

thanks! I added 'binascii' to some ugly list, so the default arguments work 
now.. there are still three functions failing.. perhaps you could have a look 
at those? I commented them out in 196.py for now. 

Original comment by mark.duf...@gmail.com on 14 May 2011 at 1:23

[GoogleCodeExporter]

Issue 108 has been merged into this issue.

Original comment by mark.duf...@gmail.com on 16 Jul 2011 at 9:58

[GoogleCodeExporter]

issue 108 is about the 'errno' module, so I merged it in here.

Original comment by mark.duf...@gmail.com on 16 Jul 2011 at 9:59

[GoogleCodeExporter]

I propose add support for urllib module.

Original comment by miros1...@gmail.com on 27 Aug 2011 at 1:26

[GoogleCodeExporter]

I recognize that chances may be slim, but I'm going ahead and putting in a 
request for support for the gzip module. It looks like there was previously 
some interest in supporting bz2 (see issue 74; 
http://code.google.com/p/shedskin/issues/detail?id=74), but it looks like it 
may have not been feasible.

I have very large files that ideally need to stay compressed to preserve 
storage space. It is convenient to be able to use gzip.readline() or 
gzip.next() to iterate through these files without decompressing them. I can 
currently achieve my personal analysis goals by compiling a subset of my python 
code as an extension with ShedSkin and just letting cPython handle the gzip 
operations. But it would ideally be better if I were able to fully compile my 
code and include gzip functions within an executable binary. The main reasons 
come down to ease of distribution: if I can compile all of my code into an 
executable, I can just pass along the binaries to someone else so that they can 
use my program. This is useful when the recipients of the programs are not 
capable of compiling or modifying things on their end. 

Again, just a request. It is something that I can overcome, but I figure it 
never hurts to ask!

-Mark

Original comment by MarkPerr...@gmail.com on 24 Sep 2012 at 4:21

[GoogleCodeExporter]

no I think bz2 is probably feasible, gzip as well. but I personally won't start 
working on them. if anyone is interested, there is some information in the 
shedskin documentation wiki about how to approach this, and I guess in lib/ 
there are lots of examples.

Original comment by mark.duf...@gmail.com on 28 Sep 2012 at 5:52

[GoogleCodeExporter]

another request that came in as a separate issue: cgi (issue 174)

Original comment by mark.duf...@gmail.com on 28 Sep 2012 at 5:57