support for simple closures

[GoogleCodeExporter]

This originally was a segfault issue, but in trying to reproduce it I got a 
different error. I'll release the original source in a few weeks.

Anyway, here's a case:

def map_fail(config):
    #a=[config[i] for i in xrange(len(config))]
    a=map(lambda i: config[i], xrange(len(config)))

def main():
    config = [1,2,6,8,5,9,9,6,5,4,8]
    map_fail(config)

if __name__=='__main__':
    main()

Basically, the map can't handle the closure-based lambda and has weird issues 
with config's type. But if you comment it out and use the list comprehension, 
shedskin can create the necessary closure by basically implementing map in the 
code (FOR_IN_NEW).

As far as I know, map(some_func,some_iterable) is equivalent to 
[some_func(iteration) for iteration in some_iterable] so converting to list 
comprehensions for instances where closure is used for the lambda could make 
shedskin more robust.

Original issue reported on code.google.com by fahh...@gmail.com on 15 Nov 2010 at 2:18

[GoogleCodeExporter]

thanks! :D

closures are officially not supported at the moment (please see the tutorial), 
but I agree they can be useful in shortcuts like this, and it should be 
technically possible to support them in most cases.. the combination with 'map' 
is just one case that can be easily rewritten, so I prefer to think about a 
more general solution. another useful case that is harder to rewrite is in 
combination with 'sorted' (as a key argument).

Original comment by mark.duf...@gmail.com on 16 Nov 2010 at 10:15

• Changed title: support for simple closures
• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I know closure is not supported, but this simple situation is easily fixed 
through an equivalent rewrite.

Closures could be added by making closure-based lambdas/defs into classes with 
the operator () set and it's initializer called with the closure data 
necessary. Though with the operator (), I'm not sure why __call__ isn't allowed 
in shedskin.

Original comment by fahh...@gmail.com on 16 Nov 2010 at 10:20

[GoogleCodeExporter]

shedskin currently maps function references to stateless C function pointers. 
they could and should be replaced with pointers to python-like class instances 
implementing __call__, but it will be some work.

Original comment by mark.duf...@gmail.com on 16 Nov 2010 at 10:33

[GoogleCodeExporter]

Actually, shedskin doesn't allow classes that have a __call__ method, or at 
least doesn't allow for calling such classes:

class CALLME(object):
    def __call__(self):
        return 1

if __name__=='__main__':
    tmp = CALLME()
    print tmp()

Shedskin will complain about about an unbound identifier 'tmp' only on tmp() 
lines. I've tried other variable names, and 'CALLME()()' is worse with 'meuk' 
being unbound. If __call__ classes were allowed, closure could be pretty close 
behind.

Original comment by fahh...@gmail.com on 17 Nov 2010 at 8:33

[GoogleCodeExporter]

I've tried a similar closure-fixing approach that shedskin also has issues with:

def some_func(config, sequence): return config[0]
class ClosureLambda(object):
    def __init__(self, sequence):
        self.sequence = sequence
    def __call__(self,config):
        return some_func(config, self.sequence)

sequence = (0,1)
population = [[1,2],[0,1]]
population.sort(key=ClosureLambda(sequence))

This, in it's original context, creates code where the ClosureLambda class is 
supposed to be an __ss_int and is implicitly-casted as such. I'm not sure how 
that comes about, but it's probably because shedskin doesn't really understand 
__call__. In the resulting C++ code, there is no reference to the __call__ 
code, nor an operator() implementation.

Original comment by fahh...@gmail.com on 17 Nov 2010 at 1:52

[GoogleCodeExporter]

yup, __call__ is not at all supported anywhere. it should give a warning though 
when you try to overload it. not sure why it doesn't here.. I will have a look.

it will be quite an operation to support __call__ and simple closures. not sure 
if I will be able to look into this any time soon..

Original comment by mark.duf...@gmail.com on 18 Nov 2010 at 7:38

[GoogleCodeExporter]

Isn't __call__ just the equivalent to implementing operator()? Once __call__ is 
implemented, any closure-based function can be translated into a global class 
(with a properly muddled name) that takes any variables being closured in 
__init__ and runs normally in __call__ with references to self. for closured 
variables.

Original comment by fahh...@gmail.com on 18 Nov 2010 at 7:42

[GoogleCodeExporter]

yes, perhaps 'quite an operation' is a bit of an exaggeration, but it would 
probably keep me busy for a few days..

Original comment by mark.duf...@gmail.com on 18 Nov 2010 at 7:51

[GoogleCodeExporter]

I'd offer to help if studies didn't keep me too busy to look at shedskin's 
source much. If this and a few other things aren't implemented in a month or 
two, I may be able to help. This would really expand the scope of shedskin's 
ability to understand/convert many python libraries.

Once this and more built-in libraries are added, shedskin's use becomes almost 
unlimited, probably overtaking much of Cython's usage.

Original comment by fahh...@gmail.com on 18 Nov 2010 at 7:55

[GoogleCodeExporter]

the problem with __call__ and closures is not so much how to generate code, but 
how to improve type inference to deal with them correctly without breaking 
other things. I don't it would be easy to help with this one..

I certainly won't be looking into this for 0.7 (bugfix release, hopefully out 
in three weeks). but perhaps for 0.8.. it would be nice to finally also have a 
look at supporting __iter__ and method references at the same time, so we can 
remove some of the last limitations from the tutorial that are bugging me.. ^^

Original comment by mark.duf...@gmail.com on 20 Nov 2010 at 1:57

[GoogleCodeExporter]

Maybe if all functions were defined as classes with __call__ defined and only 
in the generation of C++ side do they 'simplify' to functions when they don't 
have a need for __init__. As long as __init__ is only used for functions that 
need it to handle closure, then it that should be possible.

I'm not sure if this helps, but it may remove the need for two objects that 
both support () but are only subtly different.

Original comment by fahh...@gmail.com on 21 Nov 2010 at 9:13

[GoogleCodeExporter]

I have two unexpected days free now, so I think I will try to improve things 
for overloading __iter__ and __call__ a bit.. this seems a good first step, 
before generalizing function pointers and finally adding support for closures. 

Original comment by mark.duf...@gmail.com on 20 Dec 2010 at 8:09

[GoogleCodeExporter]

I hope it's not a bad unexpected free time.

Once __call__ can be overloaded, it'll allow users to replace functions with 
class instances and later need some way to find all 'undefined type' variables 
inside the inner function/lambda and pull the variable from outside in, or 
whatever makes more sense in terms of shedskin

Original comment by fahh...@gmail.com on 21 Dec 2010 at 10:28

[GoogleCodeExporter]

no not bad, thanks :) yeah, it all looks doable, but as I found out from 
looking at this again, it may be a lot of work, and my motivation is 
traditionally not that high right after a release.. we'll see :) 

Original comment by mark.duf...@gmail.com on 21 Dec 2010 at 10:33

[GoogleCodeExporter]

btw, note that method references and being able to contain function/method 
references in, say, a list would also be easily implemented once we get off the 
function-pointer train and into the __call__ world.. so going here would 
probably remove several other limitations from the tutorial as well.

Original comment by mark.duf...@gmail.com on 21 Dec 2010 at 12:09

[GoogleCodeExporter]

status report for day 1 ;) I've got basic __iter__ and __call__ overloading 
working (see tests/195.py), so I can play with making being iterable and/or 
callable (java-like) interfaces.. unfortunately that seems a bit harder in C++ 
than in java.. I will try to read a bit about this subject later today.

Original comment by mark.duf...@gmail.com on 21 Dec 2010 at 7:21

[GoogleCodeExporter]

okay, so multiple inheritance to mix in abstract classes/interfaces is a no-go 
for now (because of using the incompatible va_args macros in many places), at 
least until c++0x becomes the standard. not that big of a deal though (for 
example it means we can't have a class that is both callable and iterable), so 
I will sidestep this for now. 

today I added two C++ base-classes for being callable with one or two args, and 
as you can see in tests/195.py, using this, we can now pass our callable 
instances to the 'key' argument of max. next step (not sure when) will be to 
output callable classes instead of function pointers for anonymous functions, 
and modify all builtins that accept callables to use __call__ on them. then, I 
think the analysis can be cleaned up to use __call__ everywhere as well. and 
_then_ we can start adding state to them, such as an instance pointer. __then__ 
I think finally we can use this to implement closures.. :-P it's starting to 
look like the holy grail of writing a compiler.. ;-)

Original comment by mark.duf...@gmail.com on 22 Dec 2010 at 4:32

[GoogleCodeExporter]

The standard libraries have *v() versions of functions to support va_args being 
passed in, something like this:

int wrap_printf(fmt,...) {
    va_list args;
    va_start(args, fmt);
    printfv(fmt, args);
    va_end(args);
    return 1;
}

Does that help any?

So it seems that a recurring problem in shedskin is limiting things, would it 
help if I created some code that created an N-argument callable base-class that 
would be included in the output files instead of in the builtin code?

Original comment by fahh...@gmail.com on 23 Dec 2010 at 3:07

[GoogleCodeExporter]

thanks, but the problem is exactly with this va_args stuff.. it's type unsafe, 
because you have to know the types of args that are being passed, so you can 
cast them, but that doesn't even work anymore with multiple inheritance 
(because you need to know the original type to do the cast then).

Original comment by mark.duf...@gmail.com on 23 Dec 2010 at 8:07

[GoogleCodeExporter]

I made a few improvements in 0.7.1 into the direction of finally supporting 
closures, but completely supporting closures seemed too much work for the 
moment.. perhaps for 0.8. it would be a very nice goal for a 1.0 release in any 
case!

Original comment by mark.duf...@gmail.com on 27 Feb 2011 at 12:38