NAMD / pypelinin

Python library to distribute jobs and pipelines among a cluster
3 stars 5 forks source link

Create a job Queue for remote/parallel job execution #61

Open fccoelho opened 11 years ago

fccoelho commented 11 years ago

It would be very nice to create a simple decorator people could use to submit jobs, as many distributed processing libraries have: People could decorate functions they want to offload to a pipelinin queue for distributed processing. I have something similar in my Liveplots package: https://github.com/fccoelho/liveplots/blob/master/src/liveplots/xmlrpcserver.py

def enqueue(f):
    """Decorator that places the call on a queue"""
    def queued(self,*args,**kw):
        Q.put((f,(self,)+args))
    queued.__doc__ = f.__doc__
    queued.__name__ = f.__name__
    return queued

where Q is a Queue object.

Also a customized map/reduce (this could be lifted from other python map/reduce libraries) framework would see great adoption.