lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Add additional arguments to pass to the reduce function in map_reduce() #1022

Closed bystepii closed 1 year ago

bystepii commented 1 year ago

This change allows additional arguments to be passed to the reduce function.

For example:

import lithops

args = [  # Init list of parameters for Lithops
    (1, 2),  # Args for function1
    (3, 4),  # Args for function2
    (5, 6),  # Args for function3
]  # End list of parameters for Lithops

def my_function(x, y, extra_param):
    print(f"Extra param: {extra_param}")
    return x + y

def my_reduce_function(results, extra_param):
    print(f"Extra param: {extra_param}")
    return sum(results)

if __name__ == "__main__":
    fexec = lithops.FunctionExecutor()
    fexec.map_reduce(my_function, args, my_reduce_function, extra_args=(1,), extra_args_reduce=(2,))
    print(fexec.get_result())

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
JosepSampe commented 1 year ago

Cool! Thanks @bystepii