lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Bug: The argument of reduce function has to be named `results`. #1091

Closed Milky2018 closed 1 year ago

Milky2018 commented 1 year ago

This Lithops program works well and outputs 27:

import lithops

def f(x):
    return x + 7

def h(results):
    return sum(results)

if __name__ == "__main__":
    fexec = lithops.FunctionExecutor()
    fexec.map_reduce(f, [1, 2, 3], h)
    print(fexec.get_result())

But if we modify the definition of h into:

def h(non_results):
    return sum(non_results)

Which just change the name of the argument. Then it will fail with traceback:

Traceback (most recent call last):
  File "/var/task/lithops/worker/jobrunner.py", line 213, in run
  File "/var/task/lithops/worker/jobrunner.py", line 120, in _wait_futures
KeyError: 'results'

And here is the log:

Activation: 'lithops-default-runtime-v39' (3bcbe913-83e6-4b48-b49e-e5a0bcc2fa41)
[
    2023-04-08 08:32:24,382 [INFO] handler.py:152 -- Lithops v2.9.0 - Starting AWS Lambda execution
    2023-04-08 08:32:24,382 [INFO] handler.py:153 -- Execution ID: 813b03-0-R000/00000
    2023-04-08 08:32:24,411 [INFO] aws_s3.py:67 -- S3 client created - Region: us-east-1
    2023-04-08 08:32:24,530 [INFO] jobrunner.py:119 -- Reduce function: waiting for map results
    ----------------------- EXCEPTION !-----------------------
    Traceback (most recent call last):
      File "/var/task/lithops/worker/jobrunner.py", line 213, in run
        self._wait_futures(data)
      File "/var/task/lithops/worker/jobrunner.py", line 120, in _wait_futures
        fut_list = data['results']
    KeyError: 'results'
    ----------------------------------------------------------
    2023-04-08 08:32:24,553 [INFO] jobrunner.py:311 -- Process finished
]

There is no reason for programmers to name the argument of a function to be exactly some specific name.