Lithops configuration

[oanamoc]

Hello, I am trying to set up lithops with Was Lambda and S3. I ran this command to test it, according to the instructions: lithops test -b aws_lambda -s aws_s3

The problem is it gets stuck on a progress bar and it won't finish. I have waited over 40 minutes by this point.

oanamoc@pc-10-200 lithops % lithops test -b aws_lambda -s aws_s3
2023-07-05 14:48:30,509 [INFO] config.py:134 -- Lithops v2.9.0 2023-07-05 14:48:30,866 [INFO] aws_s3.py:67 -- S3 client created - Region: eu-west-3 2023-07-05 14:48:31,766 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: eu-west-3 2023-07-05 14:48:31,768 [INFO] invokers.py:108 -- ExecutorID 4c7835-0 | JobID A000 - Selected Runtime: lithops-default-runtime-v39 - 256MB 2023-07-05 14:48:35,518 [INFO] invokers.py:172 -- ExecutorID 4c7835-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations 2023-07-05 14:48:35,521 [INFO] invokers.py:208 -- ExecutorID 4c7835-0 | JobID A000 - View execution logs at /private/var/folders/th/p9dg33jj77q100w8147qwjkw0000gn/T/lithops-oanamoc/logs/4c7835-0-A000.log 2023-07-05 14:48:35,522 [INFO] wait.py:97 -- ExecutorID 4c7835-0 - Getting results from 1 function activations

0%|                                                                  | 0/1  

Does anyone know what could be the problem?

[JosepSampe]

Hi @oanamoc,

Can you run: lithops clean -b aws_lambda -s aws_s3 And then try again but with the debug flag: lithops test -b aws_lambda -s aws_s3 -d

[oanamoc]

It stopped here and has been writing the same thing for the past 20 minutes:

oanamoc@pc-10-200 ~ % cd /Users/oanamoc/Desktop/Work/lithops oanamoc@pc-10-200 lithops % lithops clean -b aws_lambda -s aws_s3 2023-07-05 16:07:05,064 [INFO] cli.py:83 -- Cleaning all Lithops information 2023-07-05 16:07:05,064 [INFO] config.py:134 -- Lithops v2.9.0 2023-07-05 16:07:05,354 [INFO] aws_s3.py:67 -- S3 client created - Region: eu-west-3 2023-07-05 16:07:06,154 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: eu-west-3 2023-07-05 16:07:06,362 [INFO] aws_lambda.py:320 -- Deleting function: lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB 2023-07-05 16:07:06,721 [INFO] utils.py:83 -- Going to delete all objects from bucket 'lithops-data-yey' and prefix 'lithops.runtimes/aws_lambda' 2023-07-05 16:07:09,433 [INFO] utils.py:93 -- Finished deleting objects, total found: 1 2023-07-05 16:07:09,434 [INFO] utils.py:83 -- Going to delete all objects from bucket 'lithops-data-yey' and prefix 'lithops.jobs' 2023-07-05 16:07:14,046 [INFO] utils.py:93 -- Finished deleting objects, total found: 4 2023-07-05 16:07:14,108 [INFO] cli.py:117 -- All Lithops data cleaned oanamoc@pc-10-200 lithops % lithops test -b aws_lambda -s aws_s3 -d 2023-07-05 16:07:27,197 [INFO] config.py:134 -- Lithops v2.9.0 2023-07-05 16:07:27,198 [DEBUG] config.py:93 -- Loading configuration from /Users/oanamoc/Desktop/Work/lithops/.lithops_config 2023-07-05 16:07:27,215 [DEBUG] config.py:186 -- Loading Serverless backend module: aws_lambda 2023-07-05 16:07:27,286 [DEBUG] config.py:251 -- Loading Storage backend module: aws_s3 2023-07-05 16:07:27,288 [DEBUG] aws_s3.py:36 -- Creating S3 client 2023-07-05 16:07:27,492 [INFO] aws_s3.py:67 -- S3 client created - Region: eu-west-3 2023-07-05 16:07:27,492 [DEBUG] aws_lambda.py:52 -- Creating AWS Lambda client 2023-07-05 16:07:27,492 [DEBUG] aws_lambda.py:65 -- Creating Boto3 AWS Session and Lambda Client 2023-07-05 16:07:28,107 [INFO] aws_lambda.py:94 -- AWS Lambda client created - Region: eu-west-3 2023-07-05 16:07:28,109 [DEBUG] invokers.py:94 -- ExecutorID 0fe5a0-0 - Invoker initialized. Max workers: 1000 2023-07-05 16:07:28,109 [DEBUG] invokers.py:293 -- ExecutorID 0fe5a0-0 - Serverless invoker created 2023-07-05 16:07:28,109 [DEBUG] executors.py:164 -- Function executor for aws_lambda created with ID: 0fe5a0-0 2023-07-05 16:07:28,109 [INFO] invokers.py:108 -- ExecutorID 0fe5a0-0 | JobID A000 - Selected Runtime: lithops-default-runtime-v39 - 256MB 2023-07-05 16:07:28,109 [DEBUG] storage.py:427 -- Runtime metadata not found in local cache. Retrieving it from storage 2023-07-05 16:07:28,109 [DEBUG] storage.py:430 -- Trying to download runtime metadata from: aws_s3://lithops-data-yey/lithops.runtimes/aws_lambda/2.9.0/eu-west-3/lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB.meta.json 2023-07-05 16:07:28,756 [DEBUG] storage.py:449 -- Runtime metadata not found in storage 2023-07-05 16:07:28,756 [INFO] invokers.py:116 -- Runtime lithops-default-runtime-v39 with 256MB is not yet deployed 2023-07-05 16:07:28,756 [INFO] aws_lambda.py:395 -- Deploying runtime: lithops-default-runtime-v39 - Memory: 256 - Timeout: 180 2023-07-05 16:07:28,757 [DEBUG] aws_lambda.py:304 -- Listing lambda layers 2023-07-05 16:07:28,937 [DEBUG] aws_lambda.py:308 -- Listed 0 layers 2023-07-05 16:07:28,938 [INFO] aws_lambda.py:194 -- Creating default lambda layer for runtime lithops-default-runtime-v39 2023-07-05 16:07:28,941 [DEBUG] aws_lambda.py:206 -- Creating "layer builder" function 2023-07-05 16:07:54,329 [DEBUG] aws_lambda.py:162 -- "lithops_v2-9-0_uxen_layer_builder_512MB" function is being deployed... (status: Pending) 2023-07-05 16:07:54,382 [DEBUG] aws_lambda.py:172 -- Ok --> function "lithops_v2-9-0_uxen_layer_builder_512MB" is active 2023-07-05 16:07:54,383 [DEBUG] aws_lambda.py:227 -- OK --> Created "layer builder" function lithops-default-runtime-v39 2023-07-05 16:07:54,383 [DEBUG] aws_lambda.py:237 -- Invoking "layer builder" function 2023-07-05 16:08:26,572 [DEBUG] aws_lambda.py:240 -- OK --> Layer lithops_v2-9-0_uxen_lithops-default-runtime-v39_layer built 2023-07-05 16:08:26,573 [DEBUG] aws_lambda.py:246 -- Trying to delete "layer builder" function 2023-07-05 16:08:26,973 [DEBUG] aws_lambda.py:254 -- Creating layer lithops_v2-9-0_uxen_lithops-default-runtime-v39_layer ... 2023-07-05 16:08:34,233 [DEBUG] aws_lambda.py:271 -- OK --> Layer lithops_v2-9-0_uxen_lithops-default-runtime-v39_layer created 2023-07-05 16:08:34,233 [DEBUG] utils.py:230 -- Creating function handler zip in lithops_lambda.zip 2023-07-05 16:09:03,509 [DEBUG] aws_lambda.py:162 -- "lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB" function is being deployed... (status: Pending) 2023-07-05 16:09:03,575 [DEBUG] aws_lambda.py:172 -- Ok --> function "lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB" is active 2023-07-05 16:09:03,576 [DEBUG] aws_lambda.py:449 -- OK --> Created lambda function lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB 2023-07-05 16:09:03,576 [DEBUG] aws_lambda.py:716 -- Extracting runtime metadata from: lithops-default-runtime-v39 2023-07-05 16:09:05,052 [DEBUG] storage.py:460 -- Uploading runtime metadata to: aws_s3://lithops-data-yey/lithops.runtimes/aws_lambda/2.9.0/eu-west-3/lithops_v2-9-0_uxen__lithops-default-runtime-v39_256MB.meta.json 2023-07-05 16:09:05,385 [DEBUG] aws_s3.py:88 -- PUT Object lithops.runtimes/aws_lambda/2.9.0/eu-west-3/lithops_v2-9-0_uxenlithops-default-runtime-v39_256MB.meta.json - Size: 6.2KiB - OK 2023-07-05 16:09:05,385 [DEBUG] storage.py:466 -- Storing runtime metadata into local cache: /Users/oanamoc/.lithops/cache/lithops.runtimes/aws_lambda/2.9.0/eu-west-3/lithops_v2-9-0_uxen__lithops-default-runtime-v39_256MB.meta.json 2023-07-05 16:09:05,389 [DEBUG] job.py:234 -- ExecutorID 0fe5a0-0 | JobID A000 - Serializing function and data 2023-07-05 16:09:05,392 [DEBUG] module_dependency.py:66 -- Queuing module 'lithops.scripts.cli' 2023-07-05 16:09:05,393 [DEBUG] module_dependency.py:110 -- Module 'lithops' is to be ignored, skipping 2023-07-05 16:09:05,393 [DEBUG] serialize.py:81 -- Referenced modules: /Users/oanamoc/Library/Python/3.9/lib/python/site-packages/lithops/scripts/cli.py 2023-07-05 16:09:05,393 [DEBUG] serialize.py:98 -- Modules to transmit: None 2023-07-05 16:09:05,393 [DEBUG] job.py:268 -- ExecutorID 0fe5a0-0 | JobID A000 - Uploading function and modules to the storage backend 2023-07-05 16:09:05,575 [DEBUG] aws_s3.py:88 -- PUT Object lithops.jobs/0fe5a0-0/037a22730fde1ed5217cd30d1c9730ae.func.pickle - Size: 731.0B - OK 2023-07-05 16:09:05,575 [DEBUG] job.py:308 -- ExecutorID 0fe5a0-0 | JobID A000 - Data per activation is < 8.0KiB. Passing data through invocation payload 2023-07-05 16:09:05,575 [INFO] invokers.py:172 -- ExecutorID 0fe5a0-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations 2023-07-05 16:09:05,576 [DEBUG] invokers.py:177 -- ExecutorID 0fe5a0-0 | JobID A000 - Worker processes: 1 - Chunksize: 1 2023-07-05 16:09:05,576 [DEBUG] invokers.py:302 -- ExecutorID 0fe5a0-0 - Async invoker 0 started 2023-07-05 16:09:05,576 [DEBUG] invokers.py:302 -- ExecutorID 0fe5a0-0 - Async invoker 1 started 2023-07-05 16:09:05,576 [DEBUG] invokers.py:429 -- ExecutorID 0fe5a0-0 | JobID A000 - Free workers: 1000 - Going to run 1 activations in 1 workers 2023-07-05 16:09:05,578 [INFO] invokers.py:208 -- ExecutorID 0fe5a0-0 | JobID A000 - View execution logs at /private/var/folders/th/p9dg33jj77q100w8147qwjkw0000gn/T/lithops-oanamoc/logs/0fe5a0-0-A000.log 2023-07-05 16:09:05,579 [DEBUG] monitor.py:381 -- ExecutorID 0fe5a0-0 - Starting Storage job monitor 2023-07-05 16:09:05,579 [INFO] wait.py:97 -- ExecutorID 0fe5a0-0 - Getting results from 1 function activations 2023-07-05 16:09:05,794 [DEBUG] invokers.py:377 -- ExecutorID 0fe5a0-0 | JobID A000 - Calls 00000 invoked (0.217s) - Activation ID: cde369d1-5324-45a0-8f14-6a973c5fc810 2023-07-05 16:09:07,769 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:09:41,479 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:10:15,452 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:10:49,407 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:11:23,651 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:11:57,741 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:12:31,822 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:13:06,016 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:13:40,066 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:14:14,102 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:14:58,082 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:15:46,386 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:16:22,716 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:17:04,783 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:17:53,416 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:18:37,769 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:19:22,603 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:20:09,721 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:20:48,341 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:21:22,311 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:21:56,264 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:22:40,444 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:23:30,739 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:24:04,805 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:24:38,854 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:25:16,201 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:25:55,396 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:26:35,661 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:27:11,665 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:27:52,801 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:28:39,977 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:29:42,542 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:30:36,066 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:31:16,129 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0 2023-07-05 16:31:57,686 [DEBUG] monitor.py:127 -- ExecutorID 0fe5a0-0 - Pending: 1 - Running: 0 - Done: 0

[JosepSampe]

It seems the storage connection works ok:

2023-07-05 16:09:05,385 [DEBUG] aws_s3.py:88 -- PUT Object lithops.runtimes/aws_lambda/2.9.0/eu-west-3/lithops_v2-9-0_uxen__lithops-default-runtime-v39_256MB.meta.json - Size: 6.2KiB - OK

And the runtime also works ok for extracting the metadata. This is actually a function invocation:

2023-07-05 16:09:03,576 [DEBUG] aws_lambda.py:716 -- Extracting runtime metadata from: lithops-default-runtime-v39

So it is weird that Lithops is able to invoke a function to extract the metadata, but it is not able to complete the execution of the test function.

At this point I would recommend to access the AWS console, and view the lambda invocation logs.

[aitorarjona]

Hello. There is still an issue #1104 from AWS Lambda side, which breaks Lithops for default runtimes. Please use a container runtime in the mean time.

If you go to the console and watch for the logs you will see that indeed the lambda is invoked but fails probably because of this problem.

[JosepSampe]

In my case I'm getting a different error than the one stated in #1104, although I think the cause is the same:

[ERROR] ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' 

According to this, botocore is not yet compatible with urllib3 2.0, so #1122 should solve the issue.

[JosepSampe]

@oanamoc You can try master branch with:

pip install -U git+https://github.com/lithops-cloud/lithops.git@master

[oanamoc]

Thank you, after this installation it worked. 👍