search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud β˜οΈπŸš€
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

AWS Lambda Function Name Fix #1314

Closed linuxkd closed 6 months ago

linuxkd commented 6 months ago

AWS performs regex matching on many of their services for different fields, including Lambda function names. The @ symbol is not allowed in a function name, so if a user leverages AWS SSO and their org is configured to send the email address as part of their user id then lithops will fail when creating functions.

I will also submit a PR that fixes this issue, I have already tested it on my end and confirmed it works as expected.

Steps to reproduce

  1. Have an SSO provider configured with email address as part of the user id
  2. Configure lithops for use with AWS including credentials, role ARN, etc
  3. Execute a command that would build a lambda function such as lithops hello

Runtime error

ShadowClone on ξ‚  main [!?] via 🐍 v3.11.9 (env) on ☁️  aws-redacted (us-east-1)
❯ lithops hello -b aws_lambda -s aws_s3
2024-04-17 16:25:56,314 [INFO] config.py:139 -- Lithops v3.2.0 - Python3.11
2024-04-17 16:25:56,430 [INFO] aws_s3.py:68 -- S3 client created - Region: us-east-2
2024-04-17 16:25:56,912 [INFO] aws_lambda.py:109 -- AWS Lambda client created - Region: us-east-2
2024-04-17 16:25:56,913 [INFO] invokers.py:107 -- ExecutorID 2fe613-0 | JobID A000 - Selected Runtime: default-runtime-v311 - 1024MB
2024-04-17 16:25:56,984 [INFO] invokers.py:115 -- Runtime default-runtime-v311 with 1024MB is not yet deployed
2024-04-17 16:25:56,985 [INFO] aws_lambda.py:400 -- Deploying runtime: default-runtime-v311 - Memory: 1024 - Timeout: 900
2024-04-17 16:25:57,267 [INFO] aws_lambda.py:199 -- Creating lambda layer for runtime default-runtime-v311
Traceback (most recent call last):
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 214, in _create_layer
    resp = self.lambda_client.create_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateFunction operation: 1 validation error detected: Value 'lithops_v320_redacted@redacted.com_layer_builder_512MB' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-_]+)(:(\$LATEST|[a-zA-Z0-9-_]+))?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/redacted/code/ShadowClone/env/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
             ^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/scripts/cli.py", line 206, in hello
    fexec.call_async(hello, username)
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/executors.py", line 193, in call_async
    runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/invokers.py", line 116, in select_runtime
    runtime_meta = self.compute_handler.deploy_runtime(self.runtime_name, runtime_memory, runtime_timeout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/serverless/serverless.py", line 84, in deploy_runtime
    return self.backend.deploy_runtime(runtime_name, memory, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 539, in deploy_runtime
    self._deploy_default_runtime(runtime_name, memory, timeout)
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 405, in _deploy_default_runtime
    layer_arn = self._create_layer(runtime_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 253, in _create_layer
    self.lambda_client.delete_function(FunctionName=func_name)
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/redacted/code/ShadowClone/env/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the DeleteFunction operation: 1 validation error detected: Value 'lithops_v320_redacted@secureworks.com_layer_builder_512MB' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-_]+)(:(\$LATEST|[a-zA-Z0-9-_]+))?

Cause

Note the UserId is returned as an SSO user with an email address


❯ aws sts get-caller-identity --profile aws-redacted
{
    "UserId": "redacted:redacted@redacted.com",
    "Account": "redacted",
    "Arn": "arn:aws:sts::redacted:assumed-role/AWSReservedSSO_redacted/redacted@redacted.com"
}
linuxkd commented 6 months ago

Merge of PR #1315 resolved the issue.