RFC: Add "momento" as Idempotency backend

[walmsles]

Is this related to an existing feature request or issue?

2567

Which Powertools for AWS Lambda (Python) utility does this relate to?

Idempotency

Summary

Add Serverless Momento Cache as an Idempotency backend.

I see the DX identical to the current dynamoDB DevEx.

Use case

Momento Serverless cache is a strong offering gaining momentum in the industry, and I was curious about creating a custom persistence layer so have created a POC around Momento and feel it is worth contributing as an optional extra.

Proposal

from dataclasses import dataclass, field
from uuid import uuid4

from aws_lambda_powertools.utilities.idempotency import (
    MomentoPersistenceLayer,
    idempotent,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

persistence_layer = MomentoPersistenceLayer(cache_name="my-cache", auth_token="my-super-secret-token")

@dataclass
class Payment:
    user_id: str
    product_id: str
    payment_id: str = field(default_factory=lambda: f"{uuid4()}")

class PaymentError(Exception):
    ...

@idempotent(persistence_store=persistence_layer)
def lambda_handler(event: dict, context: LambdaContext):
    try:
        payment: Payment = create_subscription_payment(event)
        return {
            "payment_id": payment.payment_id,
            "message": "success",
            "statusCode": 200,
        }
    except Exception as exc:
        raise PaymentError(f"Error creating payment {str(exc)}")

def create_subscription_payment(event: dict) -> Payment:
    return Payment(**event)

Out of scope

None.

Potential challenges

With Persistence layers without conditional store functionality care needs to be taken to ensure idempotency race conditions are not introduced by reading/writing records, which will cause idempotency to fail at scale.

Dependencies and Integrations

momento cache SDK and requires momento cache auth-key for testing.

Alternative solutions

DynamoDB or REDIS is an alternative.  Momento is a nice zero infrastruture solution.

An alternative solution is to see hold for the redis implementation to be completed and see whether the momento-redis library would be suitable.  At the moment the momento-redis supports simple operations of set/get.  I feel a true native SDK implementation would be preferable.

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (Python) Tenets
• [X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[walmsles]

I have a POC of this Persistence layer - needs some refinement and solid testing. Will also take into account feedback on the existing redis PR that is in-flight.

[heitorlessa]

Hey, thanks Michael :) I'd love to hear customer demand before we figure out how to support it (e.g., AWS Partner, End-to-end [$$] OR contract testing, etc.).

For future readers, you can create your own persistent storage in the meantime: https://docs.powertools.aws.dev/lambda/python/latest/utilities/idempotency/#bring-your-own-persistent-store

Linking with Redis #2567

[leandrodamascena]

Hi @walmsles, we have great news! We've added support for Redis as a persistence layer and you can use it with any client that uses the RESP protocol. AFAIK momento-python-redis-client is a wrapper of the redis-py library and works as expected with this new persistent layer.

But we have a problem here. 3 weeks ago I opened a issue on the momento-python-redis-client GitHub informing them that we were unable to install aws-lambda-powertools==2.32.0 with their library due to dependency resolution issues. You will be able to use this new persistent layer as soon as they solve it.

I made a hack to demonstrate this working: I installed the momento_redis library and manually copied the Powertools package for installation.

CODE

from dataclasses import dataclass, field
from uuid import uuid4
import datetime

# Import the Momento redis compatibility client.
import momento
from momento_redis import MomentoRedis

from aws_lambda_powertools.utilities.idempotency import (
    idempotent,
)
from aws_lambda_powertools.utilities.idempotency.persistence.redis import (
    RedisCachePersistenceLayer,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

_CACHE_NAME = "leo"
redis_client = MomentoRedis(
    momento.CacheClient(
        momento.Configurations.Laptop.latest(),
        momento.CredentialProvider.from_environment_variable("MOMENTO_AUTH_TOKEN"),
        datetime.timedelta(seconds=60)
    ),
    _CACHE_NAME
)

persistence_layer = RedisCachePersistenceLayer(client=redis_client)

@idempotent(persistence_store=persistence_layer)
def lambda_handler(event: dict, context: LambdaContext):
    return event

Execution

❯ sam local invoke --skip-pull-image --event events/event.json
Invoking app.lambda_handler (python3.10)                                                                                                                                                                                                                                                                                 
Requested to skip pulling images ...                                                                                                                                                                                                                                                                                     

Mounting /tmp/aa/momento/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                                                                                                                                                   
START RequestId: fc25915d-e424-4469-995a-9af181197692 Version: $LATEST
START RequestId: fc25915d-e424-4469-995a-9af181197692 Version: $LATEST
END RequestId: fc25915d-e424-4469-995a-9af181197692
REPORT RequestId: fc25915d-e424-4469-995a-9af181197692  Init Duration: 0.10 ms  Duration: 3610.30 ms    Billed Duration: 3611 ms    Memory Size: 128 MB Max Memory Used: 128 MB 
{"body": "{\"message\": \"hello world\"}", "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "Accept-Encoding": "gzip, deflate, sdch", "Accept-Language": "en-US,en;q=0.8", "Cache-Control": "max-age=0", "CloudFront-Forwarded-Proto": "https", "CloudFront-Is-Desktop-Viewer": "true", "CloudFront-Is-Mobile-Viewer": "false", "CloudFront-Is-SmartTV-Viewer": "false", "CloudFront-Is-Tablet-Viewer": "false", "CloudFront-Viewer-Country": "US", "Host": "1234567890.execute-api.us-east-1.amazonaws.com", "Upgrade-Insecure-Requests": "1", "User-Agent": "Custom User Agent String", "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", "X-Forwarded-For": "127.0.0.1, 127.0.0.2", "X-Forwarded-Port": "443", "X-Forwarded-Proto": "https"}, "httpMethod": "GET", "isBase64Encoded": false, "path": "/hello", "pathParameters": {"proxy": "/path/to/resource"}, "queryStringParameters": {"foo": "bar"}, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "httpMethod": "POST", "identity": {"accessKey": null, "accountId": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityId": null, "cognitoIdentityPoolId": null, "sourceIp": "127.0.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/prod/hello", "protocol": "HTTP/1.1", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "requestTime": "09/Apr/2015:12:34:56 +0000", "requestTimeEpoch": 1428582896000, "resourceId": "123456", "resourcePath": "/hello", "stage": "prod"}, "resource": "/hello", "stageVariables": {"baz": "qux"}}

Record

Please let me know if you were able to use it and if I can close this ticket :)

[leandrodamascena]

Closing this issue as completed! The Redis backend now fully supports Momento as a Redis backend.

Feel free to reopen if needed.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.