search

aws-powertools / powertools-lambda-python

A developer toolkit to implement Serverless best practices and increase developer velocity.
https://docs.powertools.aws.dev/lambda/python/latest/
MIT No Attribution
2.72k stars 378 forks source link

RFC: Powertools for AWS Lambda (Python) v3 #4189

Open leandrodamascena opened 2 months ago

leandrodamascena commented 2 months ago

Is this related to an existing feature request or issue?

https://github.com/aws-powertools/powertools-lambda-python/issues/4067

Which Powertools for AWS Lambda (Python) utility does this relate to?

Other

Summary

We are excited to announce that we are starting a new major version (v3) for Powertools for AWS Lambda (Python). We are expediting a new major version to align with the scheduled Pydantic v1 end-of-life on June 30th, signalling the switch to much anticipated Pydantic v2. Pydantic plays a vital role and we care deeply about supply chain security. This release highlights our dedication to providing a secure and stable library that our customers trust.

Today, we already support both Pydantic v1 and v2. In this new major version, we will make Pydantic v2 the default, removing Pydantic v1. We will take this opportunity to introduce minor breaking changes to address common pain points. As always, we will provide an extensive upgrade guide to make this transition as smooth as possible.

Community

We invite the entire Powertools community, including users and contributors, to actively participate in this RFC. We greatly value the community's contribution, ideas, and willingness to solve problems or validate proposed changes.

Use case

The primary use case is to offer Pydantic v2. That includes a new Lambda Layer for each Python version since Pydantic v2 contains non-Python code (Rust). It future-proof Layers to contain any additional compiled dependency (like cryptography) without a new major version.

Additional key items on v3 backlog

Proposal

The action plan for this release with scope and tasks is as follows:

Quick summary (Work in progress - Items can be added or removed):


Item Issue/PR Status Code change required Utility Notes Cross-Language
refactor the SSMProvider class to implement the get_multiple method #3252 🚧 Yes Parameters
Return empty Dict or List in Event Source Data Classes instead of None #2605 🚧 Yes Event Source Data Class
event_parser envelopes should handle unions of BaseModels #2734 ✅ No Parser
Pydantic v2 managed layer #2939 🚧 No Layer
Deprecate tracing.base.BaseProvider #4105 🚧 No Tracer
resolved_headers_field is not Optional #4147 ✅ No Event Source Data Class
Create a Powertools Lambda layer for each Python version #3859 🚧 No Layer
Publish layer to Gov Cloud regions #1166 🚧 No Layer ✅
Remove any specific code for Pydantic v1 #4191 ✅ Yes Codebase
Update Powertools documentation to add banner about v3 #4198 ✅ No Documentation
Add the aws-encryption-sdk dependency in the Powertools layer #4192 🚧 No Layer Depends on #3859
Review the method extract_data_from_envelope in JMESPath Functions #4218 🚧 No JMESPath Functions
Split the code from the api_gateway.py file into several other files to ease maintenance #4194 🚧 Yes Event Handler
Add support for v3 branch in the github actions #4220 ✅ No CI
Release the new version of Tracer utility with support for external providers #2342 🚧 Yes Tracer
Unmarshal DynamoDB fields when using DynamoDBStreamModel #4219 ✅ Yes Parser Reference: https://github.com/aws-powertools/powertools-lambda-python/pull/1619 ✅
Remove legacy code in the Metrics utility? - 🚧 Yes Metrics Needs discussion
Keep the compatibility with Pydantic v2 and Bedrock agent resolver #4196 🚧 No Event Handler
Change the default TTL (cache) to 5 minutes #4193 ✅ Yes Parameters ✅
Update boto3 version to 1.34.32 #4195 ✅ No Dependencies
Change behavior to fail batch on first error when using DynamoDB and Kinesis in batch processing NEED ISSUE 🚧 Yes Batch Processing Reference: https://tinyurl.com/yc335m3s ✅
Remove the flag log_uncaught_exceptions in the Logger utility #4199 🚧 Yes Logger
Support for retrieving batch of secrets #4200 🚧 No Parameters

Deadlines

We plan to release the first version of v3 by the end of June/beginning of July.

Out of scope

We will not include in v3:

Potential challenges

Adding Pydantic v2 in the Powertools layer requires layers per Python version, so the effort to implement this is not yet clear.

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

jplock commented 2 months ago

There's a lot of overlap between the Event Handler, Event Source Data Classes, Parser, and Validation

Is there a nice way to unify it all somehow?

ran-isenberg commented 2 months ago

There's a lot of overlap between the Event Handler, Event Source Data Classes, Parser, and Validation

Is there a nice way to unify it all somehow?

I agree. Since v3 will use pydantic by default, is there a need to maintain event source data classes? why not move to the parser's pydantic schemas exclusively? I suppose there are some gaps in coverage and missing schemas, but that can be fixed :) Event handler/validation already support pydantic/parser schemas.

michaelbrewer commented 2 months ago

@jplock @ran-isenberg

i contributed REST API event handler , GraphQL API eventhandler and Event Source Data Classes as a convenient what to handle AWS built-in event source. The assumption that it would not need any libraries, dependencies or validation as these event come from AWS.

If what can be done here is not a major breaking change or a regression in performance it would be greatly appreciated, as we use Powertools for many internal tools at Fiserv.

ran-isenberg commented 2 months ago

Hey Michael 🙏🏻 I agree, it's a massive breaking change. I'm not sure it's worth it tbh but im biased as i use the parser (and contributed it in the first place). I'll let the good people here decide 🤘

leandrodamascena commented 2 months ago

Hey everyone! Thanks for the contribution and for bringing ideas/suggestions to this RFC.

We have no plans to remove features or do extreme refactoring to Powertools V3 utilities/codebase. This release mainly aims to drop Pydantic v1 in favor of v2 and resolve some pain points. When it comes to event source, we will be refactoring and introducing a small breaking change, but still using the same concept of creating objects from events and without any additional dependencies. Removing utilities such as Event Source or making Pydantic mandatory for the Event Handler, for example, could be a huge breaking change and would make it extremely hard for our customers to migrate from v2 to v3.

I agree that we have some overlap between some utilities and could consider merging/removing them, but we can evaluate this in future releases, not in v3.

Thanks

trallnag commented 1 day ago

Are you planning to remove the use of parse_obj method? It has been deprecated in v2 and will be completely removed in v3. I am currently using Powertools with Pydantic v2 and I am seeing this warning in my tests:

PydanticDeprecatedSince20: The parse_obj method is deprecated; use model_validate instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/