WARNING: All interfaces are subject to change.
We are in the early stages of beginning work on low-level Python SDK modules that aim to provide basic, reusable, and composable interfaces for lower level SDK tasks. Using these modules customers should be able to generate asynchronous service client implementations based on services defined using Smithy.
This code generator, and the clients it generates, are unstable and should not be used in production systems yet. Several features, such as detailed logging, have not been implemented yet.
This repository contains two major components:
1) Smithy code generators for Python 2) Core modules and interfaces for building service clients in Python
These components facilitate generating clients for any Smithy
service. The codegen directory contains the source code for generating clients.
The python-packages directory contains the source code for the handwritten python
components.
This repository does not contain any generated clients, such as for S3 or other AWS services. Rather, these are the tools that facilitate the generation of those clients and non-AWS Smithy clients.
The first step is to create a Smithy pacakge. If this is your first time working with Smithy, follow this quickstart guide to learn the basics and create a simple Smithy model.
Once you have a service defined in Smithy, you will need to define what protocol it uses. Currently, the only supported protocol is restJson1. This is a protocol based on AWS services, but is broadly applicable to any service that uses rest bindings with a JSON body type. Simply add the protocol trait to your service shape, and you'll be ready.
The following is a basic example service model that echoes messages sent to it.
To use this model to generate a client, save it to a file called main.smithy
in a folder called model.
$version: "2.0"
namespace com.example
use aws.protocols#restJson1
/// Echoes input
@restJson1
service EchoService {
version: "2006-03-01"
operations: [EchoMessage]
}
@http(uri: "/echo", method: "POST")
operation EchoMessage {
input := {
@httpHeader("x-echo-message")
message: String
}
output := {
message: String
}
}You also will need a build configuration file named smithy-build.json, which
for this example service should look the following json. For more information on
this file, see the
smithy-build docs.
{
"version": "1.0",
"sources": ["model"],
"maven": {
"dependencies": [
"software.amazon.smithy:smithy-model:[1.34.0,2.0)",
"software.amazon.smithy:smithy-aws-traits:[1.34.0,2.0)",
"software.amazon.smithy.python:smithy-python-codegen:0.1.0"
]
},
"projections": {
"client": {
"plugins": {
"python-client-codegen": {
"service": "com.example#EchoService",
"module": "echo",
"moduleVersion": "0.0.1"
}
}
}
}
}The code generator, smithy-python-codegen, hasn't been published yet, so
you'll need to build it yourself. To build and run the generator you will need
the following prerequisites:
Now run make install-components from the root of this repository. This will
install the python dependencies in your environment and make the code generator
available locally. For more information on the underlying build process, see the
"Using repository tooling" section.
Now from your model directory run smithy build and you'll have a generated
client! The client can be found in build/smithy/client/python-client-codegen.
The following is a snippet showing how you might use it:
import asyncio
from echo.client import EchoService
from echo.config import Config
from echo.models import EchoMessageInput
async def main() -> None:
client = EchoService(Config(endpoint_uri="https://example.com/"))
response = await client.echo_message(EchoMessageInput(message="spam"))
print(response.message)
if __name__ == "__main__":
asyncio.run(main())Only for now. Once the generator has been published, the Smithy CLI will be able to run it without a separate Java installation. Similarly, once the python helper libraries have been published you won't need to install them manually.
smithy-core provides transport-agnostic core modules and interfaces
required to build a service client. This includes things like retry
strategies, URI interfaces, shared types, etc.smithy-http provides HTTP core modules and interfaces required to build
HTTP service clients, including optional HTTP client implementations.
Currently it provides two async HTTP clients that are useable with the
aiohttp or awscrt optional dependency sets respectively.smithy-aws-core provides implementations of smithy-core interfaces for
AWS, such as SigV4 signers.Components must be modular - Most importantly, these building blocks need to be composable and reusable across a wide variety of use cases, including use cases beyond an AWS SDK. Interfaces such as credential resolvers, request signing, data models, serialization, etc. should all be reusable across many contexts.
Components should be well documented and publicly exported - Both AWS and customers should have a high level of confidence that the building blocks we're creating are well-supported, understood, and maintained. Customers should not have to hack on internal or undocumented interfaces to achieve their goals.
Components must be typed - All the buildings blocks we create must be
typed and usable via mypy. Given the nature of gradual typing, it is paramount
that foundational components and interfaces be typed to preserve the integrity
of the typing system.
Components should be consistent with other AWS SDKs - When building interfaces or libraries that overlap with the required functionality of other AWS SDKs we should strive to be consistent with other SDKs as our default stance. This project will heavily draw inspiration from the precedents set by the smithy-typescript and smithy-go packages.
We're currently heavily investing in writing proposals and documenting the design decisions made. Feedback on the proposed designs and interfaces is extremely helpful at this stage to ensure we're providing functional and ergonomic interfaces that meet customer expectations.
This repository is intended to contain the source for multiple Python and Java packages, so the process of development may be a bit different from what you're familiar with.
The Java-based code generation uses Gradle, which is a fairly common Java build tool that natively supports building, testing, and publishing multiple packages in one place. If you've used Gradle before, then there's nothing in this repo that will surprise you.
If you haven't used Gradle before, don't worry - it's pretty easy to use. You will need to have JDK 17 or newer installed, but that's the only thing you need to install yourself. We recommend the Corretto distribution, but any JDK that's at least version 17 will work.
To build and run all the Java packages, simply run ./gradlew clean build from
the codegen directory. If this is the first time you have run this, it will
download Gradle onto your system to run along with any dependencies of the Java
packages.
For more details on working on the code generator, see the readme in the
codegen directory.
Building multiple python packages in a single repo is a little less common than it is for Java or some other languages, so even if you're a python expert you may be unfamiliar with the tooling. The tool we're using is called pants, and you use it pretty similarly to how you use Gradle.
Like Gradle, pants provides a wrapper script that downloads its dependencies as
needed. Currently, pants requires python 3.7, 3.8, or 3.9 to run, so one of
those must be available on your path. (It doesn't have to be the version that
is linked to python or python3, it just needs python3.9 etc.) It is,
however, fully capable of building and working with code that uses newer python
versions like we do. This repository uses a minimum python version of 3.12
for all its packages, so you will need that too to work on it.
Pants provides a number of python commands it calls goals, documented here. In short:
./pants fmt :: - This will run our formatters on all the python library
code. Use fix instead of this, since it runs all the formatters AND fixers../pants fix :: - This will run the formatters/fixers on python library
code, and apply the changes. Use this before making commits../pants lint :: - This will run our formatters/fixers/linters on all the
python library code. You should also use this before you make commits, and particularly
before you make a pull request. It will not apply formatting or fixes for you../pants check :: - This will run mypy on all the python library code.
This should be used regularly, and must pass for any pull request../pants test :: - This will run all the tests written for the python
library code. Use this as often as you'd run pytest or any other testing
tool. Under the hood, we are using pytest../pants update-build-files :: - This will auto-format all the BUILD files. Use this if
you are making changes to or adding new BUILD files.There are other commands as well that you can find in the docs, but these are the ones you'll use the most.
Important to note is those pairs of colons. These are pants
targets. The double
colon is a special target that means "everything". So running exactly what's
listed above will run those goals on every python file or other relevant file.
You can also target just smithy_core, for example, with
./pants check python-packages/smithy-python/smithy_core:source, or even
individual files with something like
./pants check python-packages/smithy-python/smithy_core/interfaces/identity.py:../source.
To list what targets are available in a directory, run
./pants list path/to/dir:. For more detailed information, see the
docs.
There is also a Makefile that bridges the Python and Java build systems together to
make common workflows simple, single commands. The two most important commands are:
make install-components which builds and installs the Java generator and the python
packages. The generator is published to maven local and the python packages are
installed into the active python environment. This command is most useful for those
who simply want to run the generator and use a generated client.vmake test-protocols which runs all the protocol tests. It will first (re)install
all necessary components to ensure that the latest is being used. This is most useful
for developers working on the generator and python packages.To see what else available, run make help or examine the file directly.
If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public GitHub issue.
This project is licensed under the Apache-2.0 License.