Include appropriate `complete_platforms` for serverless/FaaS environments `runtime`

[huonw]

Is your feature request related to a problem? Please describe.

Currently building Serverless artefacts like AWS Lambda or GCF supports both a quick-and-easy runtime setting, and a complete_platforms. Generally setting complete_platforms is required to get the right wheels, and setting runtime is not useful/misleading/causes problems (e.g. #15296, #18001, and a continual stream of questions in Slack).

Describe the solution you'd like

I think the environments for a given runtime are generally pretty stable, so pants itself could potentially include an appropriate complete platform JSON for each runtime setting (so 3 for AWS Lambda, and 4 for GCF). That is, something like python_awslambda(..., runtime="python39") would be translated into calling pex with a pants-provided complete platform JSON, rather than just a simple platform identifier.

Having pants provide a complete platform almost certainly won't be worse than not using a complete platform at all, and would hopefully mean fewer users have problems. Any users who still have problems can provide their own complete platform.

Downside: if the complete platform pants provides is changed (e.g. pants 2.35 has a new JSON than pants 2.34), built artefacts may change after a user upgrades to 2.35, due to differing wheel selection.

Describe alternatives you've considered

Just not doing this, because the downsides/risks are too large?

Additional context

The complete platforms can be generated by running PEX in each of the relevant environments. For instance, we're deploying AWS Lambdas using a complete platform generated by:

import subprocess

def lambda_handler(event, context):
    subprocess.run(
        """
        pip install --target=/tmp/subdir pex
        PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
        """,
        shell=True
    )
    return {
        'statusCode': 200,
        'body': "{}",
    }

I think this would "just" mean switching the following bit of code to select an appropriate pants-provided JSON file, rather than just constructing the platform string: https://github.com/pantsbuild/pants/blob/4b2bc114aafe05a1abdb81ab92d31cd275a7012d/src/python/pants/backend/awslambda/python/rules.py#L101-L114

[huonw]

This was done for AWS Lambda in #19253, but not GCF.

[huonw]

Discussed in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686082038188639 (aka https://chat.pantsbuild.org/t/12319378/both-aws-lambda-and-google-cloud-functions-allows-using-a-zi#b8423452-b0f9-44c1-a6e0-f5c18eb32c27 https://chat.pantsbuild.org/t/12320824/does-anyone-happen-to-have-knowledge-of-gcf-for-the-question#9e9eb90b-e6b3-4b0b-ae88-d0090a65fcec )

[huonw]

This was done for AWS by running an appropriate pex3 ... command (see #19253) in the docker images AWS provides. It seems like GCF doesn't provide such images, so it wasn't easy to do that too.

The current thinking is run the pex3 ... command in the real environment, e.g. deploy a function that calls subprocess.run. I'm personally unsure how to do this yet (never used GCP yet!), so haven't invested the time.

[ryaminal]

decided to run this on the 4 python versions that are available in GCF v2(based on top of cloud run). not certain if it's different for v1, but i can get those also. gcf_v2_38 gcf_v2_39 gcf_v2_310 gcf_V2_311

no idea if all 4 were needed but... yeah

[huonw]

That's great @ryaminal! More is better! That's great. I'm on leave for a few days so I can't offer much assistance just now, but if you were make a PR adding them to the repo, that'd be cool.

For v1 vs v2, I have no idea. Do you have a sense for how often people use v1?