Update our Lambdas that are using the Python runtime < 3.9

[alexwlchan]

We've had an email from AWS about some of our Lambdas on the Python 3.7 runtime:

We are ending support for Python 3.7 in AWS Lambda. This follows Python 3.7 End-Of-Life (EOL) reached on June 27, 2023 [1].

As described in the Lambda runtime support policy [2], end of support for language runtimes in Lambda happens in two stages. Starting November 27, 2023, Lambda will no longer apply security patches and other updates to the Python 3.7 runtime used by Lambda functions, and functions using Python 3.7 will no longer be eligible for technical support. In addition, you will no longer be able to create new Lambda functions using the Python 3.7 runtime. Starting January 25, 2024, you will no longer be able to update existing functions using the Python 3.7 runtime.

We recommend that you upgrade your existing Python 3.7 functions to Python 3.11 before November 27, 2023.

End of support does not impact function execution. Your functions will continue to run. However, they will be running on an unsupported runtime which is no longer maintained or patched by the AWS Lambda team.

[1]: https://peps.python.org/pep-0537/ [2]: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html#runtime-support-policy

I wrote a short script in the AWS account infra repo that looks for matching Lambdas in all of our accounts:

#!/usr/bin/env python3

from config import get_all_aws_accounts

if __name__ == "__main__":
    for account_id, account_name, sess in get_all_aws_accounts():
        print(f"\n# {account_name} ({account_id})\n")

        lambda_client = sess.client("lambda")

        for page in lambda_client.get_paginator("list_functions").paginate():
            for function in page["Functions"]:
                if function.get("Runtime") == "python3.7":
                    print(function["FunctionName"])

Here's the interesting output:

# digitisation (404315009621)

AV-Preingest-Copy
AV-Preingest-Convert

# platform (760097843905)

buildkite-elasticstack-na-AzRebalancingSuspenderFu-Ggu8z7mMGpFh
buildkite-elasticstack-sc-AzRebalancingSuspenderFu-LAWvOJKJF1Ij
calm_window_generator
buildkite-elasticstack-AzRebalancingSuspenderFunct-o116nKtDOJPv
deletion_check_initiator

# catalogue (756629837203)

snapshot_scheduler-prod
snapshot_recorder-prod
snapshot_reporter-prod

All of these Lambdas should be upgraded to newer versions of Python.

[agnesgaroux]

As of 2023-10-25

# digitisation (404315009621)

AV-Preingest-Copy
AV-Preingest-Convert

# platform (760097843905)

buildkite-elasticstack-na-AzRebalancingSuspenderFu-Ggu8z7mMGpFh
buildkite-elasticstack-sc-AzRebalancingSuspenderFu-LAWvOJKJF1Ij
calm_window_generator // upgraded via tf
buildkite-elasticstack-AzRebalancingSuspenderFunct-o116nKtDOJPv
deletion_check_initiator // upgraded via tf

[agnesgaroux]

As of 2023-10-30

# digitisation (404315009621)

AV-Preingest-Copy
AV-Preingest-Convert

# platform (760097843905)

buildkite-elasticstack-na-AzRebalancingSuspenderFu-Ggu8z7mMGpFh
buildkite-elasticstack-sc-AzRebalancingSuspenderFu-LAWvOJKJF1Ij
buildkite-elasticstack-AzRebalancingSuspenderFunct-o116nKtDOJPv

[agnesgaroux]

As of 2023-10-30

# digitisation (404315009621)

AV-Preingest-Copy
AV-Preingest-Convert

[agnesgaroux]

AV-Preingest-Copy AV-Preingest-Convert Upgraded directly in the AWS console as the lambdas were created there initially https://wellcome.slack.com/archives/C3TQSF63C/p1698668957186089

[agnesgaroux]

Well, actually...

[agnesgaroux]

Checks lambdas with python < 3.9

from config import get_all_aws_accounts

if __name__ == "__main__":
    for account_id, account_name, sess in get_all_aws_accounts():
        print(f"\n# {account_name} ({account_id})\n")

        lambda_client = sess.client("lambda")

        for page in lambda_client.get_paginator("list_functions").paginate():
            for function in page["Functions"]:
                runtime = function.get("Runtime")
                if runtime != None and runtime.startswith('python') and int(runtime.split(".")[1]) < 9:
                  print(function["FunctionName"])
                  print(runtime)

[agnesgaroux]

• python3.6

# platform (760097843905)

sierra_orders_window_generator // updated to 3.9 via aws console 2023-11-07 
sierra_holdings_window_generator // updated to 3.9 via aws console 2023-11-07
sierra_bibs_window_generator // updated to 3.9 via aws console 2023-11-07 
sierra_items_window_generator // updated to 3.9 via aws console 2023-11-07 
// NOTE: all the above were detected by terraform when a subsequent 'apply' was run. 
reporting_miro_transformer

reporting_miro_inventory_transformer // looks inactive, no invocation in last 6 months
lambda-v2_lambda_public // looks inactive, no invocation in last 6 months
lambda-v2_lambda_vpc // looks inactive, flagged as "idle" in console
miro_migration // looks inactive, no invocation in last 6 months

• python3.8

# workflow (299497370133)

archivematica-s3_start_transfer-staging
archivematica-start_test_transfer-staging
archivematica-transfer_monitor-prod
archivematica-start_test_transfer-prod
archivematica-transfer_monitor-staging
archivematica-s3_start_transfer-prod

# platform (760097843905)

update_embargoed_holdings

# storage (975596993436)

end_to_end_bag_test
daily_reporter
end_to_end_bag_test--staging
digital_production_report

[agnesgaroux]

2023-11-13 remaining:

# platform (760097843905)

reporting_miro_transformer -> python3.6
update_embargoed_holdings -> python3.8

# storage (975596993436)

end_to_end_bag_test -> python3.8
daily_reporter -> python3.8
end_to_end_bag_test--staging -> python3.8
digital_production_report -> python3.8

[agnesgaroux]

# platform (760097843905)

reporting_miro_transformer -> python3.6
update_embargoed_holdings -> python3.9 as of 2023-11-13

# storage (975596993436)

end_to_end_bag_test -> python3.8
daily_reporter -> python3.8
end_to_end_bag_test--staging -> python3.8
digital_production_report -> python3.8

[agnesgaroux]

All lambdas are at least on python3.9 except

reporting_miro_transformer on Python3.9 as of 2023-11-30, done though the AWS console as lambda is not terraformed

reporting_miro_inventory_transformer // looks inactive, no invocation in last 6 months
lambda-v2_lambda_public // looks inactive, no invocation in last 6 months
lambda-v2_lambda_vpc // looks inactive, flagged as "idle" in console
miro_migration // looks inactive, no invocation in last 6 months