Add Python3 UDF to orchestrate the SageMaker Autopilot training from Exasol

[tkilias]

Background

• We want to start the training of SageMaker Autopilot from Exasol
• For that, we can use a Python3 UDF which can either use boto3 or the Sagemaker SDK
  • boto3 is quite primitive but per default in the script-language-containers installed
  • the SageMaker SDK is easier and more capable than boto3, but you would need to add it to the script-language container, see https://github.com/exasol/script-languages-release/issues/336
• The following notebook shows at the end what you need to call from the SageMaker Python SDK
  • https://github.com/exasol/data-science-examples/blob/e85ce663a0474c60bd8d8700c9ae60c05c38f1e3/tutorials/machine-learning/python/sagemaker/sagemaker_autopilot.ipynb

Tips

• To get project specific container follow the example in the data-science-utils-python repo
  • build script: https://github.com/exasol/data-science-utils-python/blob/main/build_language_container.sh
  • container-context (needed to build the container): https://github.com/exasol/data-science-utils-python/tree/main/language_container
  • container-flavor: https://github.com/exasol/data-science-utils-python/tree/main/language_container/exasol-data-science-utils-python-container
  • build language container from pytest: https://github.com/exasol/data-science-utils-python/blob/main/tests/integration_tests/fixtures/build_language_container_fixture.py
  • here a tutorial for the script language containers https://github.com/exasol/data-science-examples/blob/main/tutorials/script-languages/script-languages.ipynb
• The following notebooks shows you how you can tell the UDFs to use the instance level credentials. see comment https://github.com/exasol/sagemaker-extension/issues/2#issuecomment-914043957
• The Exasol DB needs the permissions to access SageMaker and S3
• Use pytest for testing
• Separate UDF specific parts of the code from core logic for better testability
• Use https://github.com/exasol/sagemaker-extension/issues/1 for testing
• Use udf-mock-python for testing

Acceptance Criteria

• Python3 UDF which can start a SageMaker training and waits to its completion and returns the training id

[tkilias]

If you want to use the EC2 Instance credentials in UDFs you need to add the following to the UDF

    f = open("/tmp/.config", "w")
    f.write(
        "[default]\\nregion = {region!r}\\nrole_arn = {role!r}\\ncredential_source = Ec2InstanceMetadata")
    f.close()
    os.environ['AWS_CONFIG_FILE'] = '/tmp/.config'

see https://github.com/exasol/data-science-examples/blob/e85ce663a0474c60bd8d8700c9ae60c05c38f1e3/tutorials/machine-learning/python/sagemaker/UseSagemakerModelFromExasol.ipynb

[tkilias]

Overview diagram

[tkilias]

Drawio file sagemaker-extension-overview.drawio