search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.63k stars 3.91k forks source link

(aws-glue-alpha): When using the GlueVersion.of method with cdk in Python, the version check process does not work. #26616

Closed dokeita closed 1 year ago

dokeita commented 1 year ago

Describe the bug

When using the GlueVersion.of method with cdk in Python, the version check process does not work.

Expected Behavior

The version check process work normarlly.

ex. GlueVersion 4.0 is specified for pythonshell jobs.

$ cdk ls
jsii.errors.JavaScriptError: 
  @jsii/kernel.RuntimeError: Error: Specified GlueVersion 4.0 does not support Python Shell
      at Kernel._Kernel_ensureSync (/tmp/tmpbdjwndyr/lib/program.js:10385:23)
      at Kernel.sinvoke (/tmp/tmpbdjwndyr/lib/program.js:9770:102)
      at KernelHost.processRequest (/tmp/tmpbdjwndyr/lib/program.js:11602:36)
      at KernelHost.run (/tmp/tmpbdjwndyr/lib/program.js:11562:22)
      at Immediate._onImmediate (/tmp/tmpbdjwndyr/lib/program.js:11563:46)
      at process.processImmediate (node:internal/timers:476:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 9, in <module>
    GlueAlphaPythonStack(app, "glue-alpha-python")
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 118, in __call__
    inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
  File "/abc/glue_alpha_python/glue_alpha_python/glue_alpha_python_stack.py", line 16, in __init__
    executable=glue_alpha.JobExecutable.python_shell(
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/aws_cdk/aws_glue_alpha/__init__.py", line 4280, in python_shell
    return typing.cast("JobExecutable", jsii.sinvoke(cls, "pythonShell", [props]))
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 418, in sinvoke
    response = self.provider.sinvoke(
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 383, in sinvoke
    return self._process.send(request, InvokeResponse)
  File "/abc/glue_alpha_python/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 342, in send
    raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Error: Specified GlueVersion 4.0 does not support Python Shell

Current Behavior

It passes the version check process.

$ cdk ls
glue-alpha-python

$ cdk deploy glue-alpha-python

✨  Synthesis time: 5.11s

...

glue-alpha-python: deploying... [1/1]
glue-alpha-python: creating CloudFormation /abc...
12:49:49 PM | CREATE_FAILED        | AWS::Glue::Job     | samplejobA34E9B24D
Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request I
D: 288658c4-a8bc-4194-bd5c-a1db50c45d2d; Proxy: null)

 ❌  glue-alpha-python failed: Error: The stack named glue-alpha-python failed to deploy: UPDATE_ROLLBACK_COMPLETE: Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: 288658c4-a8bc-4194-bd5c-a1db50c45d2d; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:426:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:429:153307)
    at async /abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:429:136985

 ❌ Deployment failed: Error: The stack named glue-alpha-python failed to deploy: UPDATE_ROLLBACK_COMPLETE: Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: 288658c4-a8bc-4194-bd5c-a1db50c45d2d; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:426:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:429:153307)
    at async /abc/.nvm/versions/node/v18.16.1/lib/node_modules/aws-cdk/lib/index.js:429:136985

The stack named glue-alpha-python failed to deploy: UPDATE_ROLLBACK_COMPLETE: Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs. (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: 288658c4-a8bc-4194-bd5c-a1db50c45d2d; Proxy: null)

Reproduction Steps

# glue_alpha_python_stack.py

from constructs import Construct
from aws_cdk import (
    Stack,
    aws_glue_alpha as glue_alpha
)

class GlueAlphaPythonStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        glue_alpha.Job(
            scope=self,
            id="sample-job-A",
            executable=glue_alpha.JobExecutable.python_shell(
                glue_version=glue_alpha.GlueVersion.of("4.0"),
                python_version=glue_alpha.PythonVersion.THREE_NINE,
                script = glue_alpha.Code.from_asset('script/hello_world.py'),
            ),
            description='an example Python Shell job',
        )
# app.py

#!/usr/bin/env python3

import aws_cdk as cdk

from glue_alpha_python.glue_alpha_python_stack import GlueAlphaPythonStack

app = cdk.App()
GlueAlphaPythonStack(app, "glue-alpha-python")

app.synth()

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.89.0 (build 2ad6683)

Framework Version

2.89.0a0

Node.js Version

v18.16.1

OS

Ubuntu 20.04 on Windows10 ( WSL2 )

Language

Python

Language Version

Python (3.8.10)

Other information

Same problem occured with GlueVersion.JobType.

glue_alpha.Job(
            scope=self,
            id="sample-job-B",
            executable=glue_alpha.JobExecutable.of(
                language=glue_alpha.JobLanguage.PYTHON,
                type=glue_alpha.JobType.PYTHON_SHELL,
                glue_version=glue_alpha.GlueVersion.V1_0,
                python_version=glue_alpha.PythonVersion.THREE_NINE,
                script = glue_alpha.Code.from_asset('script/hello_world.py'),
            )
)
# $ cdk ls
# glue-alpha-python

glue_alpha.Job(
            scope=self,
            id="sample-job-B",
            executable=glue_alpha.JobExecutable.of(
                language=glue_alpha.JobLanguage.PYTHON,
                type=glue_alpha.JobType.of("pythonshell"),
                glue_version=glue_alpha.GlueVersion.V1_0,
                python_version=glue_alpha.PythonVersion.THREE_NINE,
                script = glue_alpha.Code.from_asset('script/hello_world.py'),
            )
)

# $ cdk ls
# ...
# RuntimeError: Error: Specified PythonVersion PythonVersion.THREE_NINE is only supported for JobType Python Shell and Ray
peterwoodworth commented 1 year ago

Thanks for the report, you're right that we should check for this at synth time

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

peterwoodworth commented 1 year ago

Closing in favor of #26620