aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.72k stars 3.94k forks source link

(glue): Add api for "--job-bookmark-option" #21954

Open jankatins opened 2 years ago

jankatins commented 2 years ago

Describe the feature

Enabling or disabling job bookmarks is an important part of a glue job. Unfortunately, it seems this is not yet supported by the glue.Jobin a nice way, only via default_arguments.

Use Case

It's a big feature of glue jobs, that you can resume from where you left off.

Proposed Solution

Add a new flag somewhere which takes an enum with the required three values and add the appropriate flag into the default arguments.

Other Information

No response

Acknowledgements

CDK version used

2.40.0 (build 56ba2ab

Environment details (OS name and version, etc.)

mac

peterwoodworth commented 2 years ago

This seems to be modeled closely after the CloudFormation API which seems to me only supports this option in defaultArguments

We could potentially expose this in a more convenient way through our API doing something similar to your suggestion. However I'm not certain that's the best thing for us to do, personally I'm not super familiar with Glue as a service or how people typically define a Job in CloudFormation

The team won't be able to get to this, but we do accept contributions! Check out our contributing guide if you're interested - we'd be happy to review a PR

jankatins commented 2 years ago

Just to clarify: I meant the the aws_glue_alpha package with the L2 constructs.

There is already this:

 executable=glue.JobExecutable.python_etl(
...
                    extra_python_files=[glue_additional_python_files],
                ),

results in these two default args:

            "DefaultArguments": {
                "--job-language": "python",
                "--extra-py-files": extra_py_files,

I've no experience with CF, but the current way results in an empty field in the glue UI, which seems wrong, given that the default for a new glue job created in the UI is a "Enable" and there is no way to actually clear that field.

peterwoodworth commented 2 years ago

Yes I'm aware you meant our alpha package 🙂

I mention CloudFormation because if our only option for specifying this is within CloudFormation's DefaultArguments then that will impact our design and implementation of this potential feature in the L2

jankatins commented 2 years ago

I find the mismatch between "add it as a default argument" (which is also used for passing in arguments to the job script itself) and "it has an extra drop down in the UI instead of an simple input field, like the arguments for your job" very strange and I'm actually looking forward to have a proper API for all these interesting switches :-)

kevinsalerno commented 8 months ago

Following. Would be nice to have this